Documentation menu

Docs Investigation guides

Capacity planning

Ask capacity planning for this server, how long until the disks fill up?, do we need more CPU? or do we need more nodes? with a database or cluster attached. Verbiqo forecasts from history and says both ways: what needs more, and what has far more than it uses.

How to read this page. Each section is one area. Why it helps says what you get from it. Ask is a question to type that runs it. The table under it lists what Verbiqo checks there: you do not run those one by one. When a check finds a problem, it appears under the answer as a finding, with its evidence and a suggested fix.

How to ask

Any of these words brings the capacity checks in: capacity, runway, forecast, prediction, how long until, when will, fill up, run out of disk, disk growth, disk usage, storage growth, growing, sizing, scale up, scale out, headroom, planning, provisioning.

AttachedAsk, for exampleWhere the history comes from
/pg or /mysqlHow long until the disks fill up?
Is this server oversized?
A connected Prometheus or Thanos that scrapes the database’s machine (node_exporter) and, for connections, the database’s exporter (postgres_exporter, mysqld_exporter). Verbiqo finds the machine by the connection’s host.
/mssqlHow long until the data drive fills up?A connected metrics source that scrapes the machine: windows_exporter on Windows, node_exporter on Linux. The drive holding the largest data files is forecast as the data disk.
/redisWhen will Redis hit maxmemory?Memory from INFO now; its history and clients from redis_exporter (including targets scraped as redis://host:port, matched by the connection’s port); the machine from node_exporter.
/esWhen do we hit the watermarks?
How many nodes do we need?
The cluster itself (index sizes and creation dates); for each data node’s CPU, a connected metrics source that scrapes it.
/k8sIs the cluster full?
Which workloads are over-provisioned?
The API server now (requests, limits, node capacity, quotas); with a connected Prometheus, throttling and memory peaks over the day. The checks are on the Kubernetes page: Room and cost.

A question reaches only what is attached: machine history comes from a Prometheus source attached in the same conversation (/prom name beside /pg name), never from a saved connection nobody attached. No metrics source attached, or none that scrapes the machine: Verbiqo says so, names the saved ones, and forecasts what it can from the database or cluster alone.

Disk runway

Why it helps: Forecasts when each disk fills, from its own growth, so you add space before writes fail.

Ask How long until the disks fill up?

CheckWhat it works out
Each diskThe dates it reaches 80%, 90% and full, and what each means. The disk holding the data directory is named.
A disk freed and refilledLogs rotated, binary logs purged: shown as the range it moves in, labelled a rough guide.

CPU

Why it helps: Shows whether CPU is close to its limit at peak, so you know when to scale up, or that you can scale down.

Ask Is this server oversized?

CheckWhat it works out
CPU highThe 95th percentile at 70% or more (critical at 85%), with the peak (the busiest five minutes, so short spikes count) and the cores that bring the 95th percentile to 60%.
CPU risingWhen a rising daily peak reaches 80%, within 90 days.
CPU oversizedA machine of 4 or more cores busy under 20% at the 95th percentile and under 40% at its peak, with the core count that would do.

Memory and swap

Why it helps: Shows whether memory and swap are close to their limits at peak, before the system starts killing processes.

Ask Is this server running out of memory?

CheckWhat it works out
Memory lowThe least memory available over the period, and when: below 5% is a warning, below 2% critical.
SwapSwap in use over 256 MB, and whether it is growing.

Connections and caches (PostgreSQL and MySQL)

Why it helps: Forecasts when connections reach the server’s limit and spots sudden bursts, so new clients are not refused.

Ask When will we run out of connections?

CheckWhat it works out
Connections highThe 95th percentile and peak against the limit.
Connections risingWhen the daily peak reaches 80% of the limit.
Connection burstA burst far above the usual level: a retry storm, a reset after a failover.
Limit far too highA limit far above anything used, which only lets a storm grow larger.
Buffer pool (MySQL)A buffer pool many times all the data, or still largely empty after days.
shared_buffers (PostgreSQL)Larger than all the databases together, or a cache hit ratio too low for the data.

Redis memory

Why it helps: Forecasts when Redis reaches maxmemory and starts evicting or refusing writes.

Ask When will Redis hit maxmemory?

CheckWhat it works out
Memory runwayThe dates memory reaches 80% and 100% of maxmemory, from redis_exporter’s history, and what happens at the limit by the eviction policy: writes fail with noeviction, keys are evicted otherwise. With no maxmemory, against the machine’s memory, where the operating system kills Redis.
ClientsConnected clients against maxclients, as for database connections above.
Snapshot diskThe disk where snapshots and the AOF are written, when persistence is on.

Elasticsearch

Why it helps: Forecasts when nodes hit the disk watermarks and how many nodes you need.

Ask When do we hit the watermarks?

CheckWhat it works out
Watermark runwayThe dates the cluster reaches the low, high and flood-stage watermarks, from its own index sizes and creation dates, and the size it levels off at where lifecycle policies delete old data.
Node countThe data nodes the data needs now and in 90 days. See Sizing, tiers and cost for the tier and cost checks.
Node machinesCPU, memory and swap for each data node’s machine, as above, and a tier whose every node is far larger than its load.

What each forecast rests on

Why it helps: Says what each forecast is based on and how far to trust it, so you know when a forecast is solid and when it is a guess.

  • Every forecast gives its readings, the span they cover and how well the trend fits (a steady, fair or noisy trend).
  • A noisy trend is labelled a rough guide, and its severity softened.
  • Less than a day of readings gives no date at all.
  • An oversized machine comes with a caution: a quiet fortnight is not the whole year. Check month-end, batch and failover load first.

How far back it looks

Why it helps: Says how much history each forecast uses, so a short history is never mistaken for a long trend.

Forecasts read 14 days of history by default. Change it under Settings › Extensions › Forecast history (1 to 90 days), or with VERBIQO_CAPACITY_HISTORY_DAYS in a terminal. See Settings reference.

Something missing or unclear? Email hello@verbiqo.app.