Documentation menu

Docs Connections: PostgreSQL

PostgreSQL checks

Every check Verbiqo can run on a PostgreSQL server, grouped by area. You never name a check: you ask in plain words; the model says which areas the question needs, and Verbiqo runs those checks first. Checks that find something serious, such as wraparound or IDs running out, are reported on any question.

How to read this page. Each section is one area. Why it helps says what you get from it. Ask is a question to type that runs it. The table under it lists what Verbiqo checks there: you do not run those one by one. When a check finds a problem, it appears under the answer as a finding, with its evidence and a suggested fix.

How to ask for each area

Attach the server with /pg and the connection’s name, then ask. Any of the words in the middle column brings that area in; a question can bring in several.

AreaWords Verbiqo goes by without the modelAsk, for example
Right nowslow, hanging, stuck, blocked, locks, waiting, busy, timeouts, spike, right now, currently, what is it doing, performanceThe app is slow, what is the database doing?
Queries and indexesslowest, top queries, expensive, heaviest, missing index, full scans, sequential scans, optimise, cache hitWhich queries cost the most? Are any indexes missing?
A statement’s planpaste the statement into the question: SELECT, UPDATE, DELETE or INSERTWhy is this slow: DELETE FROM …
Replication and availabilityreplication, replica, standby, lag, failover, switchover, primary, WAL, archiving, backups, slots, wraparound, freeze, crash, restart, disk full, recovery, outageWhy is this replica falling behind?
PgBouncerpgbouncer, pooler, pool, waiting for a connection, too many clients, max_client_conn; also read for a problem happening now and for a reviewWhy are clients waiting for a connection?
The whole clustercluster, nodes, members, Patroni, leader, standbysIs the cluster healthy? Who is the leader?
Vacuum, bloat and writesbloat, dead rows, free space, HOT updates, fillfactor, WAL volume, full-page, I/O, writes, background writer, checkpoints, vacuum, reindexIs anything bloated? Why is WAL growing so fast?
Whether a change helpeddid the changes help, since the change, before and after, check the tuning, impact of the settings, did the index help, is the new index usedDid the changes help? Is the new index used?
Settings and memorytune, settings, configuration, parameters, memory, shared_buffers, work_mem, autovacuum, checkpoints, max_wal_size, recommendations, best practicesWhat should I tune on this server?
IDs, indexes and statisticssequences, serial, identity, run out of IDs, duplicate or redundant indexes, stale statistics, analyze, health checkAre any sequences close to running out?
Securitysecurity, hardening, audit, superuser, roles, privileges, grants, TLS, SSL, encryption, pg_hba, passwords, SCRAM, MD5, complianceSecurity review of this database
Capacitycapacity, forecast, how long until, fill up, growth, sizing, scale up, headroom, planningHow long until the disks fill up?
What happenedwhat happened, timeline, incident, outage, yesterday, last night, at 3pm, when did, root causeWhat happened on this server last night?
Every database on the serverall databases, every database, across the databasesCheck all databases for sequences running out
The databases and their sizesdatabase sizes, how big, largest, list the databases, what databases; also read for any question about all databases, space or capacityFind all databases and their sizes (measured by Verbiqo from pg_database, including databases this connection cannot open)
What Verbiqo can seewhat should I enable, readiness, permissions, privileges, what does Verbiqo needWhat should I enable?

Right now: sessions, waits and locks

Why it helps: When something is slow or stuck now, it shows what the database is doing and what is holding it up, so you know whether the database is the cause and what to clear.

Ask The app is slow, what is the database doing?

Verbiqo samples the active sessions for 10 seconds (3 to 30, see sampling) and reports:

CheckWhat it looks for
BlockingA session blocking others, with the chain of who waits on whom and the statement holding the lock.
Lock waitsThe share of active session time spent waiting on locks during the sample.
Disk waitsThe share of active session time spent reading or writing disk.
Long-running statementA session running one statement for a long time.
DeadlocksDeadlocks that happened during the sample.
Temporary filesStatements that spilled to temporary files during the sample.
Connections near the limitClient connections as a share of max_connections, with who is using them.
Maintenance in progressIndex builds, vacuum, analyze, cluster and base backups running now, with their phase and progress.

Queries and indexes

Why it helps: Shows which statements cost the most in total and where an index would help, so you fix the few queries that matter most.

Ask Which queries cost the most? Are any indexes missing?

From pg_stat_statements and the table and index statistics. pg_stat_statements covers every database on the server, so each statement names its database and role; one counted once per database and role is shown once, with its figures summed. A statement that runs in another database says so, and is not planned or given an index from the database Verbiqo is connected to: name its database in the question to look at its tables.

CheckWhat it looks for
One statement dominatesA single statement taking a large share of all statement time.
Slow statementA statement with a high average time per call.
Statement spills to diskA statement writing temporary files on every call: a sort or hash too big for work_mem.
Full scansLarge tables read in full many times, which an index would avoid.
Unused indexAn index never scanned since statistics were reset, with what it costs to keep. An index that backs a unique, primary-key or exclusion constraint is never offered for dropping: where a query does not say, Verbiqo looks it up, and offers no drop when it cannot. Indexes under 1 MB are left out.
Invalid indexAn index left invalid by a failed CREATE INDEX CONCURRENTLY: maintained on every write, never used.
Plans of the heaviest readsWith PostgreSQL 16 and pg_read_all_data, the plans of up to 10 of the heaviest reads (3 if your organization limits how much Verbiqo reads per question), checked as in A statement’s plan.
Index for a joinA table a join reads in full for every row of the other side. With hypopg, an index on the join column is tested; it is suggested only if PostgreSQL would use it and it cuts the estimated cost by at least a fifth.
Index for sorted resultsA query that sorts a whole large table to return its first rows (ORDER BY … LIMIT). With hypopg, an index on the filter’s columns, then the sort columns, is tested, and suggested on the same terms.
One index for two queriesThe same index suggested for two queries is shown once. Where one query’s index starts with another’s columns, the wider index is tested on both; if PostgreSQL would use it for both, one index is suggested instead of two.

Index tests use hypothetical indexes: nothing is built, and they are cleared before Verbiqo answers. Without hypopg, only the full-scan suggestions are made, as before.

A statement’s plan

Why it helps: Explains why one statement is slow from its plan, without running it, so you can fix that statement.

Ask Why is this slow? SELECT * FROM orders WHERE status = $1

Paste a statement into the question. Verbiqo asks PostgreSQL for the plan without running it (it needs PostgreSQL 16 and pg_read_all_data, see Explain a slow statement). For an UPDATE, DELETE or INSERT, it plans the search for the rows and reads what writing each row costs.

CheckWhat it looks for
Full scan that keeps littleA sequential scan of a large table that keeps a small share of its rows: an index candidate.
Sort spilled to diskA sort that did not fit in work_mem.
Large sortA sort the planner expects to be bigger than work_mem.
Estimate far from realityThe planner expecting far more or fewer rows than it got, usually stale statistics or correlated columns.
Nested loop over a full scanA table read in full once per outer row.
Suggested index testedWith hypopg installed, whether PostgreSQL would use the index Verbiqo suggests, before you build it.
Unindexed foreign key (writes)A DELETE, or an UPDATE of a key, on a table that other tables reference with no index on their side: each row scans the other table. Gives the index to create.
Update of an indexed column (writes)An UPDATE that sets an indexed column cannot be a HOT update, so it writes a new entry in every index of the table.
Indexes and triggers (writes)How many indexes each written row updates, and which triggers run.

Replication and availability

Why it helps: Shows whether replicas keep up, WAL is archived and IDs are safe from wraparound, so you hear about a failover or data-loss risk before it happens.

Ask Why is this replica falling behind?

CheckWhat it looks for
WAL archiving failingThe archiver failing, so point-in-time recovery has a gap and WAL piles up.
Synchronous standby missingCommits waiting for a synchronous standby that is not connected.
Replay pausedWAL replay paused on a replica.
Replay lagWAL received by a replica but not yet replayed.
Vacuum held backWhat keeps old rows from being cleaned up: a long session, a prepared transaction, a standby with hot_standby_feedback, or a replication slot.
WAL growingpg_wal holding several times max_wal_size, usually a slot or failing archiving.
Freeze overdueTables past autovacuum_freeze_max_age.
WraparoundHow far each database is towards transaction ID wraparound. Reported on any question once it is close.
Logical slot without a consumerA logical replication slot nobody is reading from, holding WAL on disk; and a consumer far behind the server.
SubscriptionsOn a server that subscribes: a subscription disabled, enabled without an apply worker, or recording apply and sync errors.

PgBouncer

Why it helps: Shows whether clients wait in the pooler, and why, so you can tell a pool size problem from a slow database.

Ask Why are clients waiting for a connection?

When the attached address is a PgBouncer, Verbiqo reads its console (the virtual database pgbouncer at the same host, port and credentials; the account must be in stats_users or admin_users). Nothing beyond the attached address is reached; a direct server has no console and the read is skipped.

CheckWhat it looks for
Clients queuedClients waiting for a server connection, how long the oldest has waited, and whether the pool is at its size or the server is not answering.
Server login stuckConnections to the server stuck logging in while clients wait.
Client limitClient connections near max_client_conn.
Paused or disabledA database an operator paused or disabled, so clients wait or are refused.

The whole cluster: standbys, Patroni and backups

Why it helps: Shows every node’s role and state, Patroni’s view and backups, so you know whether the cluster survives losing a node.

Ask Is the cluster healthy? Who is the leader?

From the attached node, Verbiqo reports the others as that node sees them (from a primary, its standbys with their state and lag; from a standby, its upstream). It does not connect to them: a question reaches only what is attached, so another node is read by attaching it in a conversation of its own (see Clusters). If the question itself asks for it, for example read the other nodes with this connection’s credentials, Verbiqo reaches the nodes the attached one reports, at the addresses it gives and with the same credentials; a saved connection is still never used. With a Prometheus source attached in the same conversation it adds Patroni and pgBackRest. A standby it did not read still shows its state, sync mode and replay lag as the primary sees them. A logical replication client (Debezium, a subscriber) is reported as a consumer with its slot and lag, not as a node to attach, and is not counted as a standby.

CheckWhat it looks for
Node not readA node that could not be reached or confirmed as part of the cluster, and why.
Standby on an old timelineA standby still on a timeline the primary has left, after a failover.
Standby not streamingA standby that is catching up from archive or has stopped.
Patroni member missingA member Patroni lists that is not replicating from the primary.
Patroni leader disagreesPatroni’s leader is not the node accepting writes.
No Patroni leaderThe Patroni cluster has no leader.
FailoversTimeline changes in the last 7 days.
Restart pendingPatroni reports a restart waiting on this member.
BackupsThe last pgBackRest backup too old, page checksum errors found by a backup, WAL archiving not recorded, or the stanza reporting an error.

Vacuum, bloat and writes

Why it helps: Finds wasted space, vacuum falling behind and heavy WAL, so tables stay small and fast and disks do not fill.

Ask Is anything bloated? Why is WAL growing so fast?

CheckWhat it looks for
Autovacuum behindTables with many dead rows and no recent vacuum.
Measured bloatFree space and dead rows in the largest tables, measured with pgstattuple where it is installed.
Few HOT updatesTables where most updates also write every index. When pages are too full, a fillfactor sized from the table: room for the rows updated between vacuums, and at least one more row (70 to 95). When updates change indexed columns, it says fillfactor will not help.
Sessions writing buffersSessions writing dirty buffers or syncing files themselves, because the background writer and checkpointer fall behind.
Background writer limitThe background writer stopping at its per-round limit.
Full-page imagesA large share of WAL records carrying full page images: checkpoints too close together.
WAL buffers fullWAL buffers filling and forcing writes before commit.

Settings and memory

Why it helps: Judges each setting against what the server actually does, with the statement to change it, so you tune from evidence rather than rules of thumb.

Ask What should I tune on this server?

Settings are judged against what the server has been doing. Sizes that depend on the machine (memory, cores, storage) come from a Prometheus source attached in the same conversation (/prom name), or from what the question states (32 GB of memory, 8 cores, SSD); each finding says which. To see whether a change helped, see After a change.

CheckWhat it looks for
Unsafe settingsfsync or full_page_writes off (corruption after a crash), autovacuum off.
CheckpointsCheckpoints forced by WAL volume rather than time (raise max_wal_size), or written in a shorter burst than needed (checkpoint_completion_target). On a busy server with a day of statistics, checkpoints every five minutes (checkpoint_timeout).
WAL compressionFull-page images making up a large share of WAL while wal_compression is off (PostgreSQL 14+).
JITShort statements spending a large share of their time compiling (jit, jit_above_cost; PostgreSQL 15+).
work_memStatements by a role spilling to disk on every call, with the size that would hold them.
Temporary filesHow much the database writes to temporary files, per day.
Autovacuum per tableLarge tables waiting for too many dead rows before autovacuum starts. The table’s own threshold is sized from its size: about 100,000 dead rows per vacuum, between 0.2% and 5% of the table.
Append-only tablesTables that are only inserted into, with most pages not yet marked all-visible: index-only scans still read the table, and freezing piles up. Gives a smaller autovacuum_vacuum_insert_scale_factor for the table (PostgreSQL 13+).
Autovacuum workersEvery worker busy when Verbiqo looked.
Autovacuum speedMany tables past their vacuum threshold at once while vacuum runs at the default cost limit (autovacuum_vacuum_cost_limit).
Vacuum memoryA vacuum scanning the same indexes more than once for lack of memory.
shared_buffers and effective_cache_sizeLeft at their defaults, or out of proportion to the machine’s memory.
maintenance_work_memOut of proportion to the machine’s memory.
Planner costs for SSDrandom_page_cost and effective_io_concurrency left at the spinning-disk defaults on SSD or network storage.
Parallel workersWorker limits below what a machine with more than eight cores can use.
Memory overcommitwork_mem × max_connections larger than the machine.
Disk spaceA disk on the machine low on free space.

IDs, indexes and statistics

Why it helps: Finds IDs about to run out, duplicate indexes and stale statistics: problems that build quietly until inserts fail or plans go wrong.

Ask Are any sequences close to running out?

These are read in one database at a time: the one connected to, or the one the question names. When the server has other databases (the default postgres alone does not count), Verbiqo says which it read and which it did not, and an answer may not call the whole server clear (“nothing needs vacuuming”) from one. Ask about all databases to read each.

CheckWhat it looks for
Sequences running outA sequence against the column it feeds: a bigint sequence behind an integer key runs out at 2.1 billion. Reported on any question once it is close.
Duplicate and overlapping indexesAn index another one already covers. A covered index under 1 MB is left out; an exact duplicate is always reported.
Stale statisticsTables changed well past autovacuum’s analyze threshold and not analyzed for a day.
PartitionsA range-partitioned table whose last partition has already ended or ends within two weeks, and rows piling up in a DEFAULT partition.

Security

Why it helps: Shows who can do too much, what connects unencrypted and what anyone may change, so you close the gaps an audit would find.

Ask Security review of this database

CheckWhat it looks for
A permission error you pasteWhich role lacks which privilege on the object the error names, read from the catalog, with the grant that should fix it. Verbiqo asks which role when the error does not say (Permission errors).
Login superusersRoles that can log in as superuser.
Wide rolesLogin roles that can create roles or bypass row security.
Expired passwordsRoles whose password has expired.
TLSTLS off, or remote sessions connected without it.
Open schemasSchemas where every role may create objects.
pg_hba.confRules that need no password (trust) or send it in clear text (password). Readable only by a superuser; Verbiqo says when it could not check.
MD5 passwordsMD5 authentication still in use instead of SCRAM.

The same checks are controls in the security review, which covers every saved connection at once and maps each to CIS Controls, ISO 27001 and SOC 2.

What Verbiqo can see

Why it helps: Lists what Verbiqo can and cannot see on this server, with the statement that turns each missing item on, so its answers are as complete as possible.

Ask What should I enable?

Ask what should I enable? for each optional extension and setting (pg_stat_statements, pgstattuple, hypopg, track_io_timing and others) marked ready or not, with the statements for your server. The monitoring role itself is in Set up PostgreSQL.

Ready-made reports

A report is one fixed reading of one area: no question, no model, the same output every time. Use one to look at an area yourself, or to share what it shows.

From a terminal, it prints the report:

verbiqo pg run orders blocking-tree

In verbiqo chat, it loads the report into the conversation so you can ask about it. help lists the reports:

/pg orders blocking-tree
/pg orders help

In the desktop apps, there is no report command: ask in your own words, and Verbiqo reads what the question needs.

ReportShows
blocking-treeSessions waiting on a lock, and what holds it.
long-runningTransactions and queries open longest.
idle-in-transactionSessions holding a transaction open while doing nothing.
table-bloatDead rows and when each table was last vacuumed.
unused-indexesIndexes never scanned, and what they cost to keep.
cache-hitHow much of the working set is served from memory.
replicationStandbys, and how far behind each one is.
connectionsWhere connections are going, and how close to the limit.
settings-of-noteSettings that most often explain a performance complaint.

Capacity checks for PostgreSQL are on Capacity checks, and what Verbiqo finds in the PostgreSQL log is on Log findings.

Something missing or unclear? Email hello@verbiqo.app.