Documentation menu

Docs Connections: Prometheus

Ask Prometheus

Attach a connection and ask. You name the service or host, the time and what you want; Verbiqo chooses the jobs, hosts, metrics and queries from its local map, and answers from what it measured.

Attach

/prom prod

Use the name you gave the connection. /prom off stops reading it; /sources shows what is attached.

What to ask

You wantAsk
A measurementHow was CPU on the MySQL servers over the last six hours?
A peakWhat was the highest memory use on db01 yesterday?
A causeWhy did the payments service slow down yesterday afternoon?
Current stateIs anything down right now?

Name the service or host, the time, and whether you want a measurement or a cause. How to phrase a question, what happens when Verbiqo asks you something, and how to read the answer: Ask about your systems.

Prometheus also answers alongside a database or cluster: attach /pg, /mysql, /mssql, /redis or /k8s with it, and the machine’s history, reboots and capacity forecasts join that system’s answer and timeline.

Follow-up questions

  • Same time: a question with no time, or one that says the same time, then or during that outage, reads the window of the question before.
  • Same subject: what could be the cause? after find any down mysql looks at the services that were down.
  • Names: words a job writes together are found either way: mysql router finds monitoring-analytics-mysqlrouter. A name that matches no job is said at the end of the answer, with what was used instead.

From a terminal

In verbiqo chat, attach with /prom as in the app, or name the connection when you start: verbiqo chat --prom prod "is anything down right now". The verbiqo prom commands below need no chat session; run, query, range and describe need no model either.

verbiqo prom ask prod "why was mysql down yesterday"
verbiqo prom run prod targets
verbiqo prom query prod "sum by (job) (up)"
verbiqo prom range prod "sum(rate(node_cpu_seconds_total[5m]))" --minutes 60 --step 60
verbiqo prom describe prod
CommandDoes
ask <name> "<question>"The same investigation as the app, printing each step as it happens. --quiet prints only the answer; --json adds the queries, coverage and any unsupported figures.
run <name> <report>One of the fixed reports below.
query <name> "<PromQL>"One instant query, within the same guards as a question.
range <name> "<PromQL>" [--minutes 60] [--step 60]One range query ending now, over the last --minutes, at --step seconds.
describe <name>Version, number of series, and the ten metrics with the most series.

Connection and map commands (add, test, index-*, cache-*) are on Set up Prometheus.

Reports for run

Fixed expressions for the questions people ask a server before they know what to ask it. No model is involved.

ReportShows
targetsWhich scrape targets are down.
scrape-durationThe ten slowest scrapes, which is usually what breaks first.
cardinalityHow many series each job contributes.
rule-failuresRecording and alerting rules that are failing to evaluate.
ingestionSamples being appended per second.

Something missing or unclear? Email hello@verbiqo.app.