Docs / Connections: Prometheus

Set up Prometheus

Verbiqo reads metrics through the Prometheus HTTP API (for Thanos, the Querier). It keeps a map of what the server watches on your computer, so it finds the right jobs, hosts and metrics for a question.

At a glance

Connect

1. Name and address

   Extensions… › Prometheus. A short name (prod), the backend (Prometheus or Thanos), and the URL, such as https://prometheus.example.com. A reverse-proxy prefix such as /prometheus is kept. Credentials in the URL are not accepted.

2. Token, if needed

   A read-only bearer token when the API requires sign-in.

3. Save and test

   Save Connection, then Test Connection.

4. Build the map (required before you ask)

   Either Build Index, which reads every job, host and metric once into a local map and needs an embedding service named under Semantic index (it embeds as it builds). Or, without one, Refresh technology for each technology you will ask about (mysql, postgres, node), seconds each. On a large server a full build takes up to half an hour. Run it again to continue. Why it matters, and what happens without it.

• Keep it current: type words from service names under Refresh technology (such as mysql, postgres) and press Refresh.
• Private CA: name it once in Settings › Trusted CA (below Endpoint). A terminal on the same computer trusts it too, with nothing to set.
• Finding by meaning (optional): the Semantic index uses an embedding service you name. Only metric names and descriptions are sent, never values. What the service needs.

The local map: a prerequisite

Build it once per connection before the first question. It is the single thing that most decides whether Prometheus answers are right.

What it is

Two SQLite files on your computer, never on the server: metrics.sqlite, the map itself, and discovery.sqlite, a ten-minute cache of raw API answers. The map holds every scrape job, the hosts behind each job, the exporter ports on each host, and the list of metrics each target exposes, plus the identity rules that tie a target to a host. It is built from what the server reports. Nothing configured narrows it.

verbiqo prom cache-path prints it.

The map and the embeddings, in plain words

Think of the map as a directory and the embeddings as a thesaurus.

• The directory (always needed): which jobs exist, which machines are behind each one, which exporter runs on which port, and which metrics each machine reports. This is how Verbiqo knows that the MySQL servers means these twelve hosts and their mysqld_exporter on port 9104. It is read from Prometheus and needs no embedding service.
• The thesaurus (optional): finds a metric when your words and its name have nothing in common, such as failover and patroni_primary, or disk usage and node_filesystem_avail_bytes. It needs an embedding service.

Where embeddings come into building the index: only in one step. As Build Index reads the server, it also sends each metric’s description and a short summary of each job and host to the embedding service, and stores what comes back. Every other step (jobs, hosts, ports, each target’s metrics) is the same with or without it. That is why Refresh technology builds the same directory with no embedding service at all.

Without an embedding service, everything above still works. A part of a question that matches no metric by its words is reported as unchecked rather than guessed at. If a service is set up but not running when you ask, the question still runs, using names alone.

Why it matters

• Scope. “The MySQL servers” is resolved against the map: job names first, then host names, so mysql means every MySQL job, not one someone configured once. The model never has to remember your topology, and never invents hosts. A name that matches several things is asked back with the choices. One that matches nothing is said.
• The right metric on the right port. Each host’s exporters and their ports are in the map, so a CPU question for a database host reads node_exporter on that host, not a guess at :9100.
• Coverage you can trust. “3 of 4 hosts answered, db04 is not monitored for this job” is only possible when Verbiqo knows what should be there.
• The other extensions use it too. A PostgreSQL, MySQL, SQL Server or Redis question finds its machine’s memory, disks and history through this map.

Without it

Questions still answer, but worse and slower, and the server does more work:

• Each question reads the jobs it needs live: the job’s targets, then each target’s metric list, paced. At most two missing jobs are read per question. The rest are “read live for now and join the map on later questions”, and the answer says so.
• Subject resolution is weaker until the map fills: a broad word such as mysql may reach only the jobs read so far.
• The first questions take longer (seconds to a minute more each) and send many small discovery requests to Prometheus instead of one paced build.
• Host matching for the database extensions returns “no metrics source has a target on this host” until that host’s job is in the map.

Two ways to build it

• The whole estate at once: Build Index in the app, or verbiqo prom index-build <name>. Both embed metric descriptions and job documents as they go, so they need the embedding service below. Without one they stop and say so.
• A technology at a time, no embedding service needed: Refresh technology in the app, or verbiqo prom index-refresh <name> --jobs mysql,postgres,node. Seconds per technology. The map holds those jobs fully. Questions then fill in any job they meet that is still missing (two per question).

What it costs

• Time: a full build is about thirty minutes on a large server (hundreds of jobs). Small servers take a minute or two. One run stops after an hour and keeps what it read. Run it again to continue.
• Load on Prometheus: small metadata reads only (/series, /label/…/values, /metadata), one job at a time with a pause between, never a value or a range query. The same protections and limits as questions apply.
• Disk: mostly the list of metrics each target reports, about 0.5 to 1 MB a target: tens of MB for a small server, 1 to 2 GB for one with a few thousand targets. Embedding vectors add up to about 120 MB. The file grows as needed. Large estates.
• Keeping it current: a job you ask about that is older than a day is refreshed on that question. Refresh technology (or verbiqo prom index-refresh <name> --jobs mysql,postgres) refreshes the named jobs in seconds. Nothing else in the map is touched.

Large estates: disk space and limits

The index grows as your estate needs, and stops only where your disk would be left with less than 2 GB free. Nothing needs setting for most people.

Targets that report more metrics (node_exporter with many collectors, Kubernetes nodes) sit at the top of each range.

To set a ceiling for managed computers, set VERBIQO_PROMETHEUS_INDEX_MAX_MB to the most the index may take, in MB (64 or more). The disk floor still applies.

# macOS, for the app (then quit and reopen Verbiqo)
launchctl setenv VERBIQO_PROMETHEUS_INDEX_MAX_MB 4096

# Windows, for the app (then quit and reopen Verbiqo)
setx VERBIQO_PROMETHEUS_INDEX_MAX_MB 4096

# A terminal
export VERBIQO_PROMETHEUS_INDEX_MAX_MB=4096

• When it is full, a build stops with The local Prometheus index ran out of room (or reached its N MB limit). Everything stored so far is kept. Free disk space or raise the ceiling, then build again. It continues where it stopped.
• Clear Index gives the space back to the disk, for the selected server.
• Old copies: metric vectors made with a model you no longer use are removed 30 days after they were last refreshed, and each target’s metric list is replaced as it is read again. Clear Index removes everything for a server at once.
• Only part of the estate? Refresh technology maps just the jobs you name (mysql, postgres, node), which keeps the index small on a very large server.

The semantic index

The thesaurus above. With an embedding service named in Settings, the map also carries a vector for every metric description and every job, host and target document, so disk usage can find a metric that shares no word with it. A full Build Index needs one. A map built technology by technology works without it, and questions then find metrics by name and prefix, which is enough for the common metrics.

Embedding service requirements

Models that fit:

To run the default with Ollama, and check it answers:

ollama pull nomic-embed-text
curl -s http://localhost:11434/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"nomic-embed-text","input":["node cpu seconds total"],"encoding_format":"float"}' | head -c 200

Then in Extensions… › Prometheus › Semantic index, set Embedding endpoint to http://localhost:11434/v1/embeddings and Embedding model to nomic-embed-text.

In a terminal on the same computer, verbiqo uses the same Semantic index settings and key, so verbiqo prom index-build builds the same index. A shell can set its own instead: VERBIQO_METRIC_EMBEDDING_URL, VERBIQO_METRIC_EMBEDDING_MODEL, VERBIQO_METRIC_EMBEDDING_API_KEY, and VERBIQO_METRIC_EMBEDDING_ALLOW_REMOTE=1 for a remote address. When any of the first two is set, the app’s settings are not used.

Through a LiteLLM gateway, with vLLM behind it

A shared embedding model on a GPU server, reached through the same LiteLLM gateway as the answering model, so one key and one trusted certificate cover both. Verbiqo sends metric names to a remote address only over HTTPS, so laptops call the gateway, not vLLM’s own plain-HTTP port.

1. Serve the model with vLLM. This Qwen3-Embedding-0.6B template uses the vLLM 0.30.0 installation and service account. Its flags were checked against upstream documentation, but this service configuration has not been runtime-validated on a GPU. Copy the protected backend key file for systemd to read:

sudo install -m 600 -o root -g root /etc/llm/vllm.env /etc/llm/embedding.env

Save this root-owned unit as /etc/systemd/system/llm-embedding.service:

[Unit]
Description=Qwen3 embeddings for Verbiqo
After=network-online.target
Wants=network-online.target

[Service]
User=llm
Group=llm
WorkingDirectory=/var/lib/llm
EnvironmentFile=/etc/llm/embedding.env
Environment=HF_HOME=/var/lib/llm/huggingface
Environment=XDG_CACHE_HOME=/var/lib/llm/cache
Environment=VLLM_CACHE_ROOT=/var/lib/llm/cache/vllm
Environment=VLLM_CONFIG_ROOT=/var/lib/llm/config/vllm
Environment=VLLM_NO_USAGE_STATS=1
Environment=PATH=/opt/vllm/venv/bin:/usr/local/cuda/bin:/usr/bin:/bin
UMask=0027
ExecStartPre=/usr/bin/test -s /etc/llm/api-key
ExecStart=/opt/vllm/venv/bin/vllm serve /opt/models/Qwen3-Embedding-0.6B \
  --served-model-name qwen3-embedding-0.6b qwen3-embedding \
  --runner pooling \
  --host 127.0.0.1 --port 8001 \
  --gpu-memory-utilization 0.08 \
  --max-model-len 4096 \
  --max-num-seqs 128 \
  --max-num-batched-tokens 8192 \
  --enforce-eager
Restart=on-failure
RestartSec=10
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/llm

[Install]
WantedBy=multi-user.target

• --max-model-len at least 4096. A job’s text lists up to 200 hosts and 200 targets, about 2,000 to 3,000 tokens of host names. At 1024, vLLM refuses those texts and a full Build Index stops with A text was longer than the embedding model accepts. Keep --max-num-batched-tokens at or above it.
• Dimensions: Qwen3-Embedding-0.6B returns 1,024, within the 2,048 limit. Verbiqo does not request fewer dimensions, so the template needs no is_matryoshka override.
• Its own account, and the server only. It runs as the llm service account (how to create it), and 127.0.0.1 means only a gateway on the same server can reach it. /etc/llm/embedding.env holds VLLM_API_KEY, which the gateway must send. Keep the model files readable by llm, caches under its writable folder, and test GPU access as that account. A gateway on a different host needs an authenticated TLS backend route; this loopback template is for the same host.

Before routing it, follow the service startup and authentication checks using unit llm-embedding, port 8001 and this model’s config.json. The memory reservation of 0.08 is an example to tune on your GPU, not a measured capacity guarantee.

2. Route it through LiteLLM. This entry assumes LiteLLM runs on the same host. Put the same backend key into the gateway’s protected VLLM_EMBEDDING_API_KEY environment variable; it is separate from the client keys LiteLLM issues. Add this to its model_list:

model_list:
  - model_name: qwen3-embedding
    litellm_params:
      model: openai/qwen3-embedding
      api_base: http://127.0.0.1:8001/v1
      api_key: os.environ/VLLM_EMBEDDING_API_KEY
    model_info:
      mode: embedding

Expose only the required authenticated API through the TLS gateway. If using the Caddy allowlist for a dedicated embedding service, use port 8001 and allow /v1/models and /v1/embeddings; keep other backend endpoints blocked. Allow the model on the keys people use. To reuse the answering model’s key, that key’s allowed models must include both. Check it from a laptop:

curl -s https://verbiqo-gw.example/v1/embeddings \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-embedding","input":["node cpu seconds total"],"encoding_format":"float"}' | head -c 200

3. Point Verbiqo at the gateway. In Extensions… › Prometheus › Semantic index:

For a whole organization, the endpoint, the model and Use the model’s API key can be set and locked centrally (embeddingEndpoint, embeddingModel, embeddingUsesModelKey in Deploy and manage). API keys are never set centrally. With Use the model’s API key on, each person enters their own LiteLLM key once in Settings, and it is used for both the AI model and the embedding model.

The gateway’s certificate is trusted the same way as for the answering model (Settings › Trusted CA, or the computer’s trust store). When a Keycloak-issued key expires, save the new one where it lives (in Settings, or in Embedding key). The index is kept, because the key is not part of what the vectors are stored under.

How much it embeds

• Volume: one text per metric (up to 20,000) and per job, host or target document (up to 10,000), 16 at a time. Later builds embed only what changed.
• Time: a build stops at one hour and the next one resumes. Most of the time goes to reading the server. A small local model keeps up.
• Disk: 4 bytes per dimension per text. With a 768-dimension model, about 3 KB a text, up to about 90 MB on the largest servers. With a 1,024-dimension model such as Qwen3-Embedding-0.6B, about 120 MB.
• While answering: each question embeds its own wording, so the service must be running when you ask. If it is not, the question still runs and finds metrics by name and prefix, without saying so. Only meaning-based matches are lost.
• Freshness: vectors are used for 7 days after the build that last saw them. Build again at least weekly. Unchanged texts are not sent again.

Changing the model

Vectors are kept per Prometheus connection and per embedding identity: the endpoint, the model name and VERBIQO_METRIC_EMBEDDING_REVISION. Changing any of them starts a new set, so the next build embeds everything again. The old set is left unused. Changing the key does not. When the model behind the same name changes (a newer tag pulled, different weights), set a new VERBIQO_METRIC_EMBEDDING_REVISION and build again. Without it, a model that now returns a different vector size stops the build with Embedding model dimensions changed. verbiqo prom index-clear <name> removes the vectors for a connection.

Only metric names, help text, types and units, and job, host and target identifiers are sent, never values or other label values. The desktop apps carry the vector library (sqlite-vec) and the bundled runtime. The standalone command line needs Node 22.13 or later for any of the index commands.

From a terminal

verbiqo prom index-build prod
verbiqo prom index-status prod
verbiqo prom index-refresh prod --jobs mysql,postgres
verbiqo prom index-discover prod
verbiqo prom index-clear prod

index-status says whether discovery is complete for every target. index-discover lists what the map holds and any coverage warnings. VERBIQO_PROMETHEUS_DISCOVERY_CACHE=off turns the ten-minute cache off. The map itself stays.

Access it needs

Verbiqo never calls the admin or lifecycle endpoints (/api/v1/admin/…, /-/reload, /-/quit).

Exporters that help

Verbiqo matches a database or cluster node to its machine by host name, or by the name in node_uname_info. When nothing matches, it says so.

Before you ask

An investigation runs many queries and several model steps, all within the app’s Timeout (ms). Set it to 300000, and Context window to 32768, or root-cause questions stop early or rest on a fraction of what was measured. Requirements and settings

From a terminal

verbiqo prom add prod --url https://prometheus.example.com --token-env PROM_TOKEN
verbiqo prom test prod
verbiqo prom list

A token on the command line (--token) still works but is visible to other processes and in shell history. --insecure is refused: Verbiqo never skips certificate checks. Trust the CA instead.

In the security review

The security review checks a Prometheus connection for anonymous access, by sending one request without credentials and expecting a refusal, and reads its TLS certificate: trusted issuer, expiry, host name, key strength, signature and protocol version.

Next: Ask Prometheus.

Common errors

A failed read is shown in the answer, with what remains unchecked. Find the message here.