Docs / Connections: Splunk

Ask Splunk

Attach a connection and ask in plain words. Verbiqo writes every search itself, runs it read-only, shows each one it ran, and answers only from what came back. A question that says what you want, where to look and when gets the fastest and most precise answer.

Attach

/splunk prod

Use the name you gave the connection. /splunk off stops searching it and leaves other systems attached. /sources shows what is attached. A question only ever reaches the connections attached in this conversation.

The shape of a good question

<what you want>  <where: index, host, application>  <when>

any problems with orders-api in index shop in the last hour
top 10 hosts by events in index apps in the last 15 minutes
what changed on db01 since 1 Oct

Each part you leave out is something Verbiqo has to work out by reading the deployment first, or by asking you. On a large deployment that is the difference between seconds and minutes.

What you want: the words that choose the kind of answer

The first words of the question decide what Verbiqo does. Use the ones that match what you need.

Where to look: the words that narrow the search

Every answer marks the kinds of line Verbiqo has not seen in earlier reads of that log as new, so a log asked about before shows what started since.

When nothing matches the name, Verbiqo reads what the index holds and offers the closest entries as a numbered menu (see Menus).

When: the periods Verbiqo understands

The period is read by code, not by the model, on your computer’s clock and in its time zone. The answer always states the exact period it searched, in your zone and in UTC.

• Another time zone: name it in the question, as since 9am UTC, at 03:00 EDT, since 10:00 UTC+2 or at 9am Europe/London. To read every question in one zone, set VERBIQO_TIME_ZONE (for example UTC).
• A time needs am, pm or minutes: 9am and 09:00 are times; between 2 and 4 is not, and falls back to the last 24 hours.
• Nothing said: searches cover the last 24 hours. Listings, and the index reads behind a question, cover the last 7 days and are remembered for 12 hours.
• Longest period: 7 days per search. A longer one is shortened to its last 7 days, and the answer says so.
• Logs stamped in the wrong zone: when Splunk indexed a log’s lines hours away from the time it gave them, the answer says so. Its times are then off by those hours, in Splunk’s own search too. The fix is a TZ setting for that sourcetype in Splunk’s props.conf.

Saying a period also keeps listings cheap. Without one, listing the sources of a busy index reads 7 days of its index files, which can take minutes:

list sources in index apps in the last hour
what hosts are in index shop in the last 30 minutes
what sourcetypes are in index shop in the last hour

A listing for a stated period counts exactly that period, and is not remembered, so it never stands in for the full 7 days.

Examples

any problems in index shop in the last 15 minutes
any errors on db01 since 9am
is the orders cluster healthy today
what happened on postgresql-payments yesterday
what did orders-api log between 2 and 3pm
what happened on postgresql-payments at 3am
why did the backup on postgresql-backup fail yesterday
find every "connection reset" in index apps in the last hour
how many status 5xx per hour in index web in the last 12 hours
top 10 hosts by events in index apps in the last 15 minutes
average duration by service in index shop in the last hour
show the last 20 lines from orders-api in index shop
compare web-1 and web-3 by status in the last hour
list all the indexes
list applications in index apps in the last hour
what errors don't you recognise yet

Menus and follow-ups

When the question names an index but nothing narrower, or a name the deployment does not hold, Verbiqo lists the likeliest entries and, with an index named, everything in that index. Nothing is read until you choose. Reply with:

• a number, or several: 2, 2 and 5
• a name or part of one: event.log, the report-worker one
• all of them, or the last option for everything in the index
• your own words: the api logs, not the listener

The same question then runs on your choice, over the same period. A reply that asks something else is taken as a new question.

Follow-ups build on the last question. And how many per hour?, same for catalog or what about yesterday? keep its index, subject and period unless you name new ones. Naming something narrower, such as a file, a host or a service, asks the same question about that, inside the same index.

What makes an answer slow or vague

Recognised events are matched on the English wording each server and runtime writes. A log written in another language is still read for its error levels, but its events are not named.

With other sources, on one timeline

/splunk prod
/pg orders
/prom prod

Attach the database or cluster too, and Verbiqo joins what it logged with what it shows now. Ask what happened yesterday? and the answer carries a timeline from every source: restarts, failovers, backups and log events from Splunk, the machine’s reboots and history from Prometheus, each with its source. It places events in time; it does not say one caused another. What happened earlier

From a terminal

In verbiqo chat, attach with /splunk as in the app, or name the connection when you start:

verbiqo chat --splunk prod "any problems in index shop in the last hour"

The verbiqo splunk commands below need no chat session.

verbiqo splunk ask prod "what did mysql log on db01 yesterday"
verbiqo splunk locate prod postgresql-billing --sourcetypes
verbiqo splunk shapes prod --index main --sourcetype postgresql-backup
verbiqo splunk indexes prod

Behind a load balancer with several search heads, a saved session works only on the head that issued it. Type your password when verbiqo chat asks, or use a Splunk token. Connection commands (add, login, test) are on Set up Splunk.