Set up Splunk

Verbiqo searches Splunk through its REST API with searches it builds and checks itself, and reads only the indexes your role allows.

At a glance

Supported tasks	Find failures and known events in logs per host, show what a system logged, count and rank events, and list what the deployment holds. Ask Splunk and Log findings.
Limitations	Verbiqo sees only the indexes the account’s role allows. An index it cannot read returns no events and no error, so it looks empty. One search covers at most 7 days.
Supported versions	Splunk Enterprise 7.3 and later. Verbiqo signs in through the REST API, reads the server’s details, and runs one-shot searches using `tstats`, `metadata`, `eventcount` and `fieldsummary`, all available from 7.3. Token sign-in needs 7.3 too. Used on a distributed Splunk Enterprise deployment, whose version was not recorded. Splunk Cloud is not tested, and needs its management port (8089) open to you.
Authentication	A Splunk token, or a directory login (user and password). A password is exchanged for a session key, and only the key is kept. SAML single sign-on accounts cannot use the REST API: use a token.
Minimum permissions	The `search` capability, your log indexes in `srchIndexesAllowed`, and read on the app it searches in (default `search`). Recommended: `srchTimeWin` of a week or more, and `srchJobsQuota` of at least 2. Details: The role it needs.
Connection test	Use the management port, usually 8089. Press Test Connection in Extensions… › Splunk, or run `verbiqo splunk test <name> --indexes`. Success shows Connected., the server name, the Splunk version and the account’s roles. It also shows how many indexes the credential can read.
First useful question	Attach with `/splunk <name>`, then ask any problems in index shop in the last 15 minutes. Expect known events per host (crashes, failovers, 5xx and more) with a timeline. The answer shows every search it ran and the exact period, in your zone and in UTC.
Safety controls	Verbiqo writes every search itself. Each command must be on a read-only allowlist. An unknown command is refused. Write commands are refused by name, including `collect`, `outputlookup`, `delete`, `sendemail` and `script`. Subsearches (a second search inside brackets) are refused. Every search names its indexes, or is pinned to a host, source or sourcetype. A search across every index may only count and group. Every search is bounded: at most 7 days and 10,000 events, with a time limit (120 seconds by default).
Operational limits	One search at a time. Each search: at most 7 days, a 120-second limit (settable from 1 to 600 seconds) and 10,000 events. With no period in the question, the last 24 hours are searched. Your organization can lower these under `splunk`: Protect your systems.

Connect

Name and address
Extensions… › Splunk. The URL on the management port, usually 8089: https://splunk.example.com:8089. The web port 8000 will not work.
Sign in
Directory login (user and password) or a Splunk token. SAML single sign-on accounts cannot use the REST API: use a token.
Save and test
With a directory login, the password is kept only once Splunk accepts it.

A private CA for the management port: name it in Settings › Trusted CA (below Endpoint). A terminal on the same computer trusts it too, with nothing to set. If the test says the account can read no indexes, the role needs search access. The deployment is not empty.

From a terminal

With a token, piped in or read from an environment variable, never on the command line:

verbiqo splunk add prod --url https://splunk.example.com:8089 --token-env SPLUNK_TOKEN
verbiqo splunk test prod --indexes

With a directory account, the password is exchanged once for the session key Splunk issues, and only the key is kept:

verbiqo splunk add prod --url https://splunk.example.com:8089 --username you@corp
verbiqo splunk login prod
verbiqo splunk test prod --indexes

Several search heads behind one address (a load balancer): a session key works only on the head that issued it, so searches are refused part-way. Use a Splunk token, or a directory login whose password is saved: the app then sends it with each request. In verbiqo chat, type the password when asked.

Flag	Means
`--token-stdin`, `--token-env VAR`	The Splunk token, piped in or from an environment variable. `--clear-token` removes it. Left off in a terminal, `add` prompts without echo.
`--username <you>`	Log in with a directory account instead of a token. Then `login`, which takes `--password-stdin` or `--password-env VAR`, or prompts.
`--app search`, `--owner nobody`	The app and owner namespace searches run in.
`--search-timeout 120`	Seconds one search may take, 1 to 600. Left off, the default applies.
`test --indexes`	Also lists what the credential can read, in the same session. A credential accepted by `test` alone is known to Splunk. It may still read no index.
`test --url <url> …`	Tests these details without saving them.

The role it needs

Setting	Why
`search` capability	Running searches, including `tstats`, `metadata` and `eventcount`	Required
Your log indexes in `srchIndexesAllowed`	Verbiqo sees only these, and says when a question needs another	Required
Read on the app it searches in (default `search`)	Search jobs in that namespace	Required
`srchTimeWin` of a week or more	A shorter window cuts searches off	Recommended
`srchJobsQuota` of at least 2	Verbiqo runs one search at a time, beside yours	Recommended

An index the role cannot read returns no events and no error, so it looks empty. verbiqo splunk indexes prod lists what the credential can actually read. For security questions, srchIndexesAllowed must cover the indexes that hold the Windows Security log, Linux authentication logs and, with Enterprise Security, the notable events (notable). Otherwise those hunts find nothing.

The logs it reads

Verbiqo recognises a log by its sourcetype and by its lines, whatever the sourcetype is called, and counts known events per host for the answer and the timeline.

Log	What to send, and where the findings are listed
PostgreSQL	The server log, with its host. Settings: PostgreSQL logs in Splunk. Findings: Log findings.
MySQL	The error log and the slow log, one event per Splunk event. Settings: MySQL logs in Splunk. Findings: Log findings.
SQL Server	The ERRORLOG file, or the Windows Application event log. Settings: SQL Server. Findings: Log findings.
Redis	The Redis log and Sentinel’s. Settings: Redis. Findings: Log findings.
Elasticsearch	The server log and the search and indexing slow logs. Settings: Elasticsearch. Findings: Log findings.
Security events	Windows Security and System logs, Linux authentication logs (auth.log, secure) and Enterprise Security notable events: Security logs.
Credentials in logs	The logs a question reaches, searched for passwords, keys and tokens, each masked as it is read: Credentials written into logs.

In the security review

The security review reads nothing of Splunk’s own settings. It checks the management port’s TLS certificate: trusted issuer, expiry, host name, key strength, signature and protocol version.

Next: Ask Splunk.

Common errors

The test or the answer shows the message. Find it here.

Message or symptom	Likely cause	What to do	Needs an administrator?
Port 8000 is the Splunk web interface. Use the management port, usually 8089.	The URL points at the web interface.	Use the management port: `https://splunk.example.com:8089`.	No.
Splunk returned a web page rather than API data.	The URL reaches the web interface or a proxy, not the management port.	Use the management port, usually 8089.	Only if the port is blocked to you.
Splunk rejected the password for …. Nothing was read.	A wrong password, or a SAML single sign-on account, which cannot use the REST API.	Check the password. For a SAML account, use a Splunk token instead.	Possibly, to issue a token.
Splunk rejected the token. It may have expired, or belong to a different deployment.	The token expired, was revoked, or belongs to another Splunk.	Save a new token for this connection, then test again.	Possibly, to issue a token.
Splunk refused the session key for …, which is not old enough to have expired.	The address fronts several search heads, and a session key works only on the one that issued it.	Use a Splunk token, or save the directory login’s password with the connection. In `verbiqo chat`, type the password when asked.	No.
This credential can read no indexes. That is a permission, not an empty deployment.	The role has no search access to any index.	Give the role the `search` capability and the log indexes in `srchIndexesAllowed`.	Yes.
This token is accepted but not permitted to do that. Or an index returns nothing.	The role lacks the `search` capability, or cannot see that index.	Run `verbiqo splunk indexes <name>` to see what it can read. Add the missing index to `srchIndexesAllowed`.	Yes.
… assumed, the question named no period	The question had no period Verbiqo understands, so the last 24 hours were searched.	Add one: in the last 2 hours, since 9am, yesterday. A time needs am, pm or minutes.	No.