Docs / Connections: Splunk

Log findings

The checks page for Splunk: what Verbiqo recognises in the logs it reads through a Splunk connection. With Splunk attached (Ask Splunk), Verbiqo searches the database and cluster logs for known events, counts them per host, and adds them to the answer and the timeline. Each finding shows its first and last line.

How to read this page. Each section is one area. Why it helps says what you get from it. Ask is a question to type that runs it. The table under it lists what Verbiqo checks there: you do not run those one by one. When a check finds a problem, it appears under the answer as a finding, with its evidence and a suggested fix.

How to ask

Name the system and the time: any problems on the orders cluster in the last 24 hours?, what did MySQL log on db01 last night?, slow searches on the logging cluster yesterday. Attach the database or cluster too, and Verbiqo joins what it logged with what it shows now.

Verbiqo recognises the logs by their Splunk sourcetype and by their content. The settings that make these events appear in the logs are in PostgreSQL logs in Splunk and MySQL logs in Splunk; for SQL Server and Redis, SQL Server and Redis.

PostgreSQL log

Why it helps: Finds errors, slow statements, lock waits and restarts in the PostgreSQL log, with who logged them, so you see what happened even after it is over.

Ask Any problems on the orders cluster in the last 24 hours?

Who failed, and on which node. Each finding names the roles in its lines and says what each one is:

• a monitoring tool, recognised by its application or role name (for example SolarWinds DPA, pganalyze, Datadog, pgwatch, PMM, Zabbix);
• an application, when its lines show it writing or running statements on its own tables.

When every line of a finding comes from a monitoring tool, the finding says so: the tool’s own queries failed, and applications were not affected.

“recovery is in progress” is how a standby refuses a write, so Verbiqo names those hosts as standbys rather than calling the cluster down. A host whose lines show writes being checked against constraints, or sessions waiting on rows, is named as the primary.

The answer is held to both: a sentence saying the cluster is in recovery, or that a tool’s errors are application failures, is corrected or removed.

MySQL error and slow logs

Why it helps: Finds errors, crashes, replication breaks and slow statements in MySQL’s logs.

Ask What did MySQL log on db01 last night?

Elasticsearch server and slow logs

Why it helps: Finds node failures, memory pressure, rejected work and slow searches in Elasticsearch’s logs.

Ask Slow searches on the logging cluster yesterday

SQL Server error log

Why it helps: Finds errors, failed logins, deadlocks and I/O warnings in the SQL Server error log.

Ask Any errors on the sales server last night?

From the ERRORLOG file, or the Windows Application event log where the same messages arrive with SourceName=MSSQLSERVER.

Redis and Sentinel logs

Why it helps: Finds failovers, persistence failures and memory warnings in Redis and Sentinel logs.

Ask What happened to Redis in the last hour?

Kubernetes events and component logs

Why it helps: Finds crashes, scheduling failures and component errors in cluster events and logs, even after the pods are gone.

Ask Any problems in the shop namespace yesterday?

Recognised by their content (event fields such as reason and involvedObject, or the kubelet's log format), whatever the sourcetype is called. That covers the Splunk Connect for Kubernetes and OpenTelemetry collectors' event sourcetypes, and the kubelet's journal.

Application and service logs

Why it helps: Finds exceptions, timeouts and connection failures in your services’ own logs, grouped by kind.

Ask Why did checkout throw errors at 2pm?

Every other log is read for the failures applications have in common, in the words their runtimes, libraries and proxies use: Java, Scala and Akka, Python (with gunicorn, uWSGI and Celery), Node, Go, Rust, Ruby (with Rails, Puma and Unicorn), Elixir and Erlang, C and C++, .NET and PHP; nginx, Apache, Tomcat, HAProxy, Varnish and Envoy; the database, Oracle, MongoDB, Cassandra, Elasticsearch, Kafka, RabbitMQ, Redis, gRPC and cloud SDK clients, and the MongoDB, Kafka, RabbitMQ, ZooKeeper and Cassandra servers' own logs; and the Linux kernel. PHP-FPM logs are read here, not as PostgreSQL logs. In each question the two busiest unrecognised logs are read this way. Logs that carry no error level at all are searched by the model for what a failure is there, such as a status code (Ask Splunk).

MongoDB, Kafka, RabbitMQ, ZooKeeper, Cassandra, etcd and PgBouncer

Why it helps: Finds the failures each of these servers logs, in its own wording.

Ask Any problems on the Kafka brokers today?

Each server's own log, recognised by the line format it writes, under any sourcetype name. Their clients' errors (an application that cannot reach them) are in the application checks below.

Access logs

Why it helps: Finds server errors and spikes in web and proxy access logs, by status and path.

Ask Any 5xx errors on the web tier this morning?

Any log whose lines carry an HTTP status: nginx, Apache, IIS, Envoy, Traefik, HAProxy, ingresses and cloud load balancers alike. Recognised by the status itself, as a field (status, sc_status, elb_status_code and similar) or in the line (Common and Combined Log Format, HAProxy's HTTP log, JSON). Counted over the whole window, beside whatever else reads the log.

What Verbiqo does not recognise yet

Why it helps: Says plainly when a log’s format is not one Verbiqo knows, so silence is never mistaken for health.

Ask what errors don't you recognise yet? (or …in index apps). Verbiqo lists the error lines it has read on this connection that no check above recognises, the most frequent first: your own applications' error codes, messages from software not covered here. It answers from what earlier questions read, without searching. These lines are still found as errors by their level; they are just not named or explained.

Operating-system logs

Why it helps: Finds out-of-memory kills, disk errors and service failures in system logs.

Ask Did anything get OOM-killed on db01 last night?

Syslog and the systemd journal, recognised by their line formats under any sourcetype name. They also carry what applications write through them, so they are read for the application failures above too.

Patroni and pgBackRest

Why it helps: Finds failovers, leader changes and backup failures in Patroni and pgBackRest logs.

Ask Did the cluster fail over yesterday?

The tools around a PostgreSQL cluster, recognised by their own line formats.

Security events

Why it helps: Finds failed logins, privilege changes and suspicious access in security events.

Ask Any failed logins on the database servers today?

Windows Security, Linux authentication and SIEM alerts, with failed-then-successful logons, password spraying and cleared audit logs worked out for you: Security logs.