Kubernetes checks

Everything Verbiqo checks on a Kubernetes cluster. The checks are grouped into areas, one section each below. You never type an area’s name: you ask a question, and Verbiqo runs the areas it needs.

How to read this page. Each section is one area. Why it helps says what you get from it. Ask is a question to type that runs it. The table under it lists what Verbiqo checks there: you do not run those one by one. When a check finds a problem, it appears under the answer as a finding, with its evidence and a suggested fix.

How areas are chosen

Ask in your own words
Attach the cluster with /k8s prod, then ask, for example why is checkout failing?
The model picks the areas
It reads the question and says which areas it needs, and what it is about (a namespace, a workload, a service). Verbiqo finds that name in the cluster and reads only what those areas need, there.
Always checked
Whatever you ask, what is wrong now is checked too, and problems elsewhere in the cluster are listed separately.

Area	Covers	A question that brings it in
What is wrong now	Crashes, out-of-memory kills, image pulls, pods that cannot start or be scheduled, stuck rollouts, nodes, jobs, services, storage	Why is checkout crashing?
Networking	Service ports and selectors, ingresses, DNS, network policies, the network plugin	The api pods can’t reach other services by name. Why?
Service mesh	Istio, Linkerd and Gateway API: routes, sidecars, mTLS, authorization. Only where a mesh is installed	Users get 503 from reviews through Istio. Why?
Room and cost	Free room on nodes, requests against real use, over-provisioned containers, quotas	Do we have room for another replica?
Reliability	What breaks in a drain, a node failure or a rollout: single replicas, missing probes, disruption budgets	What would break if a node went down?
Upgrades	Version support, deprecated APIs still in use, what would stall a drain	Are we ready to upgrade?
Security	Pod Security, credentials in manifests, cluster-admins, network exposure	How secure is this cluster?
What changed	Rollouts, restarts, out-of-memory kills, node changes and failed jobs, in order	What changed in the cluster at 3am?

Good to know:

A question can bring in several areas: checkout is slow since the upgrade reads what is wrong now, room and cost, and what changed.
If the model cannot tell (no usable answer, a timeout, an unreachable model), Verbiqo reads every area. It is slower, never worse.
A name that matches more than one thing (a checkout in two namespaces) is asked back, with the choices. Naming the namespace in the question settles it.
A name the cluster does not have is said, never guessed.
To read one area yourself, without a question, use a ready-made report.

What is wrong now

Why it helps: When something is failing now, it finds which pods, nodes or rollouts are broken and why, in plain words, so you know what to fix first.

Ask Why is checkout crashing?

Check	What it looks for
Crash loops	Containers restarting, with the exit code’s meaning, and whether the liveness probe is what kills them.
Out of memory	Containers killed at their memory limit, with the limit, and how to raise it.
Image pulls	Why: the image or tag does not exist, the registry refused, the node cannot reach it, or rate limiting.
Cannot start	A missing Secret or ConfigMap, named.
Cannot be scheduled	The scheduler’s reasons in plain words: not enough CPU or memory, taints, affinity, an unbound claim, cordoned nodes.
Not ready, evicted, stuck terminating	Readiness failures with the probe’s message, evictions with the node’s reason, finalizers holding a pod.
Rollouts	Rollouts past their deadline, StatefulSet updates stuck, DaemonSets missing nodes, a ReplicaSet refused by a quota; the revision before and the rollback command.
Nodes	NotReady or unreachable, memory, disk or PID pressure, cordoned for long, kubelets too far behind.
Jobs	Failed jobs and cron jobs whose recent runs fail or have not succeeded for a day.
Services and ingresses	Services with no ready endpoints (with pods carrying a similar label), load balancers never given an address, ingresses to services that do not exist.
Storage	Claims that cannot bind (a storage class that does not exist), lost volumes, released volumes still kept.
Autoscalers	Autoscalers that cannot read metrics, and those pinned at their maximum.
Warnings	Network plugin, volume attach, node OOM and reboot warnings, grouped and explained.
Credentials in logs	When logs are read: credentials written into them, masked, as a finding.

Networking

Why it helps: When services cannot reach each other, it finds the misconfigured Service, ingress, DNS or network policy, so you fix the cause rather than restart pods.

Ask The api pods can’t reach other services by name. Why?

Check	What it looks for
Service ports	A named target port the pods do not have (connections fail), a numbered one no container declares.
Service selectors	No pod matches (with pods carrying a similar label), or pods of several workloads match.
Ingresses	A class that does not exist, no class and no default, a backend port the Service lacks, a host and path claimed twice.
DNS	CoreDNS with no ready endpoint, or only one.
Network policies	Egress rules that block DNS (port 53), Services whose pods accept no traffic at all, policies that select no pod.
Network plugin and kube-proxy	Not ready on some nodes; nodes reporting their network unavailable; pods that could not get an IP address.

A packet dropped on a node, MTU, a cloud security group or conntrack are not visible in the cluster’s objects: Verbiqo says which command would settle it (kubectl debug with a network tools image, for example) for a person to run.

Service mesh

Why it helps: When traffic fails inside Istio or Linkerd, it finds the route, sidecar or policy at fault.

Ask Users get 503 from reviews through Istio. Why?

Read only where the cluster serves the mesh’s API, at the version it serves.

Mesh	What it looks for
Istio	istiod down; workloads in injected namespaces without a sidecar; sidecars not ready or older than istiod; DestinationRule subsets that match no pod (503 UH); VirtualService routes to hosts, subsets or gateways that do not exist; several VirtualServices for one host; STRICT mTLS with clients outside the mesh; DestinationRules that turn TLS off against STRICT; authorization policies that allow nothing (403); gateways with no gateway pods; Service ports whose protocol Istio must guess.
Linkerd	The control plane down, injection gaps, proxies not ready.
Gateway API	Gateways not accepted or programmed, routes not accepted or with backends that do not resolve, in the controller’s own words.
Sidecar access logs	When logs are on: `logs POD -c istio-proxy` comes back with status codes and Envoy response flags counted and explained (UH, NR, UF, UO, URX…), and the failing upstreams.

What only the proxies know (their pushed configuration, live endpoints) is left to istioctl analyze, istioctl proxy-config and linkerd check, which the answer names for a person to run.

Room and cost

Why it helps: Shows whether new pods will fit and where requests waste room, so you can resize before pods stay pending or get evicted, and stop paying for idle capacity.

Ask Do we have room for another replica?

Check	What it looks for
Cluster full	Schedulable nodes 85% or more requested, and the largest pod that still fits.
Nodes	Memory use near the eviction threshold; memory limits far beyond what the node has.
Over-provisioned	Containers asking for four times or more what they use, and how much that holds back.
Under-requested	Containers using more memory than they asked for: evicted first.
No requests	BestEffort workloads.
Quotas	Resource quotas 90% used or full.
Throttling and peaks	With a Prometheus connection: containers throttled for a quarter or more of the day, and those that peaked above 90% of their memory limit.

Reliability

Why it helps: Finds what breaks in a node failure, a drain or a rollout before it happens: single replicas, missing probes, disruption budgets.

Ask What would break if a node went down?

Application namespaces only: single replicas behind a Service, no readiness or liveness probe, :latest or untagged images, missing requests or memory limits, several replicas without a disruption budget, budgets that allow no disruption, every replica on one node, pods with no controller.

Upgrades

Why it helps: Shows what blocks an upgrade (deprecated APIs still in use, drain blockers, version support), so the upgrade does not break workloads.

Ask Are we ready to upgrade?

Whether the version still gets upstream patches (EKS, GKE and AKS set their own dates, which may be later), deprecated APIs still called and the release that removes them (from the API server’s /metrics), budgets that would stall a drain, and pods keeping data in emptyDir.

Security

Why it helps: Shows who can do too much, what is exposed and what breaks Pod Security, so you close the gaps an audit would find.

Ask How secure is this cluster?

Check	What it looks for
Anonymous access	A request without credentials (GET only); roles bound to unauthenticated users. Public-by-design bindings such as kubeadm’s `cluster-info` are not reported.
Pod Security Baseline	Privileged containers, host network, PID or IPC, hostPath, hostPort, added capabilities.
Pod Security Restricted	Running as root, privilege escalation, capabilities not dropped, no seccomp profile.
Credentials in manifests	Environment variables with a literal password, token or connection string, by name; values masked.
RBAC	Who holds cluster-admin (service accounts above all), roles made of wildcards, who can read Secrets, who can exec.
Namespaces	No Pod Security standard enforced; no NetworkPolicy.
Exposure	LoadBalancer services open to any address; workloads on the default service account with its token.
Verbiqo’s own account	Whether it could read Secrets.

The same checks are controls in the security review, mapped to CIS Controls, ISO 27001, SOC 2 and the CIS Kubernetes Benchmark, with the API server’s certificate checked as well.

What changed

Why it helps: Puts rollouts, restarts, kills and node changes in order, so you can match a problem to what changed just before it.

Ask What changed in the cluster at 3am?

Kubernetes keeps events for about an hour. The rest is read from the objects, which keep their own times, so the timeline reaches back days (see What happened earlier).

Entry	Read from
Rollout	Each ReplicaSet of a Deployment: namespace, name, revision and the images it carries.
Container stopped, OOM kill	Each container’s last termination: reason, exit code and restart count, with the node.
Node became NotReady or unreachable, became Ready	The node’s Ready condition and when it last changed.
Memory, disk or PID pressure	The node’s pressure conditions while true.
Node cordoned	The unschedulable taint and when it was added.
Job failed	The Job’s Failed condition and reason.
Warnings and notable events	Every Warning event, and Normal ones that mark a change (scaling, node registered or removed, rebooted, preempted, evicted, cluster autoscaler scale-up or scale-down), one line per object and reason at its latest occurrence, with the count.

What Verbiqo can see

Why it helps: Lists what Verbiqo can and cannot see on this cluster, with how to turn each on.

Ask What should I enable?

Ask what should I enable? for each item as ready, partly set up or not set up, with the fix. When a question needs an item that is missing, the answer says so under To see more on this cluster.

Item	Lets Verbiqo see
A read-only account	Everything: Verbiqo connects only with an account that cannot change the cluster. Missing when the account can exec, create, delete, scale or bind.
Reading across the cluster	Nodes, every namespace and cluster-wide roles: node pressure, headroom, who is cluster-admin. Partial when the connection is limited to some namespaces.
metrics-server	Live CPU and memory per pod and node: requests against real use, over-provisioning, nodes near their limits.
Pod logs	The last lines of a crashing container, masked as they are read. Missing when the connection was added without `--logs`; partial when it was but the role lacks `pods/log`.

The role and account themselves are in Set up Kubernetes.

Ready-made reports

A report is one fixed reading of one area: no question, no model, the same output every time. Use one to look at an area yourself, or to share what it shows.

From a terminal, it prints the report:

verbiqo k8s run prod problems

In verbiqo chat, it loads the report into the conversation so you can ask about it. help lists the reports:

/k8s prod problems
/k8s prod help

In the desktop apps, there is no report command: ask in your own words. The model can also run any of these reports itself when it needs one.

Report	Shows
`overview`	The cluster at a glance: nodes, workloads, pods by status, and what is wrong now.
`problems`	What is wrong now: crashes, OOM kills, image pulls, pending pods, stuck rollouts, nodes, jobs, services, storage.
`pods`	Pods that are not running and ready, with restarts.
`restarts`	Containers that restart most, with why they last stopped.
`workloads`	Deployments, StatefulSets and DaemonSets, with ready replicas.
`nodes`	Nodes: status, roles, version, pressure, and live use.
`events`	Warning events of the last hour, newest first.
`changes`	What changed, in order: rollouts, restarts, OOM kills, node changes, failed jobs, warnings.
`capacity`	Room on each node: requested and used CPU and memory, quotas, and over- and under-requested containers.
`top-pods`	The pods using the most CPU and memory now, against their requests.
`reliability`	What will hurt in a drain, a node failure or a rollout: single replicas, missing probes, `:latest`, PDBs, spread.
`upgrade`	Upgrade readiness: version support, kubelets behind, deprecated APIs still called, drain blockers.
`security`	Security: Pod Security breaches, credentials in manifests, cluster-admins, network policies, exposure.
`rbac`	Who holds which roles: cluster role bindings and role bindings.
`storage`	Persistent volume claims and volumes, with their status.
`network`	Networking: Services and their endpoints and ports, ingresses, DNS, network policies, the network plugin, IP exhaustion.
`mesh`	Service mesh (Istio, Linkerd) and Gateway API: control plane, injection, sidecars, routes, subsets, mTLS, authorization.
`jobs`	Jobs and cron jobs: last runs, failures, suspended.
`autoscaling`	Horizontal pod autoscalers: targets, replicas, and whether they can scale.
`quotas`	Resource quotas: used against hard limits.