Security and monitoring

The Overview shows what is wrong right now. This page covers the rest: the Security tabs, the checks that explain one object’s problem, and Monitoring.

Security has six tabs: Checks, Access, Certificates, Vulnerabilities, Images and Policies. Access, what a user, group or ServiceAccount may do, has a page of its own: see Access. Policies appears only when OPA Gatekeeper or Kyverno is installed.

Security › Checks

Checks (the health scan) tests how the cluster is set up against a fixed set of rules. What is broken right now, such as a crash loop, a pending pod or a node that isn’t ready, is the Overview’s job, so Checks doesn’t repeat it. It only reads; it never changes anything. The page opens with how many findings are critical and how many are warnings, and says what it checked, where and when.

It scans the namespace chosen in the top bar, or every namespace when none is. It reads as you. If you can’t list a kind across the cluster, it reads that kind one namespace at a time, in the namespaces you can list. If you can’t read pods anywhere it looks, it asks you to choose a namespace you can read. The result names the namespaces it skipped because you can’t list their pods, and the kinds you can’t read, such as nodes or a namespace’s Secrets.

Each finding has a severity (critical, warning or info), the object it is about, and what to do. The rules:

Area	Checks
Pods	Missing readiness or liveness probes (not for init containers, which Kubernetes doesn’t let have them, apart from sidecars), :latest or untagged images
Resources	Containers without CPU or memory requests or limits
Security	Containers that may run as root (neither runAsNonRoot nor a runAsUser other than 0, the container’s own over the pod’s securityContext), privileged containers, no security context, writable root filesystems, host networking
Workloads	Deployments with one replica or none, Deployments with several replicas and no PodDisruptionBudget
Networking	Services whose selector matches no pod and no Deployment’s pods
Storage	Released or failed PersistentVolumes
Configuration	ConfigMaps and Secrets nothing in their namespace uses: no pod or workload template mounts them, reads them into its environment or pulls images with them, and no Ingress serves TLS with them. Owned ones, kube-root-ca.crt and tool state such as Helm releases and tokens are left out
Cloud	Pods that call cloud APIs without a workload identity, use the default ServiceAccount for it, or use a deprecated IAM annotation

Findings are listed one row per rule, critical first, then warnings, then info, and within each the rules that flag the most objects. Critical rules start open. Open any other rule to see its objects, each linked to its page with the scanner’s message, and the remediation once. Filter by severity, by category (or press a category’s card), or by text, which also opens the rules it matches. Security only reruns the scan with just the security rules.

Security › Certificates

Certificates lists the TLS certificates in the cluster: every Secret of type kubernetes.io/tls, with the certificate in its tls.crt decoded, and, when cert-manager is installed, its Certificates. A Secret’s certificate is the leaf, the first in tls.crt that isn’t a CA’s, so a bundle that starts with its CA shows the server’s certificate and its expiry. When a certificate further up the chain in tls.crt expires before the leaf, the row names it and its date; the status stays the leaf’s, since a client may build another path to a root. A Secret with the same namespace and name as a cert-manager Certificate is one row, with the Certificate’s issuer and Ready condition added. A Certificate without such a Secret, for example one that hasn’t been issued yet, is listed from its own status.

Each certificate has a status:

Status	Means
Valid	More than 30 days left
Expiring soon	30 days or fewer left
Expired	Past its end date
Not yet valid	Before its start date
Parse error	The Secret holds no certificate KubeGlass can read, or a cert-manager Certificate has no dates yet and isn’t Ready

Expired certificates come first, then those that expire soonest. The cards count all certificates, the valid ones, those expiring within 30 days and the expired ones; with a status picked they count only the certificates with it, and say so. Filter by status, or by text, which matches the name, namespace, common name, issuer and DNS names. The columns are the certificate (its common name or first DNS name, with namespace and name), status, expiry date, days remaining (its bar turns amber at 30 days and red at 7; an expired certificate reads Expired in red, and one that expires within a day reads Today), issuer and source (secret or cert-manager). Open a row for the serial number, key algorithm and size, whether it is a CA, when it is valid from and until, the cert-manager issuer and Ready condition, and the SANs.

Certificates reads the namespace the top bar shows, or every namespace, as you. If you may not list Secrets across the cluster, it reads them in each namespace you may list them in, and names the others. cert-manager’s Certificates are read the same way, apart from the Secrets: where you may not read Secrets, as with the built-in view role, a Certificate is listed from its own status, with its dates and Ready condition but not the details only the certificate has, such as its key and serial number, and TLS Secrets cert-manager doesn’t manage are left out. The page says it can’t read the certificates only when you may list neither, and names the namespaces where you may list neither. GET /api/v1/certificates says so in skippedNamespaces, secretsRefused, certManager (ok, notInstalled, forbidden or unavailable) and certManagerSkippedNamespaces, the namespaces whose Certificates you may not list.

Security › Vulnerabilities

Vulnerabilities lists what the Trivy Operator found: the vulnerabilities in its VulnerabilityReports (aquasecurity.github.io/v1alpha1), one row for each vulnerability in each scanned image. When it has nothing to list, the page says why instead of showing zero: Not scanned, with a link to install it, when the Trivy Operator isn’t installed; “You can’t read the scan reports” when you may not list its VulnerabilityReports; “Couldn’t read the scan reports” when the API server didn’t answer; and “No scan reports yet” when the operator is installed but hasn’t written a report. Results appear after its first scan.

The worst come first: Critical, High, Medium, Low, then unknown, and within a severity the highest CVSS score. The cards count the findings (a CVE in each package and image it is in, so one CVE in three images is three findings), the Critical, High, Medium and Low ones, those of unknown severity when there are any, the fixable ones (those with a fixed version) and the unique CVE IDs. The severity cards add up to the findings. Filter by severity, with Fixable only, or by text, which matches the CVE ID, title, package, image, namespace and workload. The columns are the severity, the CVE and its title, the package with its installed version and the version that fixes it, the image, the namespace with the workload, and the CVSS score. An image that several workloads run has a row for each; on a phone, which shows the severity and the CVE, the namespace and workload are under the CVE. Open a row for the description, the workload and report it came from, and a link to the advisory.

The table shows the 500 most severe findings, and says how many there are when there are more. The text filter filters the table as you type; when the list is cut, it also looks through every finding on the server once you stop typing, so a CVE or an image past the first 500 is found. If that search fails, the page keeps the 500 findings and searches those, says so above the table, and asks the server again when you press Try again; the counts above the table cover every finding the severity and Fixable only filters pick, whatever the text, and say so while one of those is on. When those filters match nothing, the page says so, with Clear filters. Like Certificates, Vulnerabilities reads the namespace the top bar shows, or every namespace.

Security › Images

Images lists every image the pods run, init containers included, in the namespace chosen in the top bar or in all of them, the most used first. The cards count the unique images, the pods, the images tagged :latest or not tagged at all, the images not pinned by digest, and the registries.

The columns are the image with its registry and repository, the tag (amber for :latest or none), how many pods and containers use it, their namespaces, the pull policy most of them set, and whether the image is pinned by a @sha256: digest. The :latest only button shows just the images tagged :latest or not tagged. The text filter matches the image, registry and namespaces.

Security › Policies

Policies appears when OPA Gatekeeper or Kyverno is installed. It lists the violations they report:

• Gatekeeper: the violations in the status of every constraint, for every ConstraintTemplate. Gatekeeper has no severities; KubeGlass reads the constraint’s severity label (critical, high, medium, low or info), a convention some policy libraries follow, and a constraint without one shows No severity. The action is the constraint’s enforcementAction, deny by default. Gatekeeper’s audit keeps as many violations a constraint as its --constraint-violations-limit, 20 by default, and counts the rest in totalViolations: the cards count them all, and the page names the constraints it lists only some of. In a namespace, or in a cluster’s allowed namespaces, the cards count the listed ones, since Gatekeeper doesn’t say which namespaces the others are in.
• Kyverno: the results of its PolicyReports and ClusterPolicyReports that are fail, warn or error, with the policy and rule. The resource is the one the result names, or, in the reports Kyverno 1.10 and later write, one per resource, the report’s scope. The severity is the result’s own, and a result without one shows No severity. The action column shows fail as deny, warn as warn and error as audit.

Critical comes first, then high, medium, low and info. KubeGlass doesn’t make up a severity: a violation without one sorts by what its policy does, one that denies after the high ones, a warning after the medium ones and the rest after the low ones. The cards count the violations; the high, medium and low ones, the critical and info ones and those without a severity when there are any, which add up to the violations; and the policies with violations. Filter by engine (Gatekeeper or Kyverno), by severity, or by text, which matches the policy, rule, category and resource. The cards count what the engine and severity filters pick, and say so while one is on; when they match nothing, the page says “No violations match the current filters”, with Clear filters, rather than that every resource passes. With the engine filter on an engine that isn’t installed, it says so, with Clear filters. Open a row for the full message and the policy’s category.

“No policy violations” means the engines were read and report none. When KubeGlass may not read an engine’s reports, or the API server fails, the page says so instead, and lists what it left out above the violations it could read, with a line over the cards that says they count only what was read:

• Gatekeeper: the ConstraintTemplates, or the constraints of a template you may not list or the API server didn’t give.
• Kyverno: the PolicyReports. If you may not list them across the cluster, KubeGlass reads them in each namespace you may, and names the namespaces it couldn’t read. On All namespaces, a cluster limited to some namespaces under Settings › Clusters › Allowed namespaces reads those; without allowed namespaces, credentials that may not list namespaces read the kubeconfig context’s namespace. ClusterPolicyReports, the reports on cluster-wide objects such as namespaces, are listed apart.

GET /api/v1/policy/violations says the same in engines (ok, notInstalled, forbidden or unavailable for each engine asked about; an engine read in part is ok), skippedNamespaces, skippedClusterReports, skippedConstraints, unavailableConstraints and clusterReportsUnavailable.

Checks on an object

The Overview tab of a Pod, Deployment, StatefulSet, DaemonSet, Job, Service or CronJob has a Checks card that looks for the likely cause of a problem. It runs by itself when the object looks unwell (a pod with a problem, a workload with fewer ready replicas than it wants, a Job that failed or is retrying after a failed pod) and otherwise when you press Run checks.

• For a Pod: container states and exit codes, scheduling and lifecycle events, requests against limits, image pulls, PersistentVolumeClaim binding, the node’s conditions, restart history and the previous container’s logs.
• For a Deployment: its ReplicaSets, the state of its pods, rollout conditions, HorizontalPodAutoscalers that disagree with the replica count, and ResourceQuotas that block new pods.
• For a Service: whether its selector matches any pods, whether those pods are ready, whether the target ports match container ports, and NetworkPolicies that may block traffic.
• For a Job: whether it failed (with the reason, such as BackoffLimitExceeded), completed or is suspended, read from its conditions as kubectl describe does, its pod counts against the backoff limit (per index for an Indexed Job that has one), and the state of its pods, or that they were removed with their logs. A pod that failed before the Job completed was a retry, and doesn’t fail the check.
• For a StatefulSet or DaemonSet: ready pods against the number wanted, an update still rolling out as kubectl rollout status reads it (a partitioned StatefulSet is done once the pods above its partition are updated; with the OnDelete strategy there is no rollout, and it says how many pods still run an older template), and the state of its pods.
• For a CronJob: its schedule and whether it is suspended, from its spec; the Jobs it runs now and when it was last scheduled and last succeeded, from its status (kubectl describe prints the last schedule and the active Jobs); then how its newest Job ended, read from that Job’s conditions, and while that one hasn’t finished, how the newest one that ended did. It warns when it is suspended, when its newest Job failed or is retrying after failed pods, and when the newest one that ended failed. When the Job of the last run is gone (removed by the CronJob’s history limits, a ttlSecondsAfterFinished in its Job template, or by hand), it says how that run ended is unknown instead of judging an older Job.
• For every kind, after these: the object’s warning events. They expire after an hour by default, so they alone can pass an object that failed earlier.

Checks that pass fold away, and the results copy out as a Markdown report. On a pod’s page, the command palette offers Diagnose, which runs the checks.

CPU and memory in the lists

CPU and memory use are columns of the Pods and Nodes lists, like kubectl top: sort by them to rank pods or nodes. A pod’s value turns amber when it reaches 90% of its limit, where CPU gets throttled and memory gets the container killed; its tooltip gives the share of the limit and of the request. A pod’s limit counts its native sidecars (init containers with restartPolicy: Always), whose use is in the pod’s, or is the pod-level limit when the pod sets one; with a container or sidecar that has no limit, the pod has none. Its request counts them too when the numbers come from metrics-server, and leaves out a finished init container’s request and the pod’s overhead, as the usage does. Using more than the request is normal and stays plain. The numbers come from metrics-server (kubectl top nodes must work), or from Prometheus for people with cluster-wide read access. /resources/pods?sort=-cpu opens the Pods list highest CPU first. The pages of pods, Deployments, StatefulSets, DaemonSets and nodes chart usage over up to a day; see CPU and memory history.

Monitoring

Monitoring has four tabs: Restarts, Alerts, Rightsizing and PromQL. PromQL appears when KubeGlass can reach a Prometheus. Alerts and PromQL answer only people with cluster-wide read access, because Alertmanager and Prometheus see the whole cluster.

Restarts

Restarts lists workloads whose pods restarted in the last day or failed a health check in the last hour, even if they are running now. The data comes from what the cluster keeps, so the list is complete as soon as the page opens:

• Each container’s restart count and how it last stopped: out of memory, killed, an error with its exit code, and when.
• The kubelet’s probe failure events (reason Unhealthy), with how many times and the probe’s message. Kubernetes keeps events for about an hour.

The workload that restarts most often per day of its pods’ lives comes first. Open a row for each pod, with a link to its logs. Workloads whose current state is also a problem, such as a crash loop, are marked, since they are on the Overview as well.

When a node restarts, every container on it stops in the same minute. KubeGlass takes it for a node restart when pods of at least three workloads on the node stopped in the same minute, so replicas of one Deployment that crash together are still listed. Pods whose only restart was a node’s aren’t listed, and the page says how many it left out.

Alerts

When KubeGlass finds Alertmanager (or KUBEGLASS_ALERTMANAGER_URL is set), Alerts lists firing alerts and silences. Silenced alerts aren’t listed. You can create a silence from an alert, with its matchers filled in, or expire one. Silences are made as you: for an Alertmanager KubeGlass found, your RBAC needs create (to add) and delete (to expire) on services/proxy for its Service; with an Alertmanager URL in Settings, only KubeGlass admins can change silences. Each silence names you as who created it. Firing alerts of severity critical or warning also appear on the Overview, next to the problems of the objects they name.

KubeGlass reads one Alertmanager: the configured one, or the one in the cluster it started on. On another cluster, Alerts says there is no Alertmanager for that cluster and offers to switch to the one KubeGlass reads.

KubeGlass sends drift findings to a webhook or Slack when you configure one, under Settings › Server, or with the KUBEGLASS_ALERT_* variables. See Drift policies.

Rightsizing

Rightsizing compares each container’s usage with its requests and limits. See Rightsizing.

PromQL

PromQL runs queries against the cluster’s Prometheus and shows the result as a chart or a table, with a history of your queries and a library of presets. Each cluster can have its own, set by an admin under Settings › Clusters › Prometheus URL. Without one, for the cluster KubeGlass started on it uses KUBEGLASS_PROMETHEUS_URL or a Prometheus with node metrics it finds in the cluster. Prometheus is used only for people with cluster-wide read access.

Related

Access

Rightsizing

Overview and resources