Operations

How to deploy, monitor, back up and upgrade KubeGlass in a cluster. The commands assume the Helm chart installed as release kubeglass in namespace kubeglass, so the Deployment, Service and ConfigMap are all called kubeglass. Adjust the names if yours differ.

The configuration reference lists every setting, and the chart’s values.yaml lists every chart option.

Deployment

Helm install

The chart is published as an OCI artifact at oci://ghcr.io/kubeglass/charts/kubeglass; there is no helm repo add step.

Behind an authenticating reverse proxy (the default impersonation mode), set the addresses KubeGlass believes X-Forwarded-User and X-Forwarded-Groups from, and who may reach it. The install fails without either:

helm install kubeglass oci://ghcr.io/kubeglass/charts/kubeglass \
  --namespace kubeglass \
  --create-namespace \
  --set config.trustedProxies=10.244.0.0/16 \
  --set-json 'networkPolicy.ingressFrom=[{"namespaceSelector":{"matchLabels":{"kubernetes.io/metadata.name":"ingress-nginx"}}}]'

10.244.0.0/16 stands for the range the proxy’s pods get addresses from, here the ingress controller’s in the ingress-nginx namespace; where the proxy has a fixed address, name only that. Every pod in the cluster has an address in the pod range, so the range alone would let any pod send the headers and act as anyone: networkPolicy.ingressFrom keeps all but the proxy out. The proxy must also replace any X-Forwarded-User and X-Forwarded-Groups a client sends. An install made before the chart asked for ingressFrom upgrades without it, and its install notes warn until you set it.

To have KubeGlass sign people in with your identity provider instead, use oidc mode. Create the Secret with the client secret first (kubectl -n kubeglass create secret generic kubeglass-oidc --from-literal=client-secret=<secret>), then save this as values-oidc.yaml. Sign-in has the settings for Dex, Keycloak, Okta, Entra ID and Google.

config:
  authMode: oidc
  # Stricter checks at start: the issuer and redirect URL must use HTTPS
  environment: production
  # The ingress controller's addresses, or the range its pods get them from
  trustedProxies: <ingress controller CIDR>
oidc:
  issuerURL: https://accounts.google.com
  clientID: <client-id>
  clientSecretRef:
    name: kubeglass-oidc
  redirectURL: https://kubeglass.example.com/auth/callback

helm install kubeglass oci://ghcr.io/kubeglass/charts/kubeglass \
  --namespace kubeglass \
  --create-namespace \
  -f values-oidc.yaml

trustedProxies lets KubeGlass take each person’s address from the ingress controller’s X-Forwarded-For, for its rate limits, and see the HTTPS the Ingress ends; without it everyone shares one limit. In oidc and token modes it doesn’t make KubeGlass believe identity headers.

To try it without a proxy or identity provider, use --set config.authMode=local and reach it with kubectl -n kubeglass port-forward svc/kubeglass 8090:8090. In local mode anyone who reaches KubeGlass gets its own access: read access, Secrets included, a shell in any pod, and creating and expiring Alertmanager silences. The NetworkPolicy then lets in only the pod helm test runs and what networkPolicy.ingressFrom names, so don’t expose it any other way. With ingress.enabled, or service.type LoadBalancer or NodePort, and ingressFrom empty, it lets every source in, and the install notes warn and show an ingressFrom with the ingress controller’s namespace or an ipBlock for clients outside the cluster. With networkPolicy.enabled false they warn in the same way, also when the chart doesn’t expose KubeGlass, since every pod in the cluster can then reach it, and say that the policy, once on, limits KubeGlass’s own connections to DNS and ports 443 and 6443, so a Prometheus or Alertmanager on another port needs a rule in networkPolicy.additionalEgress. With an ipBlock, set service.externalTrafficPolicy to Local, so that the NetworkPolicy sees the clients’ addresses rather than a node’s. A node that runs no KubeGlass pod then drops connections to its node port, so NodePort clients have to use a node a KubeGlass pod runs on, whose address changes when the pod moves. A load balancer that proxies connections, such as AWS’s Classic ELB, shows its own addresses whatever the policy, so there the ipBlock has to name those (see Using Source IP). Only in this mode does the chart’s ClusterRole have pods/exec; in the others shells and file transfers run as the signed-in user.

deploy/helm/kubeglass/values-production.yaml is a starting point for a production install: ingress with TLS, larger resources, a scoped impersonation rule and a NetworkPolicy that admits only the ingress controller.

To add actions, hotkeys, sidebar links, panels, list columns or palette aliases for everyone, put them under extensions: in your values; see Extensions.

Serving under a path

When KubeGlass shares a host with other tools behind one ingress, give it a path of its own with config.basePath:

config:
  basePath: /kubeglass
ingress:
  enabled: true
  hosts:
    - host: tools.example.com
      paths:
        - path: /
          pathType: Prefix

KubeGlass is then at https://tools.example.com/kubeglass/. The chart turns an Ingress path of / into /kubeglass and leaves other paths as you wrote them. Outside the chart, set KUBEGLASS_BASE_PATH=/kubeglass or start it with --base-path /kubeglass. The image is the same for every path: the server writes the path into the page as it sends it.

• The ingress or proxy must pass the path on unchanged. Don’t strip or rewrite it (no nginx.ingress.kubernetes.io/rewrite-target, no Traefik StripPrefix middleware).
• Everything moves under the path: the UI, the API at /kubeglass/api/, the WebSockets for logs and shells at /kubeglass/ws/, and sign-in at /kubeglass/auth/. In oidc mode register https://tools.example.com/kubeglass/auth/callback as the redirect URI and https://tools.example.com/kubeglass/sign-in as the post-logout redirect URI; the chart works the first one out from the Ingress host.
• KubeGlass’s cookies are for the whole host, under a path too, so the session cookie reaches every tool on tools.example.com. A path doesn’t keep cookies apart: browsers send a cookie to longer paths as well, and keep two cookies of one name when their paths differ, sending both (RFC 6265, section 8.5). A host-wide cookie can have the __Host- prefix on HTTPS, which no other host can set, so a site on a sibling subdomain (evil.example.com next to tools.example.com) can’t plant a session cookie that KubeGlass would read. By filling the browser’s cookie jar for the domain it can still push KubeGlass’s cookie out, which signs you out and does nothing else. Under a path the cookie names end with a short hash of the path, so a KubeGlass under /kubeglass, one under /kubeglass/staging and one at the root of the same host each keep their own sign-in.
• Over plain HTTP no cookie prefix applies, and another site under the domain can set a cookie with KubeGlass’s name. A request that brings two session cookies is refused, and KubeGlass trusts neither. The page then says to delete the site’s cookies; signing in again wouldn’t help while the other cookie stays.
• Every page on the same origin can call /kubeglass/api/ with your cookies. A cross-site scripting hole in another tool on tools.example.com reaches your clusters as you. When you can, give KubeGlass a host name of its own (kubeglass.example.com) instead of a path.
• /healthz, /livez and /readyz answer at the root of the pod and under the path, so probes and kubeglass --healthcheck need no change. /metrics answers only at the root of the pod, where the ServiceMonitor scrapes it; under the path it gets a 404, so the ingress, which routes only the path, doesn’t reach it.
• At the root of the pod, / redirects to /kubeglass/, which helps with kubectl port-forward. Any other address outside the path gets a 404 that says where KubeGlass is.
• logout_url is used as written, for a sign-in proxy that signs out at the host’s root, such as oauth2-proxy’s /oauth2/sign_out.
• HSTS (Strict-Transport-Security) leaves out includeSubDomains under a path, since it would bind every subdomain of the host on behalf of the other tools. Set that at the ingress if the host’s owner wants it.

Proxies and load balancers

Apart from file transfers, no API request takes longer than 55 seconds, so the 60 seconds that ingress-nginx waits for an answer by default (proxy-read-timeout) and an AWS load balancer’s default idle timeout are enough for the API. Uploading a file to a pod or downloading one from it takes up to 15 minutes; the bytes keep flowing, so an idle timeout doesn’t cut it off, but a proxy that limits a request’s whole time does. Helm installs, upgrades, rollbacks and uninstalls run in the background on the server: the request answers at once, and the page asks how the operation is going every second, so an upgrade that waits five minutes for its pods isn’t cut off. The requests that take longest download a chart repository’s index or a chart (a chart’s versions, values and README, a dry run, adding or updating a repository), check many clusters at once (importing kubeconfigs, connecting through SSO) or scan clusters for drift; each stops in time to answer with what it has. WebSockets for logs, shells and live updates stay open much longer; see WebSocket disconnections.

RBAC for impersonation

Outside local mode KubeGlass makes every Kubernetes call as the signed-in user, so its ServiceAccount needs the impersonate verb. The chart’s ClusterRole grants it on users, groups and serviceaccounts by default (rbac.impersonation.enabled), and leaves the rule out in local mode. Limit whom KubeGlass may impersonate with rbac.impersonation.resourceNames:

rbac:
  impersonation:
    resourceNames:
      - alice@example.com
      - platform-team

rbac.readSecrets (default true) controls whether the ClusterRole can read Secrets.

If you manage RBAC yourself (rbac.create=false), the rule the chart creates is:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kubeglass-impersonator
rules:
  - apiGroups: [""]
    resources: ["users", "groups", "serviceaccounts"]
    verbs: ["impersonate"]
    # Add resourceNames to restrict which users and groups can be impersonated.
  - apiGroups: ["authentication.k8s.io"]
    resources: ["userextras/scopes", "userextras/uid"]
    verbs: ["impersonate"]

Production checklist

• config.authMode is oidc, token or impersonation, not local.
• In oidc and token modes, session.keySecretRef names a Secret with the session key, and the data directory has a PersistentVolumeClaim, so people stay signed in when the pod restarts.
• In impersonation mode, config.trustedProxies names only your proxy’s addresses, networkPolicy.ingressFrom lets only the proxy reach the pod, and the proxy replaces any X-Forwarded-User and X-Forwarded-Groups a client sends.
• In oidc and token modes behind an ingress or load balancer, config.trustedProxies names it, so each person has their own rate limit and KubeGlass sends HSTS.
• config.environment is production (KUBEGLASS_ENV), as values-production.yaml sets it. This turns on stricter checks: the OIDC issuer and redirect URL must use HTTPS. API errors then leave out internal details.
• TLS is terminated at the ingress or load balancer (or KUBEGLASS_TLS_ENABLED=true with a mounted certificate).
• KUBEGLASS_CORS_ALLOWED_ORIGINS, if set, lists specific origins. KubeGlass refuses to start with *.
• networkPolicy.enabled is true (the default), with networkPolicy.ingressFrom restricted to your ingress controller.
• rbac.impersonation.resourceNames is set where you can list who may be impersonated.
• persistence.existingClaim points to a PersistentVolumeClaim, so drift policies and results and the change history survive restarts. Without it the data directory is an emptyDir.
• image.digest pins the image you verified (see Supply chain), so a moved tag can’t change what runs.
• The namespace enforces Pod Security restricted (label it pod-security.kubernetes.io/enforce: restricted). The chart’s pods meet it.
• resources are sized for your clusters.
• replicaCount stays at 1 (see High availability). The chart has no autoscaler and its schema rejects more.

Health checks

Endpoint	What it checks	Used for
/healthz	The process can answer	Liveness and startup probes
/livez	Same as /healthz	Liveness probes in manual deployments
/readyz	The Kubernetes API server answers within 2 seconds; returns 503 if not, and from the moment shutdown begins. With prometheus_url set, the body also says whether that Prometheus answered its last check, run every 30 seconds in the background ("status": "degraded" when it didn’t); that never fails the probe	Readiness probe

The chart sets up liveness, readiness and startup probes. With a base path the endpoints answer at the root and under the path. For a manual deployment:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8090
  initialDelaySeconds: 10
  periodSeconds: 15
  timeoutSeconds: 3
readinessProbe:
  httpGet:
    path: /readyz
    port: 8090
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 3

Give the readiness probe a timeoutSeconds above 2: with the kubelet’s default of 1 second, it gives up before a slow API server’s answer, or the 503 that says the API server is slow.

To check from your machine:

kubectl -n kubeglass port-forward svc/kubeglass 8090:8090 &
curl -s http://localhost:8090/healthz
curl -s http://localhost:8090/readyz

Monitoring

Metrics are at /metrics on the service port, at the root of the pod. They hold the build’s version and commit, request and error counts per route, WebSocket connection counts and the Go runtime’s stats, and no cluster names, users or objects. They need no sign-in, so anyone who reaches the address reads them. At the root the chart’s Ingress routes / with pathType: Prefix, so /metrics is reachable through it too. To keep it in, do one of these:

• Serve KubeGlass under a path (config.basePath, see Serving under a path). /metrics then answers only at the root of the pod, which the Ingress doesn’t route.
• Refuse /metrics at your ingress controller or gateway, for example with an Exact rule for it that doesn’t lead to KubeGlass. The chart’s Ingress can’t do that, as it sends each of its paths to KubeGlass: set ingress.enabled=false and write your own.
• If nothing scrapes it, set KUBEGLASS_METRICS_ENABLED=false in extraEnv. /metrics then answers 404.

Set serviceMonitor.enabled=true to have the chart create a ServiceMonitor for the Prometheus Operator, and prometheusRule.enabled=true for its alert rules (KubeGlassHighErrorRate, KubeGlassPodRestarting, KubeGlassDown). The rules find KubeGlass’s series by their namespace and pod labels, which the ServiceMonitor’s scrape adds, and its restarts in kube-state-metrics. If Prometheus scrapes KubeGlass by pod annotations instead, its scrape job must set those two labels too, or KubeGlassDown fires while KubeGlass is up.

When networkPolicy.ingressFrom names who may reach KubeGlass, as values-production.yaml does, and in local mode when the chart doesn’t expose KubeGlass, where the NetworkPolicy lets in only what it names even when it is empty, Prometheus can’t scrape KubeGlass until an entry there admits Prometheus too. /metrics is on the same port as the UI and the API, so that entry lets those pods reach the API as well: in impersonation mode, with a pod range in config.trustedProxies, they can send X-Forwarded-User and act as anyone, and in local mode they get KubeGlass’s own access, a shell in any pod included. Admit only the Prometheus pods, with one entry that has both a namespaceSelector and a podSelector, which a pod must both match:

networkPolicy:
  ingressFrom:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: monitoring
      podSelector:
        matchLabels:
          app.kubernetes.io/name: prometheus

With serviceMonitor.enabled or prometheusRule.enabled, the install notes say when the NetworkPolicy keeps Prometheus out. A Prometheus that finds KubeGlass by pod annotations also needs podAnnotations with prometheus.io/scrape: "true", prometheus.io/port: "8090" and prometheus.io/path: /metrics; values-production.yaml leaves them out.

Metrics to watch

The thresholds are starting points; tune them for your traffic.

Metric	Type	Suggested alert
kubeglass_api_requests_total	counter, by method, path (the route) and status	5xx above 5% of requests (the chart’s KubeGlassHighErrorRate)
kubeglass_api_request_duration_seconds	histogram, by method and path	p99 above 2s
kubeglass_ws_active_connections	gauge	Near the cap for a sustained period: 512 by default (KUBEGLASS_MAX_WS_CONNECTIONS), 64 per user (KUBEGLASS_MAX_WS_CONNECTIONS_PER_USER)

Log queries

In a pod KubeGlass logs one JSON object per line. Request lines carry method, path, status, duration_ms and request_id; most subsystems add a component field.

# Errors
kubectl -n kubeglass logs deploy/kubeglass | jq 'select(.level == "error")'

# Rejected sign-ins (failed OIDC or token sign-ins, bad bearer tokens, proxy headers from untrusted addresses)
kubectl -n kubeglass logs deploy/kubeglass | jq 'select((.message // "") | test("^Auth rejected|OIDC token validation failed|^Sign-in failed|^Token sign-in refused"))'

# Slow requests (over 5 seconds)
kubectl -n kubeglass logs deploy/kubeglass | jq 'select(.duration_ms > 5000)'

# Circuit breaker changes
kubectl -n kubeglass logs deploy/kubeglass | jq 'select(.component == "circuit-breaker")'

# WebSocket events
kubectl -n kubeglass logs deploy/kubeglass | jq 'select(.component | strings | startswith("ws"))'

Circuit breaker

The circuit breaker opens when the cluster health check can’t reach the Kubernetes API server three times in a row (it runs every KUBEGLASS_HEALTH_INTERVAL, 30 seconds by default), so watches and log streams back off instead of hammering it. It follows the cluster KubeGlass started on, or runs in; watches and log streams of other clusters back off on their own, and don’t wait for it. GET /api/v1/cluster/health reports the last check: degraded from the first failure, with since when, and unreachable once the breaker is open.

Signs that it is open:

• The UI shows stale data and live updates stop. Followed logs say the cluster isn’t answering.
• The log has circuit opened - cluster unreachable, subsystems will back off, then circuit still open - cluster unreachable while the outage lasts.
• /readyz returns 503, so the pod drops out of the Service.

It closes by itself (circuit closed - cluster connectivity restored) when the health check succeeds again. To find the cause:

# Is the API server healthy?
kubectl get --raw /healthz

# Can KubeGlass's ServiceAccount still list pods?
kubectl auth can-i list pods --as=system:serviceaccount:kubeglass:kubeglass

Backup and restore

What is stored

KubeGlass keeps its state in the data directory, /var/lib/kubeglass/data in the chart:

File	Contents
kubeglass.db	Drift results and policies, change history, usage history, each user’s cluster settings (colors and marks) and the sign-in sessions
session.key	The key the sign-in sessions are sealed with, when session_key isn’t set. Without it the saved sessions can’t be read, and everyone signs in again
config/settings.json	Settings changed on the Settings page
config/clusters.json	Each cluster’s own Prometheus URL, set under Settings › Clusters
helm/	Helm’s cache of chart archives and repository indexes, and the chart repositories added in KubeGlass (helm/config/repositories.yaml). The chart sets HELM_CACHE_HOME, HELM_CONFIG_HOME and HELM_DATA_HOME here, since the root filesystem is read-only; KubeGlass does the same by itself when it can’t write Helm’s usual folders. Nothing prunes the chart cache

Everything else KubeGlass shows comes from the cluster. Without persistence.existingClaim the directory is an emptyDir and is lost when the pod is replaced, so there is nothing to back up.

Backup

The image is distroless, with no shell, cp or tar, so kubectl exec and kubectl cp don’t work against the KubeGlass container. Stop KubeGlass, mount its PersistentVolumeClaim in a temporary pod, and copy from there. Replace kubeglass-data with the name of your claim.

# 1. Stop KubeGlass so nothing writes to the database
kubectl -n kubeglass scale deploy/kubeglass --replicas=0

# 2. Start a helper pod with the claim mounted at /data
kubectl -n kubeglass run kubeglass-backup --image=busybox:1.36 --restart=Never \
  --overrides='{
    "spec": {
      "securityContext": {"runAsNonRoot": true, "runAsUser": 65532, "runAsGroup": 65532, "fsGroup": 65532,
                          "seccompProfile": {"type": "RuntimeDefault"}},
      "volumes": [{"name": "data", "persistentVolumeClaim": {"claimName": "kubeglass-data"}}],
      "containers": [{
        "name": "kubeglass-backup", "image": "busybox:1.36", "command": ["sleep", "3600"],
        "securityContext": {"allowPrivilegeEscalation": false, "capabilities": {"drop": ["ALL"]}},
        "volumeMounts": [{"name": "data", "mountPath": "/data"}]
      }]
    }
  }'
kubectl -n kubeglass wait --for=condition=Ready pod/kubeglass-backup

# 3. Copy the data directory to your machine
kubectl -n kubeglass cp kubeglass-backup:/data ./kubeglass-backup

# 4. Remove the helper and start KubeGlass again
kubectl -n kubeglass delete pod kubeglass-backup
kubectl -n kubeglass scale deploy/kubeglass --replicas=1

Restore

Follow steps 1 and 2 above, then copy the files back before removing the helper, leaving out any your backup doesn’t have:

kubectl -n kubeglass cp ./kubeglass-backup/kubeglass.db kubeglass-backup:/data/kubeglass.db
kubectl -n kubeglass cp ./kubeglass-backup/session.key kubeglass-backup:/data/session.key
kubectl -n kubeglass exec kubeglass-backup -- mkdir -p /data/config
kubectl -n kubeglass cp ./kubeglass-backup/config/settings.json kubeglass-backup:/data/config/settings.json
kubectl -n kubeglass cp ./kubeglass-backup/config/clusters.json kubeglass-backup:/data/config/clusters.json

kubectl -n kubeglass delete pod kubeglass-backup
kubectl -n kubeglass scale deploy/kubeglass --replicas=1

High availability

KubeGlass runs as a single replica. bbolt is a single-file embedded database, and only one process can open it for writing.

Aspect	Constraint
Replicas	1. There is no active/active or active/passive mode
Failover	Kubernetes restarts or reschedules the pod; there is no standby
Data replication	None built in. Rely on volume replication or backups
Horizontal scaling	Not supported. The chart has no autoscaler and accepts only replicaCount: 1

With a PersistentVolumeClaim the chart uses the Recreate update strategy, so the old pod releases the volume before the new one starts. The chart’s PodDisruptionBudget is off by default; with one replica, turning it on (podDisruptionBudget.enabled=true, minAvailable: 1) blocks voluntary evictions such as node drains until you move the pod yourself.

For production:

• Use a storage class whose CSI driver supports volume snapshots, and snapshot the claim on a schedule.
• Or run the backup above on a schedule and copy the files to object storage.
• Watch /readyz and the KubeGlassDown alert.

Shutdown

On SIGTERM KubeGlass fails /readyz and serves on for KUBEGLASS_SHUTDOWN_DELAY (none by default, 5 seconds in the chart), while the Service’s endpoints and load balancers drop the pod. Then it stops taking requests and closes WebSockets, waits for the Helm installs, upgrades, rollbacks and uninstalls still running, and closes its database last. It takes at most KUBEGLASS_DRAIN_TIMEOUT (25 seconds by default), the delay included; the chart sets that to terminationGracePeriodSeconds less 5 seconds, so 55 seconds with the chart’s 60. Five seconds before the end it stops the installs and upgrades still running, and Helm marks their releases failed, so the next upgrade or rollback works; a release left pending-install or pending-upgrade would refuse both. Rollbacks and uninstalls can’t be stopped that way. The log names every operation still running at that point (A Helm operation was still running at shutdown): check those releases with helm history and roll back one left pending. Raise terminationGracePeriodSeconds if you run Helm operations from KubeGlass that take longer. Once KubeGlass is back, the page that started an operation says it lost track of it. Run on your machine, a second Ctrl-C ends the wait: KubeGlass stops what still runs, logs each Helm operation left, and exits.

Data retention

Data	Kept for	Pruned
Drift results	30 days (KUBEGLASS_DRIFT_RESULT_RETENTION)	Every hour (KUBEGLASS_DRIFT_PRUNE_INTERVAL)
Change history	30 days (same setting), and at most 32 MiB per cluster, with 4 MiB more for automatic changes. The latest 5,000 changes per cluster, and 1,000 automatic ones, are in memory	Every hour
Browser sessions (in memory)	Idle for session_max_age (default 4 hours)	Every 5 minutes

Incident response

High memory use

1. Check kubeglass_ws_active_connections for many open WebSocket connections.
2. Look for a leak: connection count that grows and never falls.
3. Check the size of kubeglass.db (with the backup helper pod: ls -lh /data). bbolt doesn’t shrink the file when data is deleted. To shrink it, compact a copy with KubeGlass stopped (bbolt’s tool, go install go.etcd.io/bbolt/cmd/bbolt@v1.5.0): bbolt compact -o kubeglass.compact.db kubeglass.db, then restore kubeglass.compact.db as kubeglass.db as in Restore.
4. Lists, watches and search come from informers that KubeGlass starts per user and type when someone uses them. Each stops after 10 minutes without use, and at most 256 run at once.
5. The rate limiter tracks at most 100,000 client IPs and sweeps idle ones every 5 minutes.

Sign-in failures

1. Check the auth mode: kubectl -n kubeglass get configmap kubeglass -o jsonpath='{.data.KUBEGLASS_AUTH_MODE}'.
2. In impersonation mode, check that the proxy’s address is inside KUBEGLASS_TRUSTED_PROXIES and that it sends X-Forwarded-User. Look for Auth rejected: impersonation headers from untrusted source and Auth rejected: missing X-Forwarded-User header in the log.
3. In oidc mode, check that the issuer URL is reachable from the pod and that the redirect URI registered at the provider is exactly KUBEGLASS_OIDC_REDIRECT_URL. Browser sign-ins log Sign-in failed with a reason (the table in Sign-in explains each), and ended sessions Sign-in session ended. For bearer tokens, look for OIDC token validation failed and check that the token’s audience is KUBEGLASS_OIDC_AUDIENCE (the client ID by default).
4. Validated bearer tokens are cached for one minute (up to 10,000), so a revoked token can keep working for up to a minute. A browser session ends when the provider refuses to refresh it, at the latest when its ID token expires.
5. If the pod doesn’t start, read the startup errors with kubectl -n kubeglass logs deploy/kubeglass. Configuration errors (a missing OIDC issuer or redirect URL, a provider KubeGlass can’t reach for its discovery document, a missing KUBEGLASS_TRUSTED_PROXIES in impersonation mode, an entry in it that isn’t a CIDR) stop the server before it listens.

WebSocket disconnections

1. Check the load balancer’s idle timeout. It should be at least 120 seconds, KubeGlass’s IdleTimeout; the server pings every 30 seconds.
2. Check that every proxy in the path supports WebSockets and doesn’t buffer them. With ingress-nginx, raise proxy-read-timeout and proxy-send-timeout (values-production.yaml sets 3600).
3. In the browser, open DevTools, then Network, then the WS filter, and check the close codes.
4. On the server, look for lines with a component of ws, ws-hub or ws-handler.

Uploads to a pod fail with 413

A file uploaded in the Files view can be up to 256 MiB, but every proxy in the path has a body limit of its own: ingress-nginx’s is 1 MiB. Raise it with nginx.ingress.kubernetes.io/proxy-body-size: "256m", and set nginx.ingress.kubernetes.io/proxy-request-buffering: "off" so the file streams through rather than landing on the proxy’s disk first; values-production.yaml sets both. The same goes for a Secret or manifest near max_dynamic_body_size.

Slow UI

1. Check the log for circuit breaker messages.
2. Check API server latency: kubectl get pods -A -v=6 prints the time each request took.
3. Check PromQL queries: they are limited to 4,096 characters and a 30-day range.
4. Check kubeglass_api_request_duration_seconds for the routes that take longest.

Terminal problems

1. Terminal sessions end after 30 minutes idle (with a warning at 24 minutes). Exec sessions in pods also end after 8 hours whatever their activity (with a 5-minute warning).
2. Check that the user may exec: kubectl auth can-i create pods/exec -n <namespace> --as=<user>.
3. A single WebSocket message may be at most 1 MB.

Secret and key rotation

TLS certificate

If TLS is terminated at the ingress (recommended), follow your ingress controller’s procedure; cert-manager renews certificates by itself.

If KubeGlass terminates TLS itself, you set KUBEGLASS_TLS_ENABLED=true, KUBEGLASS_TLS_CERT and KUBEGLASS_TLS_KEY with extraEnv and mounted a TLS Secret with extraVolumes and extraVolumeMounts. KubeGlass looks at the certificate and key files every 30 seconds and serves a new pair once they change, so a certificate cert-manager renews in that Secret needs no restart. The kubelet updates the mounted files after the Secret changes, usually within a minute or two, but never for a mount with subPath. A new pair that doesn’t load, such as a certificate copied without its key, leaves the last good one served, and KubeGlass logs a warning. To rotate it by hand:

# 1. Update the TLS Secret with the renewed certificate and key
kubectl -n kubeglass create secret tls kubeglass-tls \
  --cert=new-cert.pem \
  --key=new-key.pem \
  --dry-run=client -o yaml | kubectl apply -f -

# 2. Wait for the kubelet to update the files and KubeGlass to read them,
#    usually a minute or two; no restart needed

# 3. Check the certificate being served
openssl s_client -connect kubeglass.example.com:443 -servername kubeglass.example.com < /dev/null 2>/dev/null | \
  openssl x509 -noout -dates -subject

Upgrade and rollback

Upgrade

# 1. Review what will change (needs the helm-diff plugin)
helm diff upgrade kubeglass oci://ghcr.io/kubeglass/charts/kubeglass \
  --namespace kubeglass \
  -f custom-values.yaml

# 2. Back up the data directory (see Backup above)

# 3. Upgrade
helm upgrade kubeglass oci://ghcr.io/kubeglass/charts/kubeglass \
  --namespace kubeglass \
  -f custom-values.yaml \
  --wait --timeout 5m

# 4. Verify
kubectl -n kubeglass rollout status deploy/kubeglass
curl -sf https://kubeglass.example.com/readyz

Add --version <chart-version> to helm diff upgrade and helm upgrade to pin a release.

Rollback

# List release history
helm history kubeglass -n kubeglass

# Roll back to an earlier revision
helm rollback kubeglass <revision> -n kubeglass --wait

If the newer version changed stored data in a way the older one can’t read, restore the backup you took before upgrading (see Restore).

Contacts

MAINTAINERS.md lists the maintainers. SECURITY.md describes how to report a vulnerability.