Troubleshooting

The operator is designed to be self-explaining: when something is pending or wrong, the resource's status.conditions and Kubernetes Events tell you exactly what it is waiting on. Start there.

First steps

# The resource's conditions, events, and phase.
kubectl -n <ns> describe gateway <name>

# Just the conditions.
kubectl -n <ns> get gateway <name> -o jsonpath='{.status.conditions}' | jq

# The operator's own logs.
kubectl -n navique-system logs deploy/navique-ai-core-operator -f

Every resource exposes a Ready condition plus per-phase conditions (SecretsReady, OperatorsReady, DatastoreReady, WorkloadReady, …), each with a reason and message.

Reading conditions

Condition reason	What it means	What to do
SecretsNotReady	The referenced SecretsManagement is not ready	Check the SecretsManagement resource and its ESO/Sealed Secrets backend
WaitingForCRDEstablished	A capability operator's CRDs are not Established yet	Normal during install; check the operator's pods if it persists
DependencyNotReady	A referenced datastore / gateway / Langfuse is not ready	Inspect the referenced resource
AutoWiringUnlicensed	A reference needs auto-wiring, which is unlicensed	Supply the manual fields the message names, or apply a license
LicenseLimitExceeded	You exceeded a licensed instance cap	Reduce instances or raise the limits in the license
BackendNotImplemented	A scaffolded backend (zalando / cockroachdb) was selected	Use cnpg, or adopt / external
InfrastructureBlocked	A managed datastore pod cannot start for a cluster-infrastructure reason the operator cannot fix by waiting	Read the message — it names the pod and the cause (unschedulable, volume attach, image pull) — and fix it at the cluster level

A resource that is requeuing while waiting is normal — the operator requeues rather than erroring for "not ready yet". A resource stuck Ready=False with an error reason is the one to investigate.

Capability-operator install diagnostics

When a capability operator is not coming up, the operator inspects its namespace and surfaces the cause into the consuming resource's status — for example a container's Waiting reason, an OOM/last-termination, or the latest Warning event like FailedMount: secret "webhook-server-cert" not found. So a stalled upstream install is explained in the resource that needs it.

# Inspect a capability operator directly.
kubectl -n cnpg-system get pods
kubectl -n cert-manager get pods,events --sort-by=.lastTimestamp

Common issues

The manager is OOM-killed

The in-binary Helm engine renders charts in-process and needs memory. Give the manager ~1Gi — the default Kubebuilder limits are too low and only manifest in-cluster. See Configuration.

A capability operator won't become ready

Most often a missing webhook certificate (cert-manager not ready) or insufficient RBAC. cert-manager is treated as a universal prerequisite and is ensured first; confirm it is healthy. Check that the operator's broad install RBAC is in place.

A datastore is stuck InfrastructureBlocked

A managed PostgresCluster, ClickHouseCluster, RedisInstance, MongoCluster, or MeilisearchInstance whose pods cannot start reports Ready=False with reason InfrastructureBlocked instead of the generic "waiting…" message (the same detail bubbles up into a Stack component message and a Warning event). The condition message names the pod and the underlying cause, so you do not have to kubectl describe to find it:

• Unschedulable — the scheduler could not place a pod (most often Insufficient cpu/memory). Scheduling is per-node: a request that exceeds any single node's free capacity stays Pending even when the cluster's total free capacity looks sufficient. Add a node, raise the node-pool max/cluster-autoscaler limit (check it isn't in scale-up backoff), or lower the datastore's resources.requests / instances.
• VolumeAttachFailed — a persistent volume could not attach/mount (e.g. an Azure managed-disk cannot find Lun when a node is at its max data disks for its VM size, or a CSI attach error). Reschedule the pod onto a node with free disk slots, or use a larger VM SKU.
• ImagePullFailed — the image cannot be pulled (wrong registry/tag, missing pull secret, registry unreachable). Verify the registry mirror and pull-secret configuration.

The operator keeps requeuing throughout — it does not error or roll back — so the resource recovers on its own once the cluster-level cause is resolved.

A capability operator won't install — conflict with "argocd-controller": .spec.versions

A datastore/workload stays Ready=False with an error like install langfuse-operator: failed to install CRD … conflict with "argocd-controller": .spec.versions. This happens when an upstream CRD was previously installed by a different GitOps app that has since been deleted — the app is gone but its CRDs were never pruned (CRDs never are), so they linger carrying that app's stale Server-Side-Apply field ownership. Helm installs a chart's CRDs from its crds/ directory via a path that does not force SSA conflicts, so the lingering owner blocks the install.

The operator now auto-heals this: before installing any capability operator it detects a foreign field manager (argocd-controller, flux, …) on its declared CRDs and resets that stale ownership (non-destructively — spec and instances are untouched). If you hit it on an older operator build, clear it manually:

# Confirm no live GitOps app actually manages the CRD (only a stale tombstone):
kubectl get crd <crd> -o jsonpath='{range .metadata.managedFields[*]}{.manager}({.operation}){"\n"}{end}'
# Reset the field-ownership bookkeeping (does not touch the spec or any instances):
kubectl patch crd <crd> --type=merge -p '{"metadata":{"managedFields":[{}]}}'
# Then let the operator retry (it requeues automatically).

CRDs of a newly-installed kind aren't found

Right after a CRD-bearing chart installs, there is a small known race where the new kind isn't yet in the discovery cache. The operator treats a transient NoResourceMatchError as a requeue and recovers automatically — it should clear within a reconcile or two.

A new CRD is missing after a kustomize install

When installing from source, a CRD must be listed in config/crd/kustomization.yaml or kustomize silently omits it (envtest still passes, which can hide it). Use the provided overlays, which include all CRDs.

Auto-wiring isn't happening

Auto-wiring is licensed. Without the auto-wiring feature, the operator uses your manual fields and emits an AutoWiringUnlicensed condition naming exactly what to set. For Gateway→Langfuse key minting, remember it also needs a Langfuse Enterprise deployment — see Auto-Wiring.

Deleting a resource hangs

A finalizer is running cleanup (removing emitted CRs, uninstalling owned releases). Check the operator logs; if a referenced dependency is itself stuck, the finalizer waits. It will not delete adopted/external resources.

If a CRD is stuck Terminating (and any owner chain above it — e.g. a Stack or ChatUI that never finishes deleting), look for a leftover custom resource of that CRD that still carries a finalizer belonging to an upstream operator that has already been uninstalled. Such a finalizer is orphaned — its controller no longer exists to clear it — so the object, and the apiextensions CRD-cleanup finalizer above it, can never complete. The operator now strips these finalizers automatically during capability-operator uninstall, so this should not recur. To recover a cluster that is already wedged, clear the dead finalizer once:

kubectl patch <kind> <name> -n <ns> --type=merge -p '{"metadata":{"finalizers":null}}'

ChatUI SSO login fails behind a WAF / TLS-terminating gateway

When a ChatUI (LibreChat) is fronted by a TLS-terminating ingress or an external WAF (e.g. an Azure Application Gateway), OIDC/SSO sign-in can fail in three distinct, stacked ways — fix them outermost-first:

1. The IdP rejects the redirect (AADSTS900971: No reply address provided, or a redirect mismatch). LibreChat builds its callback from DOMAIN_SERVER, which the operator derives from the ChatUI host. Set host and ingress.tls: true on the ChatUI (or the Stack's chatUI) so DOMAIN_SERVER becomes https://<host>, and register https://<host>/oauth/openid/callback as a Web redirect URI in the IdP.

2. The edge returns its own 403 before the app (e.g. a Microsoft-Azure-Application-Gateway/v2 page). A WAF is blocking the OIDC callback: the base64url code/state/id_token parameters trip the OWASP SQLi rules (942430 / 942440) as false positives. Add a narrow, per-rule exclusion for those argument names on the /oauth/* path and keep the rest of the WAF in Prevention mode. Confirm the exact rule from the WAF/firewall logs before excluding.

3. The app shows "Authentication failed" with Unable to verify authorization request state in the LibreChat logs. The TLS-terminated hop makes the app see plain HTTP, so a Secure session cookie is dropped and the OAuth state is lost. The operator sets SESSION_COOKIE_SECURE=false automatically whenever spec.sso is enabled, so this is handled out of the box — if you still see it, confirm the running LibreChat pod carries that env (the operator owns the *-librechat-configenv ConfigMap).

A different symptom — login reaches the app but fails with auth_failed and registered with "local" provider — means a pre-existing account (e.g. one created earlier by identity management) already owns that email under a non-OIDC provider. Deleting the owning Identity/User CR now tears the LibreChat account down via a finalizer; if an orphan predates that, remove it once with the bundled npm run delete-user <email> and sign in again.

Getting the full picture

# All Navique resources across the cluster.
kubectl get -A \
  licenses.core.navique.com,secretsmanagements.core.navique.com,\
postgresclusters.core.navique.com,clickhouseclusters.core.navique.com,\
redisinstances.core.navique.com,mongoclusters.core.navique.com,\
meilisearchinstances.core.navique.com,gateways.core.navique.com,\
langfuses.core.navique.com,chatuis.core.navique.com

# License status (features + limits in effect).
kubectl get license cluster -o jsonpath='{.status}' | jq

If a resource's bundled chart version or provenance state is in question, it is surfaced in status for auditability — see Provenance & Lifecycle.