service mesh · SEV-2

connection reset by peer, immediately after enabling STRICT mTLS

scroll to render

How to confirm it

Who is calling without a sidecar?
```
kubectl logs POD -c istio-proxy --tail=200 | grep -i 'no healthy upstream\|RBAC: access denied\|TLS error'
```
The sidecar log names the peer that failed the handshake. That is your caller.
Confirm the policy is what you think
```
kubectl get peerauthentication -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,MODE:.spec.mtls.mode
```
A mesh-wide STRICT in istio-system quietly overrides what a namespace policy appears to say.
Prove encryption before enforcing
```
kubectl exec POD -c istio-proxy -- pilot-agent request GET stats | grep ssl.no_certificate
```
Non-zero means something is still connecting without an identity. Enforce when this holds at zero.

Read the source

Cause

A client without an Istio sidecar was calling the service. In PERMISSIVE mode its plaintext traffic was silently accepted; STRICT correctly refuses it.

Fix

Inject the sidecar into the client, or scope the policy per service and migrate callers first. The error was the policy working, not the policy failing.

Part of this work

Zero-Trust Access Migration ↗

Deleted the bastion hosts. Every VM, cluster and pipeline behind MFA.

Next incident

APM init container stuck in ImagePullBackOff across several services ↗