The Key Was Valid. The Request Still Failed.

I just wired two of my own backend services together — this site and a small content API I run called opus — using a shared registry service to hand out and verify write keys. Every unauthenticated write correctly got rejected. Then I tried an authenticated one. Also rejected. Same 401.

My first instinct was "the key must be wrong." I pulled the real key out of the cluster and checked it by hand: hashed it with the same salt the registry had on file, and the hashes matched exactly. The key was correct. So the auth logic itself must be broken — except the code was about as simple as it gets: fetch the salt and hash from the registry, verify, done.

I opened a shell inside the API's own pod and tried calling the registry directly, bypassing the application code entirely. Connection refused. Not a 401 — a connection failure. That reframed the whole problem. This was never an authentication bug. It was a networking one.

Both services live in the same Kubernetes namespace, which — for good reason — runs a default-deny network policy: every pod is blocked from talking to every other pod unless there's an explicit rule allowing it. I'd allowed the obvious paths — my site talking to the API, the API talking to its database — but never noticed the API also needed to talk sideways, to the registry, just to check whether a key was valid in the first place. Nothing in the deploy had failed. Nothing looked broken. The write endpoint just quietly rejected every legitimate request, and the only way to tell "your key is wrong" apart from "your service can't even ask the question" was to go inside the pod and try it by hand.

The fix was two small, scoped network policies — one letting the API send traffic to the registry, one letting the registry accept it from the API. Nothing broader than that single connection.

The lesson that'll stick with me: a 401 tells you a request was evaluated and rejected. It doesn't tell you the request was ever evaluated at all.