Everything was green. Six URLs, six 200s, containers up for three days, no alerts, no complaints.
Then I ran one command inside a running container:
$ docker exec app-api-1 getent hosts postgresIt returned an address. The wrong one. It belonged to a completely different application's database.
That app had been resolving its database by the hostname postgres for weeks. So had three others. All four containers were on the same Docker network, all four had a service named postgres, and Docker's embedded DNS was round-robining between them.
It worked because of connection pooling. The pool establishes once at container start, happens to land on the right host, and holds. It works right up until the next restart — and a deploy is a restart.
I only found it because I'd spent the day on something that sounded much more boring.
The boring thing
I have nine applications on one VPS. Four are live. All of them were migrated by hand off Railway onto a Contabo VPS, which meant the compose files, some Dockerfiles, and a cron script existed only on that machine.
The goal was unglamorous: make every app rebuildable from its git repository, then wire up push-to-deploy.
The exercise turned out to be a diagnostic. Every single app I touched had at least one defect that had been running, invisibly, for months.
Bug 1 — the image was missing a dependency it imports
The first app had an uncommitted change to a Dockerfile on the server. My instinct was to discard it — a stray edit, presumably left over from the migration.
I looked at it first:
- RUN npm run build -w apps/api
+ RUN npm run build -w @acme/amortization-engine && npm run build -w apps/api
+ COPY --from=builder /app/packages/amortization-engine ./packages/amortization-engineThat's not a stray edit. It builds a workspace package and copies it into the runner stage. Without it, the API image doesn't contain a package the API imports at runtime.
The running container was fine — it had been built with this change. But the committed Dockerfile could not produce a working image. Anyone cloning that repo, including future me on a new machine, would have got a broken build and no obvious reason why.
If I'd run git checkout on that file, the next deploy would have failed with an error pointing at a missing module, three abstraction layers away from the cause.
Bug 2 — the build instructions existed in exactly one place on Earth
The second app had no Dockerfile in its repository. And no compose file.
The live image was building from an untracked file sitting in the application directory on the server. Not in git. Not backed up. One rm -rf or one dead disk from being gone.
The app worked perfectly. It had worked perfectly for weeks. It simply could not be rebuilt by anyone who didn't have that exact machine.
That's not a tidiness problem. On a €14/month VPS with no snapshots, that's a business continuity problem wearing a disguise.
Bug 3 — the wrong entrypoint, in the image, all along
Having captured the config into git, I recreated one service to bring it under proper management. It crash-looped.
Error: Cannot find module '/app/server.js'The image's CMD was node server.js. Next.js standalone output in that monorepo lives at /app/apps/web/server.js.
The running container had never executed that CMD — it had been created months earlier with an override that got lost. So the image had shipped with a broken default command for its entire existence, and nothing noticed, because nothing had ever used it.
That one cost five minutes of downtime and taught me the check that found bug 4.
The check that changes everything
After bug 3, I added a step before recreating anything:
# capture what the WORKING container is actually doing
docker inspect <container> --format 'Entrypoint: {{json .Config.Entrypoint}}
Cmd: {{json .Config.Cmd}}
WorkingDir: {{.Config.WorkingDir}}
Env: {{json .Config.Env}}'Then build the new image and compare, before starting anything.
The point is subtle but it's the whole lesson: a running container is not the same thing as its configuration. It's a snapshot of whatever produced it, which may include manual overrides, a since-edited file, or an environment variable typed once into a terminal a year ago. The config file is a claim about the container. The container is the fact.
Diffing the claim against the fact is where every one of these bugs lived.
Bug 4 — the one that mattered
On the next app, Cmd, Entrypoint and WorkingDir all matched. The environment didn't.
The working container connected to learning-postgres-1. The compose file said postgres.
On a shared Docker network with four applications each defining a postgres service, that name resolves to four containers. Docker round-robins. So this application had a one-in-four chance, on every fresh connection, of authenticating against a different app's database.
Because the passwords differ, it failed authentication and returned HTTP 500 rather than reading the wrong data. That distinction is entirely luck. Had I reused a password across two apps — which is exactly the kind of thing you do at 1am during a migration — the connection would have succeeded, against the wrong database, and depending on schema overlap I'd have been reading or writing another application's data with no error at all.
The first thing I did after finding it was hash every database password and compare them, to establish whether I had a config bug or a data-integrity incident. They were distinct. It was a config bug.
I have rarely been so pleased about past-me's laziness in not copy-pasting a password.
Why nothing caught any of this
Every one of these had the same shape: the system worked, and the description of the system was wrong.
Uptime monitoring can't see it — the URLs return 200. Health checks can't see it — the app is healthy. Tests can't see it — they run against a correctly-built image in CI, not the one on the server. Nothing in a normal observability stack compares what is running against what your repository says should be running.
The only thing that surfaces it is trying to rebuild from scratch and diffing the result. Which is exactly the thing nobody does, because everything is working.
What I actually changed
Every deployment artifact is in the repo. Compose files, Dockerfiles, and a deploy/README.md per app naming the project, the environment variables it expects (names only), the named volumes, the network, and the exact command to bring it up on a bare machine. Secrets stay on the server; nothing else does.
Internal hostnames are unique. <project>-<service>-1, never the bare service name. The real fix is a private network per app so the databases aren't mutually reachable at all — that's written up and scheduled, because "correct later" beats "correct instead of shipping."
--no-deps on every service recreate. I learned this by accidentally restarting a database while recreating a web container. The volume was named so nothing was lost. That was luck too.
Named volumes everywhere. One Redis instance was on an anonymous volume — the kind that quietly disappears the moment you docker compose down.
Push to main deploys. Tests run, then a GitHub Action SSHes in with a key pinned via command= in authorized_keys, so a leaked key can redeploy an allowlisted app and do nothing else. No shell, no port forwarding.
The uncomfortable question
None of this was caused by carelessness in any single moment. Each one was a reasonable decision under time pressure during a migration: fix the Dockerfile on the box to get it building, use the obvious hostname, ship it, move on.
The problem is that "fix it on the box" leaves no trace. The system converges on working, and the repository quietly diverges from it. Nothing announces the gap. You find out during an outage, or when you try to rebuild, or — in my case — when you deliberately go looking.
So: when did you last rebuild a production service from a fresh clone and diff the result against what's running?
If the answer is "never," I'd genuinely encourage you to try it on one service this week. It took me a day across nine apps and found four things. My guess is you'll find at least one.
Mine had been running for months, and everything was green the whole time.
I build and deploy AI systems — retrieval platforms, in-app assistants, automated regression testing — on infrastructure like this. If your team wants something in production rather than in a notebook, the AI Readiness Audit is built around the gap between what the repo claims and what is actually running.