essayFRAMEWORK: Green by Omission

Green by Omission

For three months my API was down and I did not know, because the health check returned 200 without ever asking the database whether it was alive. That probe is the whole story of the product. I built six instruments that reported health by not looking, and the last one told me I had 226 customers when I had none.

August 22, 2026 · 1,604 words · 7 min read

Today I deleted the Railway project, the three Vercel deployments, and the three npm packages behind an API I spent five months building. The database is a 450 KB file on my laptop now. Before I turned it off I ran the only query that mattered: 226 organizations had signed up, 12 had ever posted a job, and all 12 were me — my own test suite, my own gmail address, my own verification probes. Total revenue across the life of the product: zero dollars. Not "low." Zero. No external buyer ever posted a paid job.

The part I want to write down is not that it failed. It is that for most of those five months I had instruments on the wall telling me things were fine, and every one of them was fine in the same specific way: it returned green because it was not looking.

The probe that checked nothing

Start with the literal case, because it is the cleanest.

The API had a readiness endpoint. Railway pinged it, it returned 200, and the deployment stayed marked healthy. What the endpoint did not do was open a database connection. It returned 200 because the web process was running — which is a thing the web process can always confirm about itself.

For roughly three months, an intermittent DNS failure on the platform's private network meant the API could not reach Postgres. Every request that touched data failed. The health check kept returning 200, because the health check had no opinion about Postgres. I found out by accident. 351 signups arrived during that window and hit errors.

A health check that cannot fail for the reason you actually go down is not a health check. It is a liveness test for the thing least likely to be broken.

Five more of the same

Once I had the shape, I could see it everywhere in my own build.

The packages that could not be installed. I published a TypeScript SDK and an MCP server to npm. Both shipped with "@firsthandapi/sdk": "workspace:*" in the manifest, because I ran npm publish in a pnpm monorepo and npm does not understand pnpm's workspace protocol. Every install failed with EUNSUPPORTEDPROTOCOL. This silently killed both documented onboarding paths — the SDK install in step two of the quickstart, and npx @firsthandapi/mcp-server, which was the entire basis of my "MCP-native" positioning. The publish command exited 0. The registry showed the package. The green light was "the tarball uploaded," which was true.

The deploys that shipped nothing. Railway watched /apps/api/** for changes. In a monorepo where most real work lands in /packages/**, that meant deploy after deploy came back SKIPPED — and the dashboard's top-line status showed the last successful deployment, which was four months old. Production ran a commit from April while I merged pull requests against a main branch that never reached it. Ten commits deep before I checked what was actually running.

The scorer that graded nothing. The AI scoring ensemble pinned a model ID. The model was retired. Every scoring call 404'd from April onward.

That one was not merely invisible; it was actively harmful, and this is the failure I am least willing to forgive myself for. A failed score was recorded as a one-star result. Three one-star results triggered an automatic ban. So an outage in my infrastructure was silently converted into a permanent penalty against the people doing the work. Four workers were banned for my 404. They had no way to know, and no channel to appeal, because the whole point of the automated gate was that no human was in the loop. I unbanned them and repaired the records, but the design lesson is sharper than the fix: never let an infrastructure failure resolve into a judgment about a user. If the system cannot tell "you did badly" apart from "I broke," it must refuse to score, not score zero.

The robots.txt that was not mine. I had a robots.txt in the repo. Cloudflare's crawl-control feature injected a different one over it at the edge. What I read in my editor and what the internet read were different files. I had verified the artifact. I had not verified the response.

The dashboard counting its own fixtures. Activation metrics included nine seed jobs I created to demo the marketplace and every org my end-to-end suite had ever registered at test.com. The numbers moved. They moved because my test suite ran.

The instrument that mattered

Now the one that cost five months instead of a weekend.

226 organizations signed up. I watched that number grow. It felt like the demand side working, slowly. It was the same instrument as the health check: green because it was not looking at the thing I needed to know. Signing up cost a stranger nothing and committed them to nothing. The number it produced was real, and it measured curiosity, and I read it as demand.

The query that would have gone red — how many organizations that are not me have posted a paid job — I did not run until the day I shut the product down. It had been answerable since week three. The answer would have been zero every single time I asked.

And the supply side, which I worried about constantly, simply worked. 55 people signed up to do the capture work. They submitted 134 files. Nobody had to be talked into it. I spent five months building better tooling for the side of the marketplace that was not the problem, because that side was the side with code in it.

Why this shape is getting more common, not less

The honest version of the diagnosis is not "I was careless." I was, in places, but that is not the interesting part.

I built roughly six surfaces — a REST API, an iOS app that shipped to the App Store, an AI scoring pipeline, a worker payout system on Stripe, a docs site, an MCP server listed on the official registry — at a speed that would have taken a small team a year. Every one of those surfaces is a place to be silently wrong. Build velocity went up by something like an order of magnitude. My instrumentation did not, because instrumentation is boring and the tooling does not volunteer it.

Then the multiplier: a product with no users cannot report its own bugs. In every job I have had, the alarm that actually fires is a person saying "this is broken." That channel is load-bearing and invisible until it is empty. With zero external users, no alarm ever rings — and the silence is indistinguishable from working. I had built a system whose only error-reporting mechanism was a customer, and I had no customers, which is precisely the condition the system was failing to tell me about.

That is the recursion that makes this worth a name. The absence of demand suppressed the signal that would have revealed the absence of demand.

The rule I am taking forward

One rule, and it is cheap:

Every green light must be able to go red, and I must be able to name the specific failure that would turn it red, before I trust it.

If I cannot state the condition, it is not an instrument — it is decoration. Applied to the six above, this is not abstract:

  • The health check must open a database connection, or it does not get to say "healthy."
  • The publish step must install the published tarball from the public registry into a clean directory, or it did not publish.
  • The deploy check must assert on a behavioral change in the running service — a new validation returning 400 where the old code returned 200 — never on the word SUCCESS in a dashboard.
  • The model pin must have a startup assertion that the model ID still resolves, and a scorer that refuses rather than scores zero when it cannot reach the model.
  • The site check must fetch the served response, not read the repo file.
  • The activation metric must exclude every fixture, seed, and internal domain — and if excluding them takes the number to zero, the number was always zero.

And the market instrument, which is the one I would pay real money to have installed in March: the only demand metric that may appear on my dashboard is one where a stranger gave up something they cannot get back. Money, or a scheduled hour, or a credential. Not a signup. Signups are the readiness probe of a business — they confirm the web process is running.

What I owe and what I kept

Twenty workers hold $15.76 in balances they never withdrew; the withdrawal threshold was $10 and the largest single balance ever reached was $3.04. Nobody was ever blocked from a payout they had earned, which is a thin comfort and I will take it. The Stripe account is closed and the ledger is archived rather than deleted, which keeps that decision reversible.

I kept the research. The provenance architecture — content hashing, capture-time attestation, verified GPS at about 110 metres — is the genuinely novel piece, and it is written down. Whether anyone pays for verified-authentic media is still an open question, and as of this month the answer is that no market for it exists anywhere I can find.

The five months bought one durable thing, and it was not the code. It was a list of six specific ways I lie to myself with a dashboard, each one attached to a number I can still show you.

SUBSCRIBE

Get the next one in your inbox.

Monthly dispatch today. Weekly paid tier starting 2027. No tracking pixels.

NO TRACKING · UNSUBSCRIBE ANYTIME