Observability on the Edge

I want to open with an honest inventory, because the number is the whole story. The starting position: 58 console.error calls and zero request ids. Fifty-eight places where the codebase, at the moment something went wrong, cupped its hands and shouted a string into the void.

This is part 10 of 10 in the RORBT field notes—dispatches from building a governance platform on Cloudflare. The whole series lives on the RORBT case study.

console.error('[orbit/messages] send failed:', e);
console.error('[governance] execution failed:', e);
console.error('[split] settle failed:', e);

Every one of these is a string with no thread back to the request that caused it, no correlation between the four lines a single failing request emitted, and nothing a dashboard could aggregate. On k8s I would have compensated the way I always compensated—shell into a pod, poke at the corpse, reconstruct the crime. Here there is no pod. There is no corpse to poke. The only instrument you have is what you thought to write down before the crash, which is a flight-recorder discipline, and it raises the stakes on every lazy log line considerably.

Why pino and winston don’t help

The obvious move—the move my hands were already making—is to reach for a real logging library. I looked, and the answer turned out to be more interesting than “it doesn’t work.”

Winston is a hard no. Built on Node streams, fs, and transports. No edge build. It won’t run in workerd. Next.

Pino is viable but loses the things that make it pino. It ships a browser.js build (the package is ~663 KB unpacked), and that’s what you’d get in a Worker. That build drops sonic-boom, worker-thread transports, and pino-pretty—which are the speed. Pino’s performance reputation comes from optimized process.stdout writes and offloading serialization to a worker thread. Neither exists here. You’d be importing the reputation and leaving the engine at the door.

So “pino is faster” doesn’t transfer to this runtime. On the edge, both approaches bottom out at console.log(JSON.stringify(...)), and the real cost is the platform’s log ingestion, not the serializer. This is the same lesson this series keeps teaching me in different costumes: the benchmark you memorized was measured on a machine that no longer describes your life.

And it wouldn’t solve the actual problem. The hard part isn’t formatting—it’s correlating 58 calls buried in _lib helpers that take env and know nothing about requests. Pino doesn’t do that for you; you’d still write the AsyncLocalStorage plumbing yourself, then layer child loggers on top of the thing you just built. pino-http is Express-shaped and irrelevant.

Where pino would earn its place is redaction and serializers—genuinely relevant given this project’s PII doctrine. That’s ~10 lines added to a wrapper.

I wrote 90 lines instead. Because everything routes through log.*, swapping the backend later is a one-file change. Ninety lines is not a logging framework; it’s an insurance policy with a very low premium.

AsyncLocalStorage is the actual answer

Here’s the problem stated plainly. The log calls are deep in helpers with signatures like saveOrbit(env, orbit, etag). Threading a logger through all of them is a far larger change than the problem warrants—the kind of refactor that touches forty files to fix a problem that lives in one concept.

AsyncLocalStorage gives you ambient request context without touching a single signature. It requires nodejs_compat, and—because I have been burned before and intend to stay burned—I verified it survives awaits in workerd before building anything on it:

it('carries context across awaits', async () => {
  const als = new AsyncLocalStorage<{ id: string }>();
  const seen: (string | undefined)[] = [];
  async function deep() {
    await new Promise((r) => setTimeout(r, 1));
    seen.push(als.getStore()?.id);
  }
  await als.run({ id: 'abc' }, async () => { await deep(); });
  seen.push(als.getStore()?.id);
  expect(seen).toEqual(['abc', undefined]);   // ✓
});

Middleware opens the scope once per request:

export const onRequest: PagesFunction<Env> = async (ctx) => {
  configureLogging(ctx.env);
  return await withRequestContext(
    {
      requestId: requestIdFor(ctx.request),
      method: ctx.request.method,
      path: new URL(ctx.request.url).pathname,
    },
    () => handle(ctx),
  );
};

And everything beneath it inherits, with no plumbing:

// 200ms later, four call frames deep, in a file that has never heard of a Request
log.error('proposal execution failed', {
  err: e, scope: 'governance', orbitId: p.orbitId, proposalId: p.id, kind: p.kind,
});

{"level":"error","msg":"proposal execution failed","ts":"2026-08-01T12:34:56.789Z",
 "requestId":"8a3f…","method":"POST","path":"/api/orbit/x/proposals",
 "userId":"usr_123","scope":"governance","orbitId":"orb_9",
 "errName":"TypeError","errMessage":"…","errStack":"…"}

A file that has never heard of a Request now emits lines that know exactly which request they belong to. That’s the whole trick, and it cost zero signature changes.

Design decisions worth stealing

A few choices in those 90 lines that I’d defend in any codebase.

Errors get unwrapped, not stringified. JSON.stringify(new Error('x')) is {}—the single most common way a structured log throws away the one thing you actually need:

function serializeError(err: unknown): Record<string, unknown> {
  if (err instanceof Error)
    return { errName: err.name, errMessage: err.message, errStack: err.stack };
  return { errMessage: String(err) };
}

Logging can never break a request. A cyclic field value would otherwise throw inside JSON.stringify and take down the handler it was supposed to be observing:

try {
  text = JSON.stringify(line);
} catch {
  text = JSON.stringify({ level, msg, ts: line.ts, note: 'fields were not serializable' });
}

There’s a test asserting exactly this behavior, because I hold the principle strongly enough to spend a test on it.

A logger that can crash your service is worse than no logger.

The user is attached at authentication, so every subsequent line in the request carries it:

// in requireVerifiedUser, after the session validates
setLogUser(session.user.id);

Levels gate at emit, so log.debug costs one integer comparison when off:

function emit(level: Level, msg: string, fields: LogFields = {}): void {
  if (SEVERITY[level] < minSeverity) return;
  …
}

Configured by a LOG_LEVEL var, applied per request—which means you can turn on debug in production without a deploy. And an unrecognised value falls back to info rather than silence:

// An unrecognised value falls back to `info` rather than silencing the logs,
// which is the safer failure.

A typo in a Pages variable should never turn your logging off. That’s the kind of default nobody notices in code review and everybody notices at 3am, which is exactly the wrong order.

silent exists too, and the test suite runs at it—expected-failure tests otherwise spew, and the logging tests raise the level for their own assertions.

Request ids that cross system boundaries

export function requestIdFor(request: Request): string {
  const supplied = request.headers.get('x-request-id');
  if (supplied && supplied.length <= 128) return supplied;
  return request.headers.get('cf-ray') ?? crypto.randomUUID();
}

Three sources in priority order, and each one is deliberate:

1. A caller-supplied x-request-id—so a mobile client or an E2E suite can generate its own trace id and have it flow through your logs. This is what makes cross-repo debugging possible.
2. Cloudflare’s cf-ray—ties your line back to CF’s own edge logs.
3. A fresh UUID.

The length cap is not cosmetic: an unbounded attacker-supplied string lands in every log line for that request, and I’ve spent enough of this series writing about hostile input to not hand out that invitation.

And the id echoes back on every response:

// Re-wrap rather than mutate: a Response from `next()` has immutable headers.
// Passing the body through preserves streaming (the orbit SSE endpoint).
const headers = new Headers(out.headers);
headers.set('x-request-id', id);
return new Response(out.body, { status: out.status, statusText: out.statusText, headers });

Verified end to end—a supplied trace-me-123 comes back verbatim, one is minted when absent, it’s present on the JSON 404, and the site’s HTML pages are untouched because the middleware is /api-scoped.

A bug report is now “here’s the request id” instead of “it broke around 2pm.”

The sweep: 58 calls

Then the unglamorous part: all 58 calls, by hand. They were all the same shape—a catch returning 500—and all of them are material, so none were deleted. Seven were downgraded to warn for the reason I laid out in post 09: expected rejections logged at error level drown real failures and let anyone with a keyboard spam your error log on demand.

graph LR
  A[58 console.* calls] --> B[49 → log.error]
  A --> C[7 → log.warn<br/>expected rejections]
  A --> D[2 → log.warn<br/>degraded, not failed]
  B --> E[0 console.* remaining<br/>outside log.ts]
  C --> E
  D --> E

What’s still missing

I’d be lying if I ended on “and now we’re observable,” so here is the honest ledger.

Traces. Correlation ids are in—every line carries requestId, method, path, and userId. But there are no spans: no timing, no causality, no “this request spent 340ms in R2 and 12ms in D1.”

@microlabs/otel-cf-workers patches the fetch/R2/D1 bindings to emit spans automatically and exports OTLP. Worth doing—and specifically worth doing after the Durable Object work, because a Worker to DO hop is exactly what a trace shows and a log line doesn’t.

Log aggregation. JSON lines are queryable only if something ingests them. [observability] enabled = true in wrangler.toml plus Logpush is the platform answer, and it’s a higher-leverage spend than any logging library.

Metrics. No counters, no histograms. Workers Analytics Engine is the native option.

The comparison

	k8s + Node/Go	Cloudflare Workers
Log shipping	fluentd/vector sidecar	console.log → platform
Correlation	OTel SDK + context propagation	AsyncLocalStorage, hand-rolled
Live debugging	kubectl exec, attach profiler	Nothing
Metrics	Prometheus scrape	Analytics Engine
Tracing	Jaeger/Tempo + auto-instrumentation	otel-cf-workers, manual setup
Cost	A sidecar per pod	Included, sampled

The middle row is the one to sit with, because it explains why this work happened earlier here than it ever would have on k8s. For my whole career, live debugging was the safety net under every logging decision I half-made—whatever I forgot to log, I could go dig out of a running pod. Cloudflare cut the net down. When you can’t shell in, logs stop being a nice-to-have and become the entire interface to production.

When you can’t shell into production, your logs stop being commentary and become the only witness.

And here’s the opinion I’ve argued myself into over ten posts: that constraint is good for the codebase. Every platform in this series took something away—transactions, pods, a shell—and every time, the removal forced work forward that I used to defer until the first incident made it urgent. The observability work is the cleanest example. On k8s I would have shipped with 58 orphaned console.error calls and paid for them one outage at a time. Here the platform priced the debt honestly, up front, and so I paid it up front. That’s the series in one sentence: the edge doesn’t make engineering easier, it makes procrastination more expensive—and it turns out most of what I called engineering discipline was just debt the old platforms let me hide.

---

Previous: Six Security Findings from One Day of Code Review