Splits

Takeaways

Replace this with real notes — route, weather, effort level, and anything to adjust for next week’s long run.

It recorded the real time. Not close to the fake time, not off by a rounding error — the actual, unmodified wall clock, on the very first write, as if injection hadn’t happened at all. In a real system, that’s not a cosmetic bug — it’s every end-of-day ledger entry silently timestamped as having happened on the wrong day, in the one place (the datastore of record) where that’s the hardest kind of wrong to notice after the fact.

Reproducing it before assuming Docker was the problem

My first instinct was to suspect the environment — I was running this in a container with --cap-add SYS_PTRACE, and containers are where ptrace-adjacent things go to break in surprising ways. I ruled that out directly: same result with --privileged, no cgroup or namespace change made any difference, and — more convincingly — every test I already had, including ones exercising the exact same ChildTracker/fork-following machinery from the previous post against fanned-out Go workers, was green throughout. Whatever this was, it wasn’t Docker, and it wasn’t the fork-tracking logic. It was specific to this one target in the pipeline.

Finding the actual gap

The thing that was different about Postgres wasn’t that it forked (my tests already covered fan-out workers) — it’s that it wasn’t a Go binary. Every test I’d written up to this point injected into Go processes, either test/targets/clockprinter or Go stand-ins for the services in the pipeline. Go’s runtime reads the wall clock through clock_gettime. Postgres’s C code doesn’t; GetCurrentTimestamp() — backing now(), and therefore backing every timestamp the ledger’s closing entries would have gotten — calls gettimeofday.

I went back to the vDSO discovery code and actually checked, with readelf -sD on a dumped vDSO, exactly what clock_gettime, gettimeofday, and time are. I’d been treating them as effectively the same thing wearing different names — three ways of asking the kernel “what time is it,” so surely patching one covers all three.

They aren’t the same thing. They’re three independent compiled functions, at three different addresses in the vDSO, each with its own entry point. My injector patched exactly one of them — clock_gettime — with the JMP that redirects into my trampoline. gettimeofday and time were completely untouched, on every single injection I had ever run. Any caller going through clock_gettime — which included every Go service in my pipeline tests, since that’s what the Go runtime uses — saw the fake time perfectly, which is exactly why the ledger service itself correctly believed the day had ended. Any caller going through gettimeofday — Postgres, and as I found while checking, glibc and bash’s own wall-clock reads like $EPOCHREALTIME too — saw completely real time, silently, with no error of any kind. The service deciding whether to close the day was fooled correctly; the datastore recording when it closed the day was not — and nothing in the pipeline would have told you the two disagreed unless you went looking.

The fix, and why it isn’t three times the previous work

The trampoline already had the right shape for this — one shared state struct, read by a stub. The fix was writing two more stubs of the same shape:

• gettimeofday_entry — same structure as the clock_gettime stub, adding the offset and writing back a timeval (microseconds) instead of a timespec (nanoseconds) — offsetNsec / 1000.
• time_entry — this one had its own trap. My first version called the real time() syscall internally and added the offset to the result. But the time() syscall only returns whole seconds — it discards the real fractional second before my stub ever sees it — so time_entry could disagree with the other two by up to a full second depending on where in the second the real call landed. For a ledger entry, a one-second disagreement between the service’s timestamp and the database’s timestamp is exactly the kind of thing that turns into a confusing off-by-one-second ordering bug in an audit trail. The fix was to have time_entry call clock_gettime(CLOCK_REALTIME) internally instead, apply the identical carry-normalized offset add the other two stubs use, and only then truncate to whole seconds. Now all three agree to the second, always.

pkg/vdso.Locate grew resolution for all three symbols instead of one, pkg/inject now performs three JMP patches instead of one during the injection sequence, and all three stubs share the same 32-byte state struct — one SetTime, Freeze, or Advance call updates the fake time for all three functions in a target process at once, keeping the ledger service and its datastore consistent with each other by construction rather than by convention.

While I had the vDSO symbol table open, I fixed two smaller things in the same pass: clock_gettime_entry now also catches CLOCK_REALTIME_COARSE (same wall clock, lower precision, a distinct clock ID some callers request), and I confirmed — by dumping the full vDSO symbol table, not just the three I already knew about — that clock_getres, getcpu, and __vdso_sgx_enter_enclave are the only other exports on x86-64, and none of them read current time. Three wall-clock functions really is the complete list.

Why this needed a real pipeline to find, not a better unit test

I could have written a unit test asserting gettimeofday gets patched, once I knew to ask the question. What I couldn’t have done is think of the question from inside my own test suite, because every test I’d written was, by construction, exercising services I controlled — and I’d only ever written Go test targets standing in for them. The bug wasn’t a logic error I could have caught by trying harder at the same kind of test; it was a wrong assumption (clock_gettime and gettimeofday are basically the same call) that no amount of testing against Go binaries would ever surface, because Go binaries never call the path where the assumption was wrong. A pipeline test that only ever used Go stand-ins for “the database” would have stayed green forever, confidently testing the wrong thing.

That’s the real argument for testing a distributed pipeline against its actual dependencies before trusting the fake-time story end to end: not “more tests,” but a fundamentally different, non-Go caller in the loop, exercising a code path your own assumptions never thought to route around. pg-faketime-test — a separate project running a real initdb/postgres cluster under WithChildTracker and asserting SELECT now() tracks the faked clock — is what actually caught this, and it’s stayed in the loop since as the check that a Go-only pipeline test structurally cannot replicate.

This is also, for now, where the series pauses. pkg/faketime covers testing a pipeline like this end to end on a single machine; the other half of epochd — turning the same injection mechanism into a Kubernetes-native agent and controller for shifting time across whole pods in a real cluster — is its own story, for another time.

The fix has to be automatic. A test asserting an end-of-day batch job’s fanned-out workers all agree on the simulated cutover instant shouldn’t need to know, or care, that each worker is a process that came into existence after the fake clock was set. So pkg/faketime grew a mode that watches the process tree and injects into new children as they appear, with no action required from the caller beyond opting in.

Watching for fork and exec with ptrace options

Linux’s ptrace has options for exactly this, set via PTRACE_SETOPTIONS:

• PTRACE_O_TRACEFORK (and TRACEVFORK) — when the tracee calls fork/vfork, the tracer gets a stop event carrying the new child’s PID, and the child starts life already ptrace-stopped, before it runs a single instruction.
• PTRACE_O_TRACEEXEC — when the tracee calls exec, the tracer gets a stop event before the new program image starts running.

Both matter here for different reasons. TRACEFORK is the direct answer to fan-out workers and to Postgres’s connection-per-backend model — a new process appears mid-pipeline, and I need to inject before it does any work, including the very first clock read it makes to decide what “today” is. TRACEEXEC covers a related case: a batch-job launcher that re-execs itself into the actual worker binary, or forks and then immediately execs a different program. Without catching the exec event, an injection performed before the exec would just get thrown away — the new program image replaces the entire address space, vDSO included, so the trampoline page and the JMP patch are both gone, along with everything else the old process had mapped. A worker that silently reverted to the real wall clock mid-fan-out is exactly the kind of bug that would show up as one shard’s ledger entries landing on the wrong day.

ChildTracker: the piece that owns “keep injecting as things appear”

This became ChildTracker, reachable through StartWithTracking:

ct, err := faketime.StartWithTracking(exec.Command("./end-of-day-batch"), target)
defer ct.Reset()
defer ct.Close()

// advances the launcher AND every currently-tracked forked worker in one call
if err := ct.Advance(24 * time.Hour); err != nil { ... }

fmt.Println(ct.PIDs())                       // launcher PID first, then workers
fmt.Println(ct.Handle.EffectiveTime())        // same accessor pattern as a plain Handle

Internally, a background goroutine calls a non-blocking wait (WaitAnyNonBlocking in pkg/procmem) in a loop, watching for fork/exec stop events from any tracked PID. On a fork event it reads the new child’s PID via PTRACE_GETEVENTMSG, arms the same tracer options on it, injects the same fake-time state, and adds it to the tracked set — so a worker forked to handle shard 47 sees exactly the same simulated end-of-day instant as the launcher and every other shard’s worker. On an exec event it re-injects into the same PID, since the exec wiped out whatever was there before.

Session gained the equivalent as an option rather than a separate type — faketime.NewSession(target, faketime.WithTracking()) — because the same need (auto-inject into anything that appears, across every process in the pipeline) applies just as much to a multi-service Session as to a single tracked launcher.

The failure mode that shaped IsAlive and Prune

Tracking children well surfaces a problem that a single untracked process never has: forked workers, and forked database backends, die on their own, all the time, as a completely normal part of the pipeline finishing its work. Every worker that finishes its shard exits. Every client that disconnects from Postgres ends its backend process. A SetTime call that assumes every tracked PID is still alive will hit ESRCH (“no such process”) the moment it reaches a handle for a worker that finished five seconds ago — right when a test is trying to advance the clock again for the next simulated day.

I didn’t want callers to have to defensively check IsAlive() before every clock update just to avoid a spurious error from a worker that was supposed to exit. So applyAll — the internal helper both Session and ChildTracker route bulk updates through — silently drops any handle that returns ESRCH, and Session.Prune() exists for callers who want to know the count without waiting for the next update to discover it. IsAlive() from a couple posts back turned out to be exactly the primitive this needed underneath, and IsFrozen() followed the same pattern once assertions in tests started wanting to check current mode, not just current time.

With fanned-out workers and dead handles handled gracefully, the theory was that pkg/faketime was ready for its actual target: a real end-to-end run against a real Postgres cluster backing the ledger, not a synthetic Go test binary standing in for it. It mostly was. The part that wasn’t is the subject of the next post, and it’s the bug I’m least proud of and most glad was found before it silently corrupted a test’s idea of when something happened.

One process at a time, first

The base primitives map directly onto the two clock modes the trampoline supports:

h, err := faketime.Start(cmd, target)        // advancing: clock ticks forward from target
h, err := faketime.StartFrozen(cmd, target)  // frozen: clock is pinned at target forever

Both return a *Handle, and both share h.Advance(d) (shift forward by a duration, regardless of mode), h.Freeze(t), and h.SetTime(t). A single service’s scheduling test doesn’t want to think about “am I in frozen or advancing mode,” it wants to say “move forward one day” and have that mean the same thing either way — whether the test is simulating a single instant (frozen, useful for asserting exactly what the pipeline does at the cutover) or letting time keep ticking forward from a chosen point (advancing, useful for watching a sequence of triggers fire in order after the jump).

Session: the part that actually matters for a pipeline

A real end-of-day flow is rarely one process. It’s a ledger service, whatever it talks to, and the datastore underneath — each independently deciding, from its own time.Now() calls, whether the day has ended. If I advance each one separately, one SetTime call at a time, there’s a real window where one service believes it’s tomorrow and another still believes it’s today — exactly the kind of skew that doesn’t happen (or happens for microseconds, not test-visible seconds) when a real day actually ends, but very much can happen if a test advances four processes’ clocks one after another with unrelated Go function calls in between.

Session exists to close that window:

s := faketime.NewSession(target)
if err := s.Start(ledgerCmd); err != nil { ... }
if err := s.Start(reconciliationCmd); err != nil { ... }
if err := s.Start(notifierCmd); err != nil { ... }

// every tracked process jumps together — writes are issued out
// concurrently so the skew between them is as small as it can be
if err := s.Advance(24 * time.Hour); err != nil { ... }

Internally this is still just one process_vm_writev per process — there’s no cross-process transaction, no lock stepping the writes together at the kernel level. What Session.Advance does is fire all of those writes concurrently rather than one after another in a loop, which is the difference between a skew window measured in the time it takes several syscalls to complete versus one measured in however long the slowest step of a sequential loop happens to take. For a test asserting ordering between services — reconciliation shouldn’t start before the ledger closes — that’s the difference between a test that’s reliably testing the real ordering logic and one that’s occasionally just testing how fast your test harness happens to be that run.

The methods that came from actually writing pipeline tests, not from a design doc

Start, Advance, and Session were there from the beginning. Handle.EffectiveTime(), Handle.PID(), Handle.IsAlive(), and Session.Close() were not — they came out of a backlog I only accumulated once I started writing real multi-service tests and kept reaching for something that wasn’t there:

• EffectiveTime() time.Time — for asserting “what time does this service currently believe it is” without redoing the offset arithmetic in the test itself. Useful on its own, and essential once a test is checking that two services in the pipeline agree with each other, not just that either one individually jumped.
• PID() int — needed to correlate a handle with a log line from the actual service, when a pipeline test is trying to work out which process’s scheduler fired first.
• IsAlive() bool — a kill(pid, 0) check. A pipeline test that expects one service to shut down cleanly after finishing its end-of-day work wants to check that without accidentally erroring out the whole Session because one handle no longer points at a live process.
• Session.Close() — exists because a Session isn’t always just a bag of handles; once you add tracking (next post), it owns a background goroutine watching for forked children across every tracked process, and something has to stop that goroutine when the test ends.

None of these are exotic. That’s the point — they’re the small, obvious things you don’t notice are missing until you’re mid-test, watching four services’ worth of output, and reach for them.

Cleanup that survives a pipeline test failing halfway through

The rawest form of the API still leaves cleanup up to the caller — kill each process, wait on it, reset its clock. For a single service that’s a three-line defer block; for a pipeline of several, it’s easy to leak one if the test fails partway through starting them. So the idiomatic entry points wire cleanup through t.Cleanup, which is the one mechanism in the standard testing package that reliably runs even when the test body panics or fails early:

faketime.WithSession(t, target,
    func(s *faketime.Session) error {
        if err := s.Start(ledgerCmd); err != nil { return err }
        return s.Start(reconciliationCmd)
    },
    func(t *testing.T, s *faketime.Session) {
        if err := s.Advance(24 * time.Hour); err != nil { t.Fatal(err) }
        // assert both services agree the day has ended
    },
)
// every started process is killed, waited, and reset here — even if
// the assertion callback called t.Fatal partway through

A leaked, fake-time-shifted service sitting around after a pipeline test finishes is a much worse failure mode than a verbose defer block, because it fails silently in CI and shows up as a mystery in some other test’s flakiness later.

Everything above assumes each process in the pipeline stays exactly one process for its whole life. That assumption breaks the moment any of them forks — which the datastore behind a real ledger pipeline (Postgres, forking a new backend per connection) does immediately. Making injected fake time survive a fork, and an exec after that, is the subject of the next post.

Step 1: get the target a page of memory it doesn’t know it has

A JMP rel32 instruction can only reach ±2GB from where it’s placed, so the trampoline needs to land close to the vDSO — and I need the target process itself to allocate that memory, since I can’t map pages into another process’s address space from the outside. The trick is to make the target call mmap on my behalf, without it ever running any code of its own to do so:

1. Scan /proc/<pid>/maps for a free address within ±2GB of the vDSO — close enough for the eventual jump to reach.
2. Ptrace-stop the target. Temporarily stash three bytes at the clock_gettime entry point (safe — I haven’t patched it yet, and I restore these bytes immediately after) with 0F 05 CC: the syscall instruction followed by int3.
3. Set the target’s registers by hand — RAX = __NR_mmap, and the arguments for mmap(hint, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON|MAP_FIXED_NOREPLACE, -1, 0) — then resume the tracee with PTRACE_CONT.
4. The tracee executes the syscall, immediately hits the int3 I planted right after it, and traps back to me with SIGTRAP.
5. Read RAX — that’s the address of the freshly mapped page, chosen by the kernel to satisfy my hint. Restore the original three bytes and registers at the clock_gettime entry as if nothing happened.

This is the part of the codebase I’m proudest of, mostly because it’s the part that felt like cheating when it finally worked: the target process — mid-flight, doing whatever it was already doing in its pipeline — just ran a real mmap syscall, with real kernel bookkeeping behind it, and has no idea it did.

Step 2: write the payload into that page

With a rwx page now sitting in the target’s address space, I copy in the trampoline: three tiny hand-written x86-64 stubs (one each for clock_gettime, gettimeofday, and time — why three separate ones is a couple posts away) plus a 32-byte shared state struct, all pre-assembled from trampoline.asm into trampoline.bin and embedded directly in the Go binary. This step uses process_vm_writev, which needs no ptrace stop at all — just the ptrace relationship (or CAP_SYS_PTRACE) already established.

state struct, at offset 436 in the page:
  +0   int64  offsetSec
  +8   int64  offsetNsec
  +16  uint64 enabledMask   // 1 = advancing, 3 = frozen
  +24  uint32 generation    // bumped on each update
  +28  uint32 _pad

offsetSec/offsetNsec are written pre-populated with the fake time’s offset from real time before the page is ever wired up, so there’s no window where the trampoline exists but reads garbage state — no window where a scheduling loop in the target could read a half-written clock and misfire.

Step 3: patch the vDSO to jump into it

Now the actual hook: for each of the three functions, ptrace-stop the target again and use PTRACE_POKETEXT to overwrite its first five bytes with E9 <disp32> — JMP rel32 — targeting that function’s stub offset inside the new page. PTRACE_POKETEXT is what makes this possible at all despite the vDSO being mapped read-only-but-executable; it’s the same primitive a debugger uses to plant a breakpoint byte in your .text section, just writing five bytes of jump instead of one byte of int3.

Step 4: detach, and never come back (for reads)

The target resumes. From here on, every call to clock_gettime, gettimeofday, or time lands in the trampoline stub first, which does the real syscall (or, for time, an internal clock_gettime; more on why in the Postgres post), adds the offset from the shared state struct, and returns — all without a single ptrace round-trip. Updating the fake time later — SetTime, Advance, Freeze — is just another process_vm_writev write of a new 32-byte state struct. No stop, no signal, no coordination with whatever the target happens to be doing at that moment — no risk of catching a scheduling loop mid-check and leaving it in a torn state.

That last property is the entire payoff of everything above: the cost of changing the fake time, for the rest of the process’s life, is one bulk memory write — and when the pipeline has several processes that all need to leap forward to the same end-of-day instant together, that’s one bulk write per process, issued back to back, rather than a coordination protocol between them. Compare that to the ptrace-every-call approach from two posts back, where every single read, on every process, carried the overhead this whole design exists to avoid.

The mechanics above are what make pkg/inject work in isolation, on a single process. Turning that into an API that’s actually pleasant to use from a Go test — starting several services, advancing them together, cleaning up automatically when the test ends — is a different kind of design problem, and it’s what the next post is about.

Every process on Linux gets a small shared library mapped into its address space automatically, without ever calling mmap for it — the vDSO, “virtual dynamic shared object.” You can see it in any process’s memory map:

$ cat /proc/self/maps | grep vdso
7ffd8a1fe000-7ffd8a200000 r-xp 00000000 00:00 0  [vdso]

Glibc’s wall-clock functions — clock_gettime, gettimeofday, time — don’t trap into the kernel through the syscall instruction at all. They call into this mapped vDSO region, which reads the kernel’s timekeeping data directly out of a shared memory page. No context switch, no scheduler involvement, just a function call and a couple of memory reads. That’s the whole reason these calls cost ~20ns instead of the ~200ns-plus a real syscall would cost, and it’s exactly why a ptrace-per-call approach (the previous post) adds four to five orders of magnitude of overhead onto every service in a pipeline, all day, just so one call a day can eventually be faked.

Here’s the insight that makes injected fake time practical: the vDSO is just a normal mapped region of executable code, sitting in the target’s own address space. It happens to be mapped read+execute, not read+write+execute — but a region being marked read-only doesn’t mean ptrace can’t write to it. PTRACE_POKETEXT — the same primitive debuggers use to plant breakpoints in your program’s .text section — can write to any mapped page in a ptrace-stopped tracee, read-only or not. Debuggers rely on exactly this to insert an INT3 byte at a breakpoint address; nothing distinguishes the vDSO’s pages from any other executable page as far as PTRACE_POKETEXT is concerned.

So instead of intercepting every call, forever, in every process in the pipeline, the plan becomes:

1. Stop the target process once, briefly, with ptrace.
2. Overwrite the first few bytes of clock_gettime (and, it turns out, two other functions — more on that in a later post) inside the vDSO with a jump instruction to code I control.
3. Detach. The process resumes and keeps running exactly as before — logging, scheduling loop, request handling, all of it — except now every call to those functions redirects through my code first.
4. Never touch ptrace again for the rest of the process’s life, unless I want to change the fake time. Updating the fake time afterward — say, jumping the whole pipeline forward by a day to trigger the end-of-day run — is a single process_vm_writev per process. No ptrace stop required at all.

The cost of injection becomes a fixed, one-time price paid once at startup, and the cost of every subsequent clock read is back to native vDSO speed — it’s still just a function call into mapped memory, just a different one. This is the difference between “slow down every process in the distributed system, all day, for one eventual trigger” and “pay a millisecond once per process, then advance the whole pipeline with a handful of bulk writes whenever the test wants the day to end.”

There’s an obvious follow-up question this raises: a jump instruction has to jump somewhere, and that somewhere needs to be memory I control, close enough to the vDSO to be reachable, containing code that knows how to fake a timestamp and then get out of the way. Building that — the trampoline — is the subject of the next post.

Mechanically, this is completely possible. PTRACE_ATTACH lets you stop a process at every syscall entry and exit (PTRACE_SYSCALL), inspect its registers with PTRACE_GETREGS, and mutate them with PTRACE_SETREGS before resuming it. Catch a clock_gettime on entry, let it run, catch the exit, and rewrite the timespec it just wrote into the caller’s buffer via process_vm_writev. Repeat forever.

I got a proof of concept working. Then I pointed it at a real service — the kind that’s supposed to sit around quietly for hours doing nothing until an end-of-day trigger fires — and watched it fall over.

The problem is volume, not mechanics. A service that “only” does something once a day is still, moment to moment, calling the wall clock constantly: a scheduling loop checking the cutoff time, the Go runtime’s own internal timekeeping for the scheduler and GC, logging timestamps, request-handling code that stamps every event it processes. None of that is specific to the once-a-day trigger I actually care about — it’s just the ordinary background hum of a running service — but it means the wall clock gets read tens of thousands of times per second even while nothing “interesting” is happening. Under the ptrace-every-syscall approach, every one of those reads means:

1. Stop the tracee (a context switch into the kernel, and a wakeup for the tracer).
2. Wake the tracer process, have it call waitpid, then PTRACE_GETREGS.
3. Decide what to do, then PTRACE_SETREGS and PTRACE_CONT.
4. Wake the tracee back up (another context switch).

That’s two full scheduler round-trips per clock read. A clock_gettime call that would normally cost ~20-200ns turns into something costing tens of microseconds — a three-to-four-order-of-magnitude slowdown, applied to a function a real service calls constantly just to exist, long before the specific batch trigger I’m trying to test ever fires. I tried it against test/targets/clockprinter — a trivial program that just prints time.Now() in a loop — and it was already visibly janky. Against a real service under any load, it was worse.

There’s a second problem layered on top, and it matters more for a distributed-systems test than a single-process one: this approach requires the tracer to stay attached and responsive for the entire lifetime of every process in the pipeline, for as long as the test runs. Test an end-of-day pipeline with three or four services plus a database, and that’s three or four processes’ worth of clock reads all competing for the same tracer’s attention at once, for the full duration of a test that might be simulating several simulated days back to back. If the tracer falls behind on any one of them, every process it’s attached to slows down in lockstep, which is exactly the kind of test flakiness that erodes trust in a test suite until people stop running it.

That reframing — intercept once, then get out of the way, for however many processes the pipeline has — is what sent me looking for a different mechanism entirely. If I could make the first clock read after injection permanently different, without needing to be in the loop for every subsequent one, the whole performance problem disappears, for one process or a dozen. That’s exactly what Linux’s vDSO makes possible, and it’s the subject of the next post.

Testing this properly means testing the actual trigger, not just the handler it eventually calls. Calling RunEndOfDaySettlement() directly in a test tells you the settlement logic is correct. It tells you nothing about whether the scheduler that’s supposed to call it actually fires at the right moment, across every service in the pipeline, in the right order, without one service’s clock disagreeing with another’s about whether the day has actually rolled over. That gap — between “the handler works” and “the system correctly decides to invoke the handler” — is exactly where I’ve seen real incidents come from.

The two conventional ways to close that gap both have a problem:

• Wait for real time to pass. Technically correct, completely unusable in CI. Nobody is running a test suite that blocks for 24 hours.
• Give every service an injectable clock — a Clock interface, a test-only flag, a debug endpoint that sets the time. This works for code you own and control completely. It falls apart the moment the pipeline includes anything you don’t own the source of — the database backing the ledger, a message broker, a vendored scheduler binary — none of which ships a “pretend it’s tomorrow” hook. And even within your own services, any code path that reads time in a way the injected clock doesn’t cover (a library’s internal timer, a syscall issued somewhere you didn’t thread the interface through) quietly falls out of sync with the rest of the system, which is worse than not faking it at all — the test looks like it’s exercising real timing behavior and isn’t.

What I wanted instead: take the real, compiled, unmodified binaries — every service in the pipeline, and the datastore underneath them — and shift each one’s actual OS-level wall clock forward, from outside the process, with zero source changes and zero test-only code shipped into production. If a service decides “the day changed” using an ordinary time.Now() call somewhere deep in a library it doesn’t control, injected fake time makes that call return tomorrow, and the service’s own real scheduling logic does the rest — I never call its internal handler myself. And because the injection happens at the OS level rather than the application level, I can advance every service in the pipeline together, so the whole distributed system crosses midnight at (approximately) the same instant — the way a real day boundary actually arrives, not staggered by whichever service’s test harness happens to poke it first.

That’s the shape of the problem epochd’s pkg/faketime solves. A test for an end-of-day pipeline ends up looking roughly like this:

func TestEndOfDaySettlementFires(t *testing.T) {
    ledger := exec.Command("./ledger-service")
    target := time.Now()

    faketime.WithProcess(t, ledger, target, func(t *testing.T, h *faketime.Handle) {
        // jump the whole service straight to just before midnight
        if err := h.Advance(24 * time.Hour); err != nil {
            t.Fatal(err)
        }
        // the service's own scheduler — untouched, unmodified — now believes
        // the day has ended, and fires settlement on its own
        assertSettlementRan(t, ledger)
    })
}

No sleeping, no shortened “test day” config, no handler called directly. The service under test is a real, unmodified binary, deciding for itself that time has moved on.

Getting from “I want this” to “this reliably fools a real service’s scheduler, and every other process downstream of it in the pipeline” took going further down the stack than I expected — ptrace, the vDSO, hand-assembled trampoline code, and eventually a real Postgres cluster that exposed a gap none of my own test binaries ever would have. This series is that build, in the order it actually happened.