How Synthetic Players Used ActiveGraph to Verify 4,919 Runs Without New Model Calls

When a language model takes part in an experiment, its reply is data. Calling the model again does not reproduce the original reply; it collects a new observation.

That distinction shaped the infrastructure for Synthetic Players, my study of persona-conditioned language models in strategic games. The National Academies defines computational reproducibility as obtaining consistent results with the same data, code, methods, and conditions of analysis. Work on reproducibility in machine learning similarly emphasizes preserving code and data so that reported results can be checked (Pineau et al., 2021). In this project, the original model responses had to be part of that preserved data.

I used ActiveGraph to create that record. The project depends directly on version 1.10.0. ActiveGraph ran the games, stored the event history, and provided the replay, fork, diff, and trace operations used by the research pipeline.

The study itself asked a separate question: can a panel of sixteen one-sentence GPT-4.1 personas preserve response to an experimental treatment, rather than merely produce plausible averages? The answer was mixed. The panel met several coarse, human-referenced checks, but its estimated response to a continuation-probability treatment was imprecise. In another experiment, changing one sentence and its position moved cooperation from 0/40 to 37/40. The paper does not claim that the personas substitute for people.

The event record matters because those conclusions depend on the exact prompts, responses, parsing rules, actions, payoffs, exclusions, and analysis code used in each run.

Where ActiveGraph sat in the system

Synthetic Players was not built entirely inside ActiveGraph. It used a TypeScript application for study management and a Python service for execution:

Browser and batch runners
          |
          v
Express API ----------------------> PostgreSQL
  study orchestration,              experiments, rounds,
  analysis, adjudication            analyses, claims
          |
          v
Python FastAPI service
          |
          v
ActiveGraph Runtime -------------> SQLite event store
  game execution, model calls,      prompts, responses, actions,
  replay, fork, diff, trace          payoffs, failures, metadata

Express remained the only PostgreSQL writer. The TypeScript code computed the study metrics and applied the registered claim predicates. A localhost-only Python service owned the ActiveGraph runtime and its SQLite event store. Its TypeScript client exposed a small HTTP interface and returned an engineRunId for every simulation.

This kept the responsibilities clear. ActiveGraph recorded execution. PostgreSQL held application records. The analysis code interpreted completed runs. Pre-registration, statistical validity, and claim adjudication were implemented around the engine, not delegated to it.

Recording a repeated game as events

Event sourcing records state changes as an ordered sequence of events from which prior or current state can be rebuilt. In Synthetic Players, a run began with a graph object containing the game definition, both player strategies, the round count, and the seed.

An ActiveGraph behavior listened for round.requested. It reconstructed the prior history, obtained both actions, calculated the payoffs, created a round object, and emitted round.played. If more rounds remained, it requested the next one. The implementation is compact; this is an abridged version:

@behavior(name="round_player", on=["round.requested"])
def round_player(event, graph, ctx):
    # Reconstruct history and the seeded random stream.
    # Resolve both actions and calculate their payoffs.

    graph.add_object("round", round_data)
    graph.emit("round.played", round_data)

    if round_number < num_rounds:
        graph.emit("round.requested", {"roundNumber": round_number + 1})
    else:
        graph.emit("run.completed", summary)

The engine created a Graph and Runtime, emitted the first request, and processed events until the run completed:

graph = Graph(run_id=run_id)
runtime = Runtime(
    graph,
    behaviors=[round_player],
    persist_to=event_store_url,
)

graph.emit(
    Event(
        id=graph.ids.event(),
        type="round.requested",
        payload={"roundNumber": 1},
        actor="engine",
    )
)
runtime.run_until_idle()
runtime.save_state()

A conventional results table would contain the final actions and payoffs. The event store also retained the order of operations, the actor responsible for each event, and causal links between events. That history supported three different operations:

• Trace: list the events that produced a run.
• Fork: copy a run at a selected round.played event, change a strategy, and continue from there.
• Diff: compare the parent and fork, including their shared events, unique events, divergent objects, and first divergent round.

For deterministic strategies, the engine stored the number of pseudorandom draws consumed in every round. It could reconstruct the correct position in the seeded random stream without relying on process memory. A fork therefore reused the parent's history through the fork point and continued with a reproducible random stream.

Deterministic replay and model-response replay are different

For a deterministic player, replay can recompute an action from the game state, strategy, and seed. A sampled model response cannot be treated the same way.

During a live model run, the engine emitted two linked events for every decision:

• llm.requested stored the rendered system and user prompts, prompt hash, model, temperature, token limit, prompt-registry hash, seat, round, and attempt.
• llm.responded stored the raw response and available provider metadata. Its caused_by field referenced the request event.

The parser then converted the recorded text to an action. That action produced the round payoff and the history shown in the next prompt.

Replay made no network request. The replay function did not instantiate a provider. It loaded the saved events, rebuilt an ActiveGraph LLMCache, rendered each prompt again from the versioned registry, and fetched the original response by prompt hash:

events = list(store.iter_events())
cache = LLMCache.from_events(events)

prompt = build_prompt(...)
response = cache.get(prompt.hash())

if response is None:
    mismatches.append("rebuilt prompt was not in the recorded cache")

The verifier reparsed the stored response and recomputed the actions, payoffs, random-draw counts, and study metrics. A changed prompt failed the hash lookup. A changed parser, game definition, or metric could produce a mismatch. Repeating the model call was neither necessary nor permitted on this path.

This follows a standard event-sourcing rule for external queries: if an external result affects later state, preserve that result and use it during replay. In this case, the external result was the model completion.

The distinction also constrained counterfactual analysis. A fork could not keep a model-controlled seat after changing the preceding history. Reusing its recorded choices in a context it never saw would invent counterfactual model behavior. Forks either retained the original history or replaced the model seat with a deterministic strategy.

Failed calls stayed in the record

The project allowed one retry when a response could not be parsed. A second failure emitted trial.invalidated. If a provider failed after some requests had completed, the run returned its partial call and token counts. The runtime saved its state in a finally block, including on invalid trials and provider failures.

This separated exclusion from deletion. Invalid trials did not enter the study evidence, but they still counted against the registered budget. Failed requests remained available for audit. Replacement runs could be connected to the trials they replaced. A reviewer could also verify that an incomplete response was never parsed into an action.

Provider adapters and a provenance gap

The project used ActiveGraph's OpenAIProvider directly. ActiveGraph 1.10 did not include a Gemini provider, so Synthetic Players implemented the LLMProvider interface for Gemini. A second OpenAI adapter retained metadata returned by the proxy used in later phases. Both adapters produced the same internal request and response types, which allowed the replay code to remain provider-independent.

The first version of the provider path was not complete. Phase 3 did not retain provider response IDs or a deterministic hash of the serialized request body. Phase 4 and Phase 5 added both. The event log faithfully preserved the fields supplied by the adapter; it could not supply fields that had never been recorded. The released paper discloses this gap rather than treating later instrumentation as if it had existed from the start.

What the public verifier checks

The final event store contains 5,505 completed runs, 54,276 rounds, 108,552 seat decisions, and 36,251 provider-request events. The repository includes a one-command verification capsule:

git clone https://github.com/yoheinakajima/synthetic-players
cd synthetic-players/capsule
bash verify.sh

It checks 4,919 archived Phase 3–5 runs without credentials or live model calls. Of those, 4,916 are confirmatory runs and three are retained legacy diagnostics. The verifier replays 4,896 model runs byte for byte and independently recomputes 20 deterministic baselines.

The replay establishes that the archived prompts, responses, actions, payoffs, and metrics produce the released computational results. It does not establish that a prompt measures the intended construct, that a statistical test family is correct, or that the results generalize to people or other model configurations.

That limitation was consequential. External review found errors involving test families, dependence, and construct validity after the runs were complete. The saved execution record allowed the analysis and interpretation to be corrected without changing the observations. The project retains the earlier adjudications and manuscript versions alongside the corrections. Twelve registered author predictions were refuted and remain in the public record.

Implementation choices I would reuse

Five choices proved useful beyond this particular study:

1. Store the exact request and response when the call occurs. A later reconstruction is weaker and may omit fields that seemed unimportant at the time.
2. Hash the rendered prompt. A template name does not capture substitutions, history, retries, or formatting.
3. Keep replay offline. If replay can reach the provider, it can accidentally create new observations or incur new costs.
4. Record failures and exclusions. They affect budgets, denominators, replacement rules, and the credibility of the final data set.
5. Keep procedural verification separate from scientific judgment. The runtime can verify execution. It cannot decide whether the research design supports the claim.

ActiveGraph was useful here because the study needed a durable execution history, not because the analysis itself required a graph database. Each response changed a game state and influenced later prompts, so the sequence and causality of events mattered. The stored history made those transitions inspectable and allowed the released results to be checked without contacting a model provider.

For systems that use model outputs as research observations, evaluation inputs, or evidence for later decisions, the same principle applies: preserve the original external response and the state changes it caused. Code and seeds alone are not enough.

Links: Synthetic Players repository · Project Q&A · ActiveGraph on PyPI · ActiveGraph documentation