SCOTUS Reasoning Study
Pre-registered, in progress
Judicial opinions are documented reasoning. The author states what they decided and why, at length, over decades. That makes a court corpus one of the few places where a behavioral specification can be tested against a held-out record rather than against a judgment call. This study is running now. The design below is pre-registered. There are no results yet, and none are reported here.
The corpus (scotus.db) is assembled from the Harvard Caselaw Access Project and CourtListener. The subject is Justice David Brewer, whose 457 opinions are split by filing date at 1898-10-24 into 229 training opinions and 228 held out. Brewer was chosen precisely because models do not reliably recognize his cases. If the model already knew the subject cold, an apparent gain would be indistinguishable from recall, so a subject the model knows poorly isolates what the specification contributes from what the model brought with it.
That choice was measured rather than assumed. A probe gave the model nothing but case names, no opinion text and no specification, and asked whether it recognized them. On a well-known modern justice it scored 100 percent and recognized half the cases outright. On Brewer it scored at roughly chance and recognized one case out of twenty. The subject was selected because the meter said the model does not already know him.
The problem being tested is that a single specification is unfalsifiable. Read one specification against one held-out opinion and nothing distinguishes “this reads like Brewer” from “this reads like 1890s judicial prose.” A specification that only has to sound plausible cannot fail.
The design answer is comparison subjects. Two are being built alongside Brewer: Justice Harlan I as a documented contrast, and Justice Fuller as a same-era control. A blind judge is then given a held-out opinion and asked which specification predicts it. Era alone cannot win that test, because every candidate shares the era. Only a specification carrying something specific to its subject can.
The scoring rule was fixed before any comparison specification existed. 120 trials, 40 per justice, drawn from opinions no build ever saw. The judge sees three unlabeled specifications and one held-out opinion and picks one, so chance is 33.3 percent and the result is a binomial test against it. Presentation order is balanced across all six permutations so position cannot encode identity, and every justice name is stripped from both the specifications and the opinion before judging. Scoring above chance means the specifications discriminate. Scoring at chance means the pipeline is reading a genre rather than a person, and that outcome gets reported as it stands.
A practising attorney is reviewing the output blind. Mechanical scoring measures whether the held-out opinion was matched. It does not measure whether the reasoning would survive contact with someone who reads judicial opinions for a living. Blind review by a legal professional is the check on that, and it is run blind so the reviewer cannot tell which arm produced which text.
On provenance. Specifications in this study are provenance-traced, but the trace is not uniform. On the Brewer build the citation path returned zero usable links and vector provenance carried the trace on its own. Claims are traceable to supporting evidence. They are not all backed by a verbatim quotation, and the study does not claim otherwise.
What Moves a Specification
One discontinuity in three years
People change. If a behavioral specification is going to describe someone durably, we need to know which parts move and which parts hold. So we took one person’s corpus, cut it at seven points in time, and built a separate specification from each slice using only what existed by that date. Then we measured how far each specification sat from the next.
The hard part is that the authoring step is not deterministic. Build the same specification twice from identical facts and you get two slightly different documents. So every cutpoint was built three times, and the spread between those rebuilds became the noise floor. Nothing counts as change unless it clears the distance the system produces from doing nothing at all. Without that floor, ordinary regeneration variance reads as personal transformation.
Across roughly three years, one change stands out as a sharp discontinuity. The self-concept layer moved abruptly in a single quarter, at 2.3 times its threshold, when the subject wound down a startup and turned to research. That was a cliff, not the only motion. The behavioral layer drifted steadily the whole time, at a low level that cleared the floor at most steps without any single jump. The one thing that did not move at all was the axiom layer. A specification has one part that reorganizes at a life event, one part that shifts continuously, and one part that holds.
Email Against Chat
The operating layer moves, the axioms do not
Everything the pipeline had ever been built from was chat. That is a real worry: if a behavioral specification only describes how someone types to an assistant, it is a description of a habit, not a person. So we built one from a different medium entirely, four years of sent professional email, and asked whether the same person came out the other side.
Email did change the specification, and it changed the operating layer the most of anything in the study. The self-concept layer, how this person operates and presents, cleared the noise floor against every chat comparison, by the widest margins anywhere in the whole study. What the email version captured that chat never records: fundraising mechanics, a named team, scheduling constraints, and an explicit instruction to match register for external audiences. So the medium moves the surface a great deal. What it did not move at all was the axiom layer, what this person reasons from, which sat inside noise against all four comparisons. The reasoning core is the same person; the operating surface is not.
The disposition layer looked stable, but read that carefully. What this person avoids, enjoys, and is good at also sat within noise across the comparisons. That is weaker evidence than it sounds, because the email specification is about twice as variable internally as any chat one, so the noise floor it is judged against is high. Sitting inside a wide band is not the same as being demonstrably stable. Treat the dispositions as not-clearly-moved rather than proven invariant.
The confound is significant and we are not going to bury it. That separation is entangled with time, not just medium. The email corpus is weighted toward 2022 to 2024 and asserts stale facts in the present tense, because it was extracted from text frozen in that era. The tell is in the ordering: the gap is largest against the most recent comparison and smallest against the all-time one, which is the signature of an era effect rather than a medium effect. The fair comparison still clears, but by the narrowest margin in the table. There is a genuine register component here and it is smaller than the raw numbers suggest.
A correction this study forced. An earlier comparison of two chat platforms had found the source effect landing in the behavioral layer. That was measured between two platforms of the same medium. Across an actual change of medium it lands one layer up, in self-concept. The refined statement is that the deeper the source difference, the higher up the layer stack the effect reaches, and that it never reaches the axioms.
One honest side effect. The email specification is noticeably less reproducible than any chat-built one, roughly twice the internal variance. The corpus spans several registers at once, so the authoring step has more freedom and settles less consistently. Higher variance is itself a property of the material.
Serving Layer Evaluation
A null result on architecture
Once a behavioral specification exists, there is a second question nobody had answered: how should it actually be handed to the model? We had built increasingly clever machinery for this. Retrieve facts relevant to each question. Use a first model call to decide which parts of the specification the question needs, then a second call to answer. This study asked whether any of it beats simply pasting the whole thing in every time.
Null result. The clever machinery does not earn its complexity. All six formats land within 0.40 points of each other on a ten-point scale, and all eight pre-registered comparisons came back non-significant. Per-question fact retrieval added nothing. The two-stage router added nothing. Pasting in one block does as well as anything we built to replace it.
Where Self-Editing Memory Breaks
A result against ourselves
Letta takes a different route to the same destination. Rather than compressing a corpus in a pipeline, it reads material turn by turn and lets an agent continuously rewrite its own memory block. Two systems designed independently, both converging on the idea that what you want is an interpretation rather than a pile of retrieved facts. We ran it properly, on the same three subjects, at the same response model, and reported what came back.
Letta scored higher than our behavioral specification on all three subjects. By 0.27, 1.21, and 0.38 points. We are reporting this because it is what the data says. A self-editing agent loop, run properly, produces a representation that predicts behavior at least as well as the artifact this project exists to produce.
The ceiling is in scaling, not quality. Letta’s memory block grows roughly in proportion to the corpus. On the largest subject, roughly 223,000 source words, it hit a hard limit: the service began rejecting writes at around 333,000 characters, and the last 22 of 242 chunks, about a tenth of the corpus, never made it in. Our specification stayed between 34,000 and 40,000 characters across a ninefold range of corpus size, because compression is what the pipeline is for.
Coherence fails before size does, which is the more interesting finding. At the two smaller scales, verbatim sentence duplication inside the memory block was zero. At the largest, more than a quarter of sentences were exact repeats of another sentence. One sentence appeared twelve times. The consolidation loop starts looping on itself well before it runs out of room, which means the effective unique content is meaningfully smaller than the block size suggests.
Scope, stated plainly. Three subjects, one version of Letta, one response model. This is an exploratory case study and it is reported that way in the paper. It is not a benchmark result and should not be read as one. What it supports is a claim about convergence, that two independently designed systems arrived at the same target, and a claim about where the agent-loop approach runs into a wall.
Retrieval Divergence
Same facts, different retrieval
Before an AI answers a question it decides what information the question is about. If two systems pull different facts from the same store, their reasoning has already diverged before a single word is generated. We asked 50 questions across 10 life domains to two systems holding identical facts: a flat memory baseline that replicates Mem0’s published architecture, and a behavioral specification that interprets the question before retrieving. Same model, same temperature. The only variable was the system prompt.
The divergence pattern is a fingerprint, and it is different per person. A founder diverged most on relationships (4.75), Warren Buffett on finance (3.19), and Howard Marks on technical and decision questions (4.52 and 4.50). Each peak matches that subject’s known reasoning style. Nobody designed those peaks. They came out of the data.
Known Failure Modes
Where the system breaks
Every system has failure modes. We document ours publicly so you know what to expect, how we caught each problem, and what we did about it. Hiding failures doesn’t make them go away. Showing them builds the trust that lets you use the system seriously.
8 documented failure modes. Topic skew (fixed via 73-word guard), sycophancy amplification (verified via stacking study, mitigated architecturally), thin data overconfidence (partially fixed, density matters), cognitive anchoring (fixed via blind authoring), pronoun effects (open research question), extraction positional bias (fixed via chunking), ceremonial pipeline steps (10 of 14 cut after ablation), and provenance gap (fixed by re-adding Embed step).
Running Without an API
What local models can and cannot do
A system that reads your entire personal history should not require sending that history to a company. The pipeline uses paid models by default, so the question is what you give up by running it entirely on your own hardware. We ran it overnight on a single consumer graphics card and measured the gap.
The local path works for reading your data, not for writing about it. The best local model recovered roughly three quarters of the paid extraction quality at zero cost. Authoring is a different story: local output is structurally reasonable and reads flat. Extraction is the task worth running at home.
Model choice matters more than model size. The best throughput came from a 12B model pulling 10.8 facts per conversation in about six seconds. A 14B model found more, 14.8 facts, but took roughly seven times as long for the privilege. Meanwhile one popular 8B model was effectively broken for this task, returning 0.7 facts per conversation, and a 32B model would not fit in memory at all. Bigger is not the axis.
The right instruction set depends on the model. One 7B model nearly tripled its yield, from 3.0 to 8.5 facts per conversation, purely by being restricted to behavioral fact types instead of the full vocabulary. Smaller models do better when asked for less. Larger ones handle the full vocabulary and benefit from it. There is no single configuration that suits every model.
Where the local path is genuinely slow. Every local embedding option took about two seconds per query, which is far too slow to sit in a live serving path. The workaround is structural rather than a matter of model choice: compute embeddings once when the behavioral specification is built, not per question.
One caveat on reading these numbers. This run predates the current 46-predicate vocabulary and used an earlier 19-predicate set, so the absolute fact counts are not comparable to current output. The comparison between models, which is what the study was for, is unaffected. It also counts facts rather than scoring their accuracy, so it measures yield and not quality.
Behavioral Grammar
46 predicates, formally specified
Before predicates, 57% of extracted facts started with “The user is...”, generic LLM artifacts that inflated scores and wasted tokens. The fix: a constrained vocabulary of 46 verbs that force the extraction model to classify every fact into a structured triple: {subject, predicate, object}.
This is not a knowledge graph. It is a behavioral grammar, a finite set of verbs that can describe how any human thinks, acts, values, fears, builds, and relates. The vocabulary is organized into five categories: cognitive (believes, values, fears), behavioral (practices, avoids, builds), relational (collaborates, mentors, trusts), contextual (works_at, lives_in), and experiential (experienced, struggled_with).
46 predicates. Epistemic precision over convenience: “attended” is not “graduated_from,” “wants_to” is not “aspires_to.” Behavioral predicates (values, avoids, fears) are the most predictive for identity compression. Biographical predicates (works_at, lives_in) provide context but rarely discriminate.
Coding Agent Study
An honest null result
The drift study showed that structured descriptions change how an AI approaches problems. The natural next question: does that actually make it better at solving them?
We tested this on real software engineering tasks. We took 30 hard bug reports from a well-known benchmark (SWE-Bench) and gave an AI coding agent different kinds of help: design principles from the project, generic encouragement, principles from a completely unrelated project, or no help at all. Then we measured: did it fix the bug?
The AI with no extra help performed best. The bare baseline solved 37% of problems. Every condition where we injected design knowledge performed worse, including our best treatment at 30%.
Relevant and irrelevant knowledge produced identical results. Django principles on Django bugs, and machine learning principles on Django bugs, both scored 30%. The AI was equally unaffected by relevant and irrelevant information. This is the cleanest finding. It rules out “our principles were just poorly written.”
The AI didn’t actually use the information. When we reviewed the AI’s step-by-step reasoning, it acknowledged the design principles in its first message and then completely ignored them. On some problems, it wasted time writing summaries about how its approach aligned with the principles instead of actually fixing the bug.
Understanding how someone thinks helps AI work with people. It doesn’t help AI fix code. Those are different problems, and this study proved it. Base Layer is built for humans, not coding agents.
Behavioral Drift
Does format matter?
V4 BriefV4 was used in this study. V5 is the current version.When you teach an AI something new about a person, does it update the right behavior, or does it change everything randomly? Imagine telling your assistant “this person once got burned by over-engineering a project.” Ideally, that should change how the AI approaches software architecture decisions, but it shouldn’t change how it helps with debugging or security reviews.
We tested this across four different AI models, from free local models to expensive frontier APIs. We described the same person three different ways:
Flat preferences
“Likes simple code, prefers TypeScript, wants tests.” How most AI memory systems work today.
Structured reasoning
“Avoids premature abstraction because they’ve seen it fail. Requires three concrete cases before extracting a pattern.”
Narrative prose
A flowing description of the person’s approach. Same information, written as paragraphs.
The structured format won decisively. When the AI was given structured reasoning about why someone thinks a certain way, new information was routed to the correct behavior. An architecture lesson changed architecture behavior specifically, not debugging, not security, not everything at once.
Flat preferences produced random change. A list of likes and dislikes gave the AI no way to figure out which behavior a new piece of information should update. The change was scattered across every dimension equally, or missed the target entirely.
A free 7B model with the right format outperformed a frontier API model with the wrong format. The way you describe someone to an AI matters more than which AI model you use. This was the most surprising finding.
An AI that understands why you avoid over-engineering routes new lessons to the right place. An AI that just knows you “prefer simple code” can’t. How you describe someone to AI determines whether the AI can actually learn from new information about them.
Brief Optimization
31 versions tested
V5 BriefV5 is the current brief format — citation-stripped, cleaner prose.The final step of the pipeline takes everything we know about someone and writes a summary that other AI systems will read. How you write that summary, the instructions you give the AI that writes it, dramatically affects how useful the result is. We tested 31 different versions across 7 rounds to find what works.
The best summary is shorter, marks where it’s uncertain, and tells the AI when NOT to apply a pattern. Less confident, more useful.
Pipeline Simplification
Less is more
V4 BriefV4 was used in this study. V5 is the current version.We originally built a 14-step pipeline to turn conversations into an identity summary. Before shipping, we asked: which of these steps actually matter? We tested every single one by removing it and measuring what happened to quality.
10 of the 14 steps were unnecessary. Scoring, classification, contradiction detection, adversarial review. They all sounded rigorous. None of them improved the final output. Removing them actually made it better.
The 3-layer structure is essential. We split identity into three layers: what you reason from (your foundations), how you behave (your patterns), and testable predictions (things we can verify). Combining all three into one pass scored lower.
Raw facts without synthesis don’t work. Just dumping extracted facts into the AI without organizing them scored worst. The synthesis step, where facts become structured patterns, is where real compression happens.
Simpler is better. The ablation reduced 14 steps to 4 — quality went up. We later added a 5th step (Embed) for traceability, not quality. Most of the complexity we built was ceremony, not substance.
Compression & Format
How much data is enough?
V4 BriefV4 was used in this study. V5 is the current version.How much of someone’s conversation history does the system actually need? And does it matter whether the output is written as prose, bullet points, or a structured guide? We tested both questions.
The pipeline’s value is in compression, not accumulation. The best summary is short, behavioral (not biographical), and structured rather than narrative.
Prediction Benchmark
Can it predict real people?
V4 BriefV4 was used in this study. V5 is the current version.Can the system actually predict how a real person would respond to questions? We used a dataset of 100 real people, each with detailed descriptions of who they are. We compressed each description into a short summary and tested whether an AI could use that summary to predict the person’s actual survey responses.
The result
Our compressed summary (18x shorter) matched or outperformed giving the AI the entire description. On one model, the compressed version actually predicted better than the full dump, statistically significant at p=0.008.
Why compression works
A 130,000-character description contains a lot of noise: irrelevant details, repetition, tangents. Compressing it to 7,000 characters forces the system to keep only what actually predicts behavior. Less noise, more signal.
A compressed summary predicts real human responses better than giving the AI everything. Compression doesn’t lose signal. It concentrates it.
Quality Measurement
Testing our own work
V4 BriefV4 was used in this study. V5 is the current version.How do you measure whether a behavioral summary is actually good? We built five tests and ran them on a summary of Benjamin Franklin (extracted from his autobiography). Two passed, two failed, one couldn’t be measured. The failures taught us as much as the passes.
What passed
Claim traceability (99.98%): Nearly every claim in the summary traces back to something Franklin actually said or did. The system doesn’t make things up.
Signal retention: After heavy compression, the summary actually captures more of what matters than the raw source. Forcing brevity makes the system prioritize better.
What failed (and why that’s informative)
Adversarial resistance: The more accurately the summary captures someone, the easier it is to exploit their real contradictions. Accuracy and security are in tension. This is a real tradeoff, not a fixable bug.
Cross-model consistency: Different AI models interpret the same summary differently. Portability across models needs work.
Faithful summaries expose real tensions in someone’s worldview, and that makes them both more useful and more vulnerable. You can’t have perfect accuracy and perfect security. We think accuracy is the right trade.
Traceability
Can we prove it?
V4 BriefV4 was used in this study. V5 is the current version.Using an AI to judge another AI’s output is circular. You’re trusting the same kind of system you’re trying to evaluate. We built an evaluation framework where every result can be checked by a human, costs nothing to run, and produces the same answer every time. No AI judges.
If a human can’t check the claim, it’s not evidence. Every metric in this framework is verifiable without running an AI model.
Output Format
How should a brief look?
V4 BriefV4 was used in this study. V5 is the current version.The last step of the pipeline writes the final summary that other AI systems will read. We tested six different formats, from flowing prose to structured guides to dense shorthand, to find what makes a summary most useful.
Same information, restructured, is dramatically more useful. The winning format doesn’t just describe patterns. It tells the AI when not to apply them and how to resolve contradictions.
Design Decisions
80 decisions, all public
Every architectural choice is documented with reasoning, alternatives considered, and status. 78 decisions across 97 sessions. Most projects publish their code. We also publish why the code looks the way it does: every wrong turn, every superseded idea, every decision that survived. The full log is in the repository.
Architecture
Quality & Privacy
Evaluation Philosophy
What Didn't Work
Nothing is hidden. The prompts are in the code. The reasoning is in the log. We publish what didn’t work alongside what did.