Hypotheses about form
How should an agent remember, decide, act, and rest? Where should structure live in a learned model? Each hypothesis is written down before it is built.
The lab · a broader narrative
We research the architecture of intelligent systems. Then we build the systems that test it.
Think.dev is small on purpose. One principal, a handful of collaborators, two research programs, six open-source packages, one running product, and a short list of partners each year. This page explains how those parts fit together, how the research is actually run, and what it looks like to engage with the lab as a user, a contributor, a researcher, or a partner.
2research programs
6open-source packages
27Foundation Agent generations
~42kPyPI downloads across packages
Open-source tools
The hypotheses become libraries anyone can install. ThoughtFlow and CGEM are the two that carry the programs.
Running products
Similar.dev and the Foundation Agent put the primitives under production load, where weak ideas fail fast.
Problems we did not choose
Contracts bring problems from outside the lab's own taste. That is where the ideas are most likely to break, and to improve.
How the lab runs
Research that ships, so it can be wrong in public.
Most research labs publish papers and most engineering shops publish products. Think.dev publishes both halves of one loop. A position is stated in plain language on the philosophy page. It is embodied in a primitive with a test suite. The primitive is used in a system that real people depend on. Whatever breaks is traced back to the position that caused it, and the position is revised, in public. The loop is slow by startup standards and fast by academic ones, which is roughly the pace we want.
The lab is funded by a mix of partner contracts and its own products. That mix is deliberate. Partner work keeps the research grounded in what practitioners need; the products keep the lab realistic about what it costs to run these ideas at scale; and the open-source tools are how the research pays its debt to the field it draws from.
01
Write the position first
Every architectural decision has a written rationale before the code exists. The doctrine is append-only; rules are never deleted, only superseded with a reason.
02
Build it as plain data
Everything crossing a boundary is a dict, a list, or a string. Printable, diffable, serializable exactly as it appears in code.
03
Record, then replay
Runs against live models are recorded once. Tests replay them forever, deterministic and offline, asserting on decisions rather than prose.
04
Evaluate by simulation
Behavior is judged by long multi-turn batteries of LLM-driven personas, with every failure traced to the exact memory event that caused it.
05
Document more than you code
English usually outweighs Python in the repositories. The documentation is infrastructure for the next engineer, human or otherwise.
The two programs
One conviction, asked at two layers of the stack.
In both programs the question is the same: how do you house a powerful, partly opaque learned component inside a form that an accountable person can read, test, and control? Program A asks it about agents. Program B asks it about models.
Program AAI engineering
The optimal primitives of AI engineering
Question. What is the smallest set of parts from which any agent can be assembled, in ordinary code, with the state always visible and the control flow always owned by the engineer?
Method. ThoughtFlow is the running answer: fourteen primitives, a memory object that is the whole state, and no hidden runtime. Its design is tested two ways: by its own ~870-test suite, and by being the substrate under the Foundation Agent, where any awkwardness in a primitive shows up as a bug in behavior.
Open questions. How much of planning belongs in code versus in the model. Whether a closed activity registry (a finite, named set of things the agent may do) scales beyond a few dozen activities. What the right primitives are for memory revision: superseding, qualifying, forgetting.
Artifactsthoughtflow, thoughtbase, taskatlas, membase
Tests~870
Providers7 + MCP
Program BStatistical learning
Structure, control, and explainability in learned models
Question. How do you embed machine-learned effects inside a formula a domain expert can read, keep the whole causally coherent as it converges, and retain the ability to constrain or replace any single effect?
Method. CGEM writes the model as an explicit effects formula. Each term can be a constant, a categorical lift, a linear response, a smooth curve, or a boosted learner. Terms are refit collaboratively, iteration by iteration, until the decomposition stabilizes. The result is a model whose reasons are its parameters.
Open questions. Convergence guarantees for mixed term types. Principled ways to let an agent propose, test, and justify a formula. How to expose effect uncertainty so that a decision-maker knows which terms to trust.
Artifactcgem 0.2.1
Since2023
Downloads~17k
where the programs meet
# An agent that proposes a structure, and a model that # must earn it. Both halves stay legible. propose = THOUGHT(name="formula", llm=llm, prompt="Columns: {columns}. Propose a CGEM effects formula.") memory = propose(memory) model = CGEM() model.load_df(train) model.define_form(memory.get_var("formula_result")) model.define_terms(terms) model.fit(25) # The agent explains the model in the model's own terms. memory.set_var("effects", model.summary()) memory = explain(memory)
Why one lab does both.
Agents increasingly decide when and whether to act on a model's output. If the model cannot surface its reasons, the agent cannot weigh them, and the whole system becomes opaque from the bottom up. Conversely, a structured model is only as useful as the system that decides how to use it. Keeping both programs under one roof means the seam between them is a design object rather than an afterthought.
It also keeps the lab honest. It is easy to preach legibility about agents while shipping a gradient-boosted black box underneath them. We would rather be caught by our own standard.
The snippet shows the shape of the seam using real ThoughtFlow and CGEM calls; it is a pattern, not a packaged integration.
Foundation Agent · how the evaluation works
Twenty-seven generations of one loop, and a doctrine that only grows.
The Foundation Agent is the lab's longest-running experiment. It asks whether a single, immutable behavioral template can support many kinds of useful agency through configuration alone, and it answers with measurements rather than demos.
The template
Every generation runs the same five-phase loop: triage what deserves attention, plan a bounded intention, execute one finite activity from a closed registry (47 activities in 11 families), reflect on what changed, and rest when nothing is owed. Around that loop the prompts, knowledge, permitted activities, policies, and memory can vary. The loop itself cannot. That constraint is what makes generations comparable.
The battery
Each generation is evaluated by 12 to 20 LLM-driven personas, multilingual, each running long multi-turn conversations designed to probe a specific failure family: silent turns, broken commitments, unearned confidence, unnecessary interruptions, unsafe actions, and failure to rest. Every failure is diagnosed forensically, back to the exact event in memory that produced it, and classified against a behavioral failure taxonomy.
The doctrine
Every fix becomes a rule, attached to the failure that earned it, in an append-only design document. Rules are never deleted, only superseded with a reason. After 27 generations the doctrine is the most valuable artifact the program has produced, because it is the only one that explains why the agent is shaped the way it is.
triage
plan
execute
reflect
rest
v01each generation diffable and replayablev27
Internal result · our battery, our taxonomy
Latest generation versus the one before it.
Across a 214-turn battery, major behavioral failures fell from 16.0% to 11.7% of turns, p90 turn latency fell from 17.5 s to 10.4 s, and there were zero silent turns. These are the lab's own measurements against its own failure taxonomy. They are reported to show what the program measures, not to claim a benchmark position. A methods write-up is available to partners and collaborators on request.
Major failures16.0% -> 11.7%
p90 latency17.5 s -> 10.4 s
Silent turns0
Activities47 in 11 families
Ways to engage
Use the tools, argue with the ideas, or bring us a problem.
Everything the lab publishes is free to use. The conversations we most want are with people who are building something the average answer will not solve.
UseFree, MIT
Build with the primitives
pip install thoughtflow for agents, pip install cgem for structured models. Both have documentation in the repository, examples that run, and an author who reads the issue tracker. If you ship something with them, tell us; it is the feedback the research runs on.
CostNone
SupportGitHub issues
LicenseMIT
CollaborateResearch
Work on the open questions
Researchers, graduate students, and practitioners who want to attack one of the open questions above, contribute an evaluation battery, or replicate a Foundation Agent result are welcome. Co-authorship on write-ups is offered where the contribution warrants it.
CostNone
FormatAsync, written
StartOne email
PartnerSelective
Bring a problem without a playbook
A small number of partner projects each year. The best fits are startups entering new markets, products with a genuinely new value proposition, and deployments where the system has to be explainable to the people accountable for it. Every project starts with a short architecture sprint.
Starts with1 conversation
First stepArchitecture sprint
DeliverableWorking system
What a partnership actually looks like.
Week one is an architecture sprint. We learn the problem, write down the principles that constrain it, and produce a reference architecture with the rationale for every decision. If the fit is wrong, this is where both sides find out cheaply.
Then we build, in slices. Agent systems on ThoughtFlow, retrieval and embedding pipelines, interpretable models where a black box would not be defensible, lean serverless production paths. Each slice ships with tests that assert on behavior and documentation that explains why it is shaped the way it is.
Then we hand it over, fully. Your team owns the code, the doctrine, and the test suite. The goal of every engagement is that you do not need us afterward, which is also why partners come back.
Start here
Tell us what you are building.
One form, one structured note into James's inbox, one reply from James, usually within a few days.