Research / Benchmark

Nordform Trueline

A benchmark for whether a design factory produces what it claims, and stops where it should.

1. Why this benchmark

General language-model benchmarks reward a confident answer. In design work a confident wrong dimension costs more than a question back: it travels into drawings, orders and site work. The public text-to-CAD benchmarks we know of grade shape similarity. They do not grade whether an answer stops when the brief cannot be met, or whether another vendor's tool opens the files. Trueline grades those directly, and leaves model judges out: every measure has a typed answer to compare against.

2. What is measured

The headline measure is verified designs: of the cases whose answer is a design, the share where the factory ran, an independent check accepted the result, every delivery was ready, and the typed proposal is the one the case expects. The other measures are reported beside it and never folded into one score.

Typed proposal

The answer's workflow, outcome and every operation match the case, with no extra operations.

Honesty

Correct stops on cases that must be refused or asked about; false claims, where a stop was required and a design was delivered; over-refusals, where work was possible and was refused.

Constraints

Protected dimensions or openings changed; numbers off by a power of ten.

Files open

Delivered IFC and DXF files that IfcOpenShell and ezdxf open and find a model in. Our own parser never counts.

Reruns identical

The same request, run again from the start, gives identical output. When a run replays recorded model answers, the model stage is fixed and only the factory is rerun.

Reliability

Runs, failures by code and timeouts.

Cost and time

Run time at the median and 95th percentile, model calls, tokens, and euros per verified design.

3. Corpus

The held-out cases were written by the people who build the factory. In plumb-v1, held out means unpublished; a corpus written by people who do not tune the factory is the control we intend for a later version.

4. Systems compared

1. Nordform factory. A model reads the brief and answers with a typed proposal: named operations on named parameters, never geometry or code. Deterministic code builds the design from it, the registered checks run, and the IFC and DXF files are written.
2. Model alone. The same model, the same case and the same plan schema, with no factory prompt, catalogue, kernel or checks, graded by the same grader. It cannot make a design, so the comparison is on the typed proposal, honesty and constraints.

5. Results

Run of 2026-10-21 · nightly c2dd1170 · corpus plumb-v1 · model claude-opus-5-5 · results JSON

Run notes

What this run measured nothing on

• Cancelling a running job is not exercised: the harness runs the factory step by step, not as a whole job.
• Timings come from a benchmark runner, not from production. Runs on an emulated runner are slower than production.
• Room-study proposals are graded on the typed proposal and honesty only, and are left out of the verified-artifact count.
• Steel cases are not in plumb-v1.
• Laser cutting is not measured.

Run history

6. Scope

• A verified design matches its brief, was accepted by the factory's own verification and was delivered. Checks inside it can report an input as required or a question as not assessed. Whether its files open in independent readers is reported separately. Structural, regulatory and manufacturing approval stay with the qualified engineer who signs.
• plumb-v1 covers one design kernel, the timber kit-house frame. It says nothing about other kernels or families.
• The corpus is small, so intervals are wide. A difference inside overlapping intervals is not claimed.
• We wrote both the benchmark and the factory. The held-out split, the published graders and outside reruns are the controls on that.
• Time saved on real projects is reported only from real cases, with the client's permission, and never from this corpus.

7. When we rerun

Our rule is to rerun Trueline after every substantial change to what produces a design: the geometry kernels, the checks, the instructions and context the model reads, or the model itself. Each run that covers the held-out split is added to the run history above with its date, whether the result rose or fell.

8. Reproduce

npm install
ANTHROPIC_API_KEY=… node runners/model-alone.mjs --model claude-opus-5-5
node graders/grade.mjs --answers answers-model-alone-claude-opus-5-5.jsonl
PLUMB_FACTORY_COMMAND="./my-factory --one-case" node adapters/factory.mjs

Run the commands from the downloaded folder. The cases, graders, the model-alone runner, the factory adapter's contract and every result are published in the Trueline repository and with the method. Another factory that answers the adapter's contract can be graded on the dev cases; we run the held-out cases on request.

9. Cite

@misc{nordform2026trueline,
  title  = {Nordform Trueline: Does a Design Factory Produce What It Claims?},
  author = {{Nordform}},
  year   = {2026},
  note   = {Corpus plumb-v1},
  url    = {https://nordform.ai/research/trueline/}
}

Cases and results: CC BY 4.0. Graders, runners and schema: Apache-2.0.