An assistant for cancho that proves its work

Why

Most coding assistants hand you a transcript and ask you to read the diff. cancho-code hands you a verdict:

[ISSUE_VERDICT]	verified	attempts 1

The model does not decide when it is done. The compiler does. A run of the loop is: the agent edits the file, cancho check --output json answers, the loop decides — retry, stop the issue, stop the run — and after a verified answer it checks again before believing it, then snapshots it. Every way the run can end prints one named verdict line; there is no state in which it runs silently.

The failure it is built not to repeat is lex-code's own, measured and documented there: runs that continue for a long time without solving anything. Loop-safety here is contracts with defaults, not tuning flags — and each one has a gate where a stand-in loops forever and the assertion is that the loop stops, in seconds rather than hours.

Run it

Build the fix tool from a checkout of this repository, with the compiler's packages beside it:

git clone https://github.com/alpibrusl/cancho-code.git
cd cancho-code
export CANCHO=<path-to-the-cancho-binary>      # the commit named in .cancho-rev
export CANCHO_PACKAGES=<compiler-checkout>/packages

"$CANCHO" build tests/programs/fix.cho \
    src/driver.cho src/checker.cho src/signature.cho \
    src/loop_check.cho src/snapshot.cho src/trail.cho src/acceptance.cho \
    --std --backend cranelift -o ./fix

Then point it at a broken program and an agent:

./fix "$CANCHO" /abs/path/project.cho "$(pwd)/scripts/cancho-agent.sh --provider ollama --model qwen3.8:27b" 4

The three arguments are the compiler (by its absolute path — the child of exec_spawn gets no PATH of its own), the file to fix, and the agent: any shell command that edits the file. The fourth is the attempt ceiling; an optional fifth is the seconds one agent turn may take (default 30), and a sixth is an acceptance command that must also hold before a file counts as verified. Exit codes: 0 verified, 1 the budgets named a stop, 2 usage, 3 the checker or the agent is away — a wait, rerun. Beside the file the run leaves <file>.trail.jsonl (one record per attempt), <file>.feedback (what the agent is shown) and, once verified, <file>.snap0.cho.

A realistic walkthrough: Ollama, qwen3.8:27b, a broken program

This is the whole point of the tool, end to end, on one machine: a local model with no key and no egress, a real program with a real error, and a compiler that decides when the model is done.

1. Install Ollama and pull the model

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen3.8:27b

27B wants a machine with roughly 20 GB of memory to spare; on a laptop-class machine a whole run takes minutes, not seconds. That is fine — the budgets are in attempts and turns, not milliseconds.

2. Make a program with a real error

mkdir demo && cat > demo/wordcount.cho <<'EOF'
edition 7;

// A broken word counter. Two errors, deliberately the same rule with
// different messages, so a model that fixes one and not the other
// changes the failure signature and gets its full budget back.
fn counts[&i](io: &!i Io) -> [io_read] int {
    var words = 0;
    var inside = false;
    var c = getchar(io);
    while c >= 0 {
        if c == 32 || c == 10 {
            inside = false;
        } else {
            if inside == false {
                words = words + 1;
            }
            inside = true;
        }
        c = getchar(io);
    }
    return inside;
}

fn label(total: int) -> [] int {
    if total > 0 {
        return "many";
    }
    return 0;
}

fn main(world: World) -> [] int {
    let Split { io, ffi, fs, heap, args, net, clock, signals, exec } = split(world);
    release(ffi); release(fs); release(net); release(clock);
    release(signals); release(exec); release(args);
    release(io);
    release(heap);
    return 0;
}
EOF

Two errors, both type-mismatch with different messages: counts returns a bool where its signature says int, and label returns a string where it says int. The checker answers both in one pass:

"$CANCHO" check demo/wordcount.cho --output json --backend cranelift

3. Run the loop with the local model as the agent

./fix "$CANCHO" "$(pwd)/demo/wordcount.cho" \
    "$(pwd)/scripts/cancho-agent.sh --provider ollama --model qwen3.8:27b" 4 300

The agent has no environment. Nothing you export reaches it: the driver passes it exactly one variable, PATH. The model name goes inside the agent command, as above (on Apple silicon the installed tag may be qwen3.8:27b-mlx; ollama list shows it). Paths must be absolute and may not contain ...

What happens, turn by turn:

• The driver runs scripts/cancho-agent.sh --provider ollama --model qwen3.8:27b demo/wordcount.cho: the agent (a cancho program) sends the file, the instruction ("reply with the complete corrected file") and the compiler's current refusals (the driver left them in <file>.feedback) to Ollama's HTTP API, keeps the program out of the reply (it cuts markdown fences and the model's reasoning, and retries a reply that has no program in it or was cut off), and overwrites the file with the answer.
• The driver runs the real cancho check --output json on the result. The model's word is worth nothing here; the checker's JSON is the only oracle.
• signature.of turns the refusal into a failure signature — the rule verbatim, the message with its backticked spans kept and numbers normalised — and loop_check.decide decides with the §4 budgets: attempt ceiling 4, same-signature stop at K=2.
• If the model fixes counts and not label: the signature changes ("expected int, found bool" becomes "expected int, found a string"), the counter resets, the loop tries again — a struggling model gets its full budget.
• If the model returns the same wrong answer twice: the same-signature stop fires — [ISSUE_VERDICT] failed signature — and the run ends, named, instead of burning the night.
• When the check says clean: the driver checks again (the re-verify pass) and only then snapshots the file and prints [ISSUE_VERDICT] verified attempts N.

4. What a good run looks like

$ ./fix "$CANCHO" demo/wordcount.cho "$(pwd)/scripts/cancho-agent.sh --provider ollama --model qwen3.8:27b" 4
[ISSUE_VERDICT]	verified	attempts 2
$ echo $?
0
$ "$CANCHO" check demo/wordcount.cho --output json --backend cranelift
{
  "refused": []
}

Done is a proof, not a status.

5. What a bad run looks like (and why it is cheap)

A model that keeps answering with the same broken shape stops on the second identical refusal:

[ISSUE_VERDICT]	failed	signature
$ echo $?
1

Beside the file, demo/wordcount.cho.trail.jsonl has one line per attempt (how long the agent took, what the checker said, the signature the stop rule compared, what the loop decided), and a verified state is kept as demo/wordcount.cho.snap0.cho: a stopped run explains itself.

Cloud models

The same loop drives a cloud provider over native HTTPS — scripts/test-provider.sh mistral with MISTRAL_API_KEY in the environment runs the end-to-end gate against the real API, the checker still the oracle. The provider is written in cancho on packages/http-client and packages/tls; the key rides the Authorization header and exists nowhere else.

Run the cases: build a library from a plan, then audit it

The walkthrough above repairs one file. This section builds whole libraries from the lex-code backlog with a local model, and grades them against a hidden oracle. Three cases are wired up: 07-tokenbucket, 05-duration and 03-csv. Each is a plan (what to build, unit by unit), a reference (an implementation known to pass), and an audit (the oracle's cases, which the model never sees).

1. Build the two tools

export CANCHO=<path-to-the-cancho-binary>      # the commit named in .cancho-rev

"$CANCHO" build tests/programs/project.cho src/project.cho src/plan.cho \
    src/driver.cho src/checker.cho src/signature.cho src/loop_check.cho \
    src/snapshot.cho src/trail.cho src/acceptance.cho src/splice.cho src/pool.cho src/parallel.cho \
    --std --backend cranelift -o ./project

"$CANCHO" build tests/programs/plancheck.cho src/plan.cho \
    --std --backend cranelift -o ./plancheck

2. Prepare the input

A run needs four things, none of which the model sees except the first three.

A plan is JSON. This is the shape (a real one, shortened; docs/plan.md has the twelve rules the checker enforces):

{"name": "tokenbucket",
 "units": [
  {"id": "refill",
   "signature": "fn bucket_refill(tokens_milli: int, last_ms: int, now_ms: int, capacity: int, per_sec: int) -> [] int",
   "description": "Amounts are milli-tokens... never more than capacity * 1000.",
   "examples": ["bucket_refill(0, 0, 1000, 10, 1) == 1000"],
   "deps": []},
  {"id": "take",
   "signature": "fn bucket_take(tokens_milli: int, cost: int) -> [] option.Option[int]",
   "examples": ["bucket_take(5000, 5) == option.Option::Some(0)",
                "bucket_take(4999, 5) == option.Option::None"],
   "deps": []}]}

• examples are CALL == EXPECTED and become tests; == compares scalars, and an Option result is run as a match.
• For a function an example cannot call (a result that is text, or a Result), a unit carries its own tests: cancho source appended verbatim to the unit's tests file.
• description says what the signature and the examples cannot: units of measure, edge rules. Without it the first tokenbucket runs failed on every unit.

./plancheck eval/duration/plan.json
[PLAN]	valid

Every problem is reported, not the first, as [PLAN_ERROR]<TAB>rule<TAB>unit<TAB>message, with exit code 1.

3. Run a case

The work directory must exist, and every path must be absolute and free of ... The scaffold is the only file you write in it:

# 07-tokenbucket (the brief names a record type, so the scaffold declares it)
WORK=/abs/path/work-tokenbucket; mkdir -p "$WORK"
printf 'edition 7;\nimport std.option;\n\nstruct Allow { allowed: bool, tokens_milli: int, retry_ms: int }\n' > "$WORK/project.cho"
PLAN="$(pwd)/eval/tokenbucket/plan.json"

# 05-duration
WORK=/abs/path/work-duration; mkdir -p "$WORK"
printf 'edition 7;\nimport std.result;\nimport std.buffer;\n' > "$WORK/project.cho"
PLAN="$(pwd)/eval/duration/plan.json"

# 03-csv
WORK=/abs/path/work-csv; mkdir -p "$WORK"
printf 'edition 7;\nimport std.list;\nimport std.result;\n' > "$WORK/project.cho"
PLAN="$(pwd)/eval/csv/plan.json"

Then the same command for each, with the model and the primer written inside the agent command (the agent has no environment). The numbers are the attempts per unit and the seconds one agent turn may take:

./project "$CANCHO" "$PLAN" "$WORK" \
  "$(pwd)/scripts/cancho-agent.sh --provider ollama --model qwen3.8:27b --num-ctx 24576 --primer $(pwd)/guidelines.md" \
  6 900

It prints one line per step and ends with a verdict (exit 0 done, 1 stuck, 3 no_plan, 4 provider_error):

[PLAN]	valid
[UNIT]	refill	building
[ACCEPTANCE]	pass
[UNIT]	refill	verified
...
[PROJECT_VERDICT]	done

Rerun the same command after a stop: the verified units are kept (state.txt), re-checked, and only what is left is built. trail-<unit>.jsonl in the work directory shows every attempt.

4. Grade it against the hidden oracle

done means the plan's own examples held. The audit is the oracle's full set, written before any run:

python3 eval/audit_tokenbucket.py "$WORK/project.cho"      # 46 cases
python3 eval/audit_duration.py   "$WORK/project.cho"      # 64 cases
python3 eval/audit_csv.py        "$WORK/project.cho"      # 50 cases

Each prints a line per function group and exits 0 only if every case holds. A function of the wrong shape costs its own group only.

5. Check the setup before spending GPU time

The reference implementation of each case must pass its plan's tests through the whole driver and score full marks on the audit, and a deliberately broken copy must not. This needs no model:

CANCHO="$CANCHO" sh scripts/test-eval.sh        # prints: eval gate: ok
CANCHO="$CANCHO" sh scripts/test-project.sh     # the driver against a stand-in agent

What these runs have shown so far

Three runs each, one model (qwen3.8:27b), the compiler's own page as the primer; the detail and the caveats are in issue #9.

Read these as measurements of a setup, not of a model. Samples are small, the plans were written by hand, and the lever that moved results every time was what the model was told about cancho's standard library.

The loop-safety contracts

Each one has a gate: scripts/test-checker.sh, test-snapshot.sh, test-driver.sh (three cases: a fix verifies in one attempt; a looping agent stops at the signature rule; a cycling agent gets its full ceiling). Thirteen unit tests cover the decision functions and the signature normalisation.

Status and limits, stated plainly

• Built: the decision module, the failure signature, the subprocess checker driver, the snapshot of a verified state, the composed loop with its verdict lines, the per-attempt trail, the acceptance command, the Mistral provider over native HTTPS, the Ollama agent script (HTTP API, with the compiler's refusals in the prompt), the fix front door, the plan check (plancheck) and the project driver (project: dependency order, a sweep after every unit, resume). The authority reports are pinned in CI. All gates are green in CI without a network or a key.
• Not built: a model that drafts the plan, hardening (invariants on generated inputs), acceptance scenarios for services, integration repair, an unattended supervisor with a report, lessons mined from trails, a dashboard. docs/parity.md maps each to lex-code.
• The local-model path shells out to ollama through the Exec capability: the tool's authority report names the one directory its agents may live in.
• Measured on this repo's own gates, four silent false-passes were found and fixed by the gates themselves, not by review: a reversed-signature normalisation, a shell $0 argv bug, an edition-less fixture, a negative ring position.