Bench Book · Architecture

About this system

The Bench Book answers questions about the Michigan Court Rules by finding the governing provisions first and writing from them second. It never answers from general knowledge: every sentence is composed from retrieved rule text, every citation is checked against the parsed rules before the page renders, and every passage can be traced to the printed page of the source PDF.

The pipeline

1 · Parse874-page official PDF → 625 rules, 11,860 citable provisions. Verified three ways against the document itself.

2 · Chunk3,580 passages, each scoped to a single rule (never crossing one), ~256 tokens.

3 · RetrieveMeaning-based search (Qwen3-Embedding-4B). Exact citations bypass search entirely via a deterministic router.

4 · ExpandA cross-reference graph (943 classified edges) supplies provisions the retrieved rules are expressly subject to — or overridden by.

5 · Composeglm-5.2 writes from the retrieved passages only, within a 4,096-token reading window.

6 · AuditEvery citation in the answer is checked: does it exist, and was it actually retrieved? Only then does the page render.

Why you can check it

Citations link to their passages, and passages link to the printed page of the source PDF (both numbering systems shown).
The audit is visible. A citation the model repeated from inside a passage — rather than one it was shown — is labelled exactly that.
Refusal is a feature. When the rules do not answer a question, the system says so in amber and names where the answer lives instead. On a 32-question test of genuinely unanswerable questions it refused all 32.
Every search is recorded on this machine (question, answer, retrieved provisions) and nothing leaves it except the generation request.

Measured performance

property	result	how measured
parse completeness	384,507 = 384,507 words	word-for-word identity with the source PDF
structure	625/625 rules	reconciled against the document's own table of contents
right provision in the reading window	94%	1,060-question benchmark, gold = citation strings
exact-citation lookups	125/125	deterministic router
answer cites the correct provision	0.91	paired evaluation, independent judge model
fabricated citations	0	every emitted citation checked against the corpus
refuses genuinely unanswerable questions	32/32	negative test set

What was tested and rejected

Each of these was measured on paired benchmarks and made the system worse: keyword-search fusion (BM25), a reranking model, query rewriting (HyDE), heading prefixes in passages, merged-rule context assembly, and larger passage sizes. The system is deliberately simple because the simpler configuration measured better at every step — the full record is in the project repository (docs/TESTING-AND-LESSONS.md).

Limits

This is a research prototype, not legal advice. It covers the Michigan Court Rules as amended through July 31, 2026 — not statutes (MCL), case law, the Rules of Evidence, or local administrative orders; when an answer depends on those, the system says so rather than guessing. It is an administrative support tool developed by the Child & Adolescent Data Lab at the University of Michigan School of Social Work, not a product of the Michigan courts.

← Back to search