The corpus

What the checks are measured against, beyond the two examples each one ships.

python3 corpus/build.py                # the three source files into the query files in local/corpus
python3 corpus/harvest.py [path/to/.env]   # score them, 250 calls
python3 corpus/measure.py              # recall on the gold negatives, firing rate on the rest
python3 corpus/drift.py                # this harvest against the one it replaced

python3 corpus/squad.py --items=40    # one call per item, balanced across the two labels

python3 corpus/pages.py               # mirror the pages into local/docs
python3 corpus/states.py --write      # recover each docs example's state from its page
python3 corpus/decision.py --items=40 # the policy ablation, three calls per case
python3 corpus/advice.py --items=30   # does the advice improve the answer, two calls per arm
python3 corpus/rewrite.py             # the same for a wording check, five arms
python3 corpus/wording.py --items=20  # three wording checks, seven arms
python3 corpus/families.py --items=6  # do the error-severity checks fire where answers fail
python3 corpus/levels.py              # does a Score check's advice improve where reviews land
python3 corpus/levels.py --check=score/multi-dimension   # with the arms in local/score-arms.json

decision.py fetches decision-v7 into local/ on first run, and advice.py, boundary.py, families.py, planted.py and unread.py read it from there, so run decision.py before them. The arm experiments - contradiction.py, rewrite.py, levels.py, wording.py - read their sample and their rewrites from corpus/arms/, which is committed because those files are the experiment's inputs rather than its output: the rewrites were written blind by a panel and cannot be regenerated by running anything. A copy in local/ takes precedence, which is how a different rewrite is tried without touching the committed one.

The checks with no gold negative are measured by planting the defect in material somebody else wrote, or by fetching a dataset that is the defect:

python3 corpus/planted.py --items=20  # a line addressed to the reviewer, put into a document
python3 corpus/unread.py --items=20   # a field added to a state that no question reads
python3 corpus/external.py --check=query/overlapping-questions --items=30   # Quora's pairs
python3 corpus/contradiction.py       # criteria inverted against the instruction they belong to
python3 corpus/boundary.py            # a boundary taken out of the policy that defines it
python3 corpus/encoded.py             # the same comparison over a value written plainly and as hex
python3 corpus/dogfood.py             # the catalogue's own wordings, as queries jevlint can read

decision.py is the strongest evidence here about a state-scoped check. decision-v7 ships states holding a policy and a case, the question asked about them, and the answer it should get, so removing the policy makes the state unable to answer the question and changes nothing else. It also records what Jev answers as written, scored against the label, which is the baseline advice.py, rewrite.py, wording.py and levels.py measure their repaired arms against.

squad.py is the only other measurement here whose labels come from outside this repository for a state-scoped check. It fetches SQuAD 2.0 into local/ on first run, samples a seeded balanced set so two runs measure the same material, sends each paragraph as the state and its question as the question, and records what state/answer-absent read. Evidence carries the result.

harvest.py uses TYPESAFE_API_KEY from the environment. Pass an env file as its argument, or set JEVLINT_ENV_FILE, if the key lives somewhere else; it refuses to start instead of running every check against a file that is not there.

The three source files hold one question per entry with its label attached, which is the shape a label is easiest to read and write in. build.py turns them into the query files jevlint takes, grouped as they were asked, with the labels stripped and the ids normalised so that measure.py can split a score id back into its file and its question. It refuses a source entry whose primitive and criteria do not agree, naming the entry.

harvest.py runs jevlint check --all, which reports the model checks that ran and cleared alongside the ones that fired. That records a probability whether or not it would have been reported, without touching the catalogue other processes are reading.

Query files that carry a state are asked with it. states.py recovers the state each docs example was run against from its own page, so 32 of the 76 entries carry one and the state-scoped checks are measured on material somebody else wrote. The other 44 belong to cookbook pages that fetch or loop over their state at run time, so the page holds none to recover, and gold.json is questions quoted from prose with no material anywhere upstream. Those files carry no state and the run says which checks that left out.

What the gold set is for

A check's own two examples show it can tell an obvious defect from an obvious non-defect. They cannot show it fires on the right things, because the person who wrote the check wrote them both. The gold set is labelled by somebody else: each entry quotes the documentation saying this question is right, or is wrong and why.

Fourteen are wrong with the defect named, so a check that claims to find that defect either finds it or does not. Eight are right.

A positive carries clears: the checks its citation speaks to. The docs call "Does this message convey urgency?" a good question in a passage about snap judgements, which clears question/compound-judgment and says nothing about where the question's boundary falls. A blanket good label would have scored every check against a claim the citation does not make.

Six entries are tier: derived instead of quoted: the documentation defines the failure but gives no example, so the example is written from the definition. They are weaker evidence than the quoted ones and are marked so.

Reading the numbers

Some docs examples carry no criteria. They are extracted from code where the options and levels are built from variables, so 34 of the 76 kept theirs and the rest are instruction only. A check that reads criteria is asked about criteria that are not there on those, which is one source of the firing rates on this tier.

The docs repeat examples across pages. 76 entries hold 52 distinct questions, so measure.py counts each question once. Reading the raw entry count inflates every rate.

The clean-material column is the docs tier plus the gold positives. measure.py counts a gold positive for a check only where that entry's clears names it, and no check is named by more than two of the eight, so the column adds at most two to a denominator: 52 distinct docs questions plus two gold positives is the 54 the widest checks are out of. The two tiers are labelled clean by the same standard, and the column is a rate over everything labelled clean.

The field tier has no labels. A firing rate on it is a rate, not an error rate. Where a finding there has been checked by hand, Evidence says so.

A denominator is the questions the check was asked about. A check that applies only to Choice questions is asked about the Choice questions and no others, so its rate is out of those, not out of the tier. Counting a question the check never saw as a question it cleared would turn silence into a clean bill of health, and measure.py prints the checks that were never asked at all instead of scoring them 0.

Twenty-two gold entries is a small set, fourteen of them negatives. It is enough to show whether a check finds the example the documentation uses to define its own failure mode. It is not enough to put an interval on a recall figure.