Settle the coding by a majority of three runs instead of an arbitration

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-17 22:38:34 +08:00
co-authored by Claude Opus 5
parent 7008e14972
commit c51c9d71e1
24 changed files with 101 additions and 1575 deletions
+13 -13
View File
@@ -12,23 +12,23 @@
step of the research procedure, not the file: the
deterministic vocabulary step (step 2) has no definition
file yet holds its own number. Zero padding is for
sorting only -- prose says "step 1", "step 3-2".
sorting only -- prose says "step 1", "step 3".
- Per-song LLM judgments (coding) run the same definition
file twice, then a separate arbitration step settles only
the script-computed disagreements
("2 runs + 1 arbitration"). Free-generation steps run
twice and both outputs are pooled, unarbitrated. The
vocabulary is built by a deterministic subcommand
(embedding + clustering), not by an LLM. If an arbitration
or validation outcome is unexpected, revise the definition
file and repeat that cycle; never patch results by hand.
file three times, independently, over the same input; a
deterministic subcommand then assigns a (song, keyword)
pair when at least two of the three runs assign it
("3 runs + majority vote"). Free-generation steps run
twice and both outputs are pooled. The vocabulary is
built by a deterministic subcommand (embedding +
clustering), not by an LLM. If a validation outcome is
unexpected, revise the definition file and repeat that
cycle; never patch results by hand.
- Each run of a step is archived self-contained under the
destination directory given explicitly on the `run-llm`
command line (by convention `runs/<step>/run<N>/`):
prompt snapshot, raw output, and `meta.json` (model ID,
parameters, timestamps, batch ID). The two runs of a step
are two separate invocations of `run-llm`. An arbitration
pass is a step of its own with its own archive. Replacing
parameters, timestamps, batch ID). The runs of a step are
that many separate invocations of `run-llm`. Replacing
an existing run archive requires an explicit flag;
superseded runs live in git history. Deterministic steps
archive under `runs/<step>/` with no `run<N>` level.
@@ -51,7 +51,7 @@
## Documents
- `results/` holds final arbitrated tables (what the paper
- `results/` holds the final tallied tables (what the paper
cites); `runs/` holds raw audit records. The paper cites
`results/` only.
- Any change to a definition file, the codebook, or the plan is