Settle the coding by a majority of three runs instead of an arbitration
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -12,23 +12,23 @@
|
||||
step of the research procedure, not the file: the
|
||||
deterministic vocabulary step (step 2) has no definition
|
||||
file yet holds its own number. Zero padding is for
|
||||
sorting only -- prose says "step 1", "step 3-2".
|
||||
sorting only -- prose says "step 1", "step 3".
|
||||
- Per-song LLM judgments (coding) run the same definition
|
||||
file twice, then a separate arbitration step settles only
|
||||
the script-computed disagreements
|
||||
("2 runs + 1 arbitration"). Free-generation steps run
|
||||
twice and both outputs are pooled, unarbitrated. The
|
||||
vocabulary is built by a deterministic subcommand
|
||||
(embedding + clustering), not by an LLM. If an arbitration
|
||||
or validation outcome is unexpected, revise the definition
|
||||
file and repeat that cycle; never patch results by hand.
|
||||
file three times, independently, over the same input; a
|
||||
deterministic subcommand then assigns a (song, keyword)
|
||||
pair when at least two of the three runs assign it
|
||||
("3 runs + majority vote"). Free-generation steps run
|
||||
twice and both outputs are pooled. The vocabulary is
|
||||
built by a deterministic subcommand (embedding +
|
||||
clustering), not by an LLM. If a validation outcome is
|
||||
unexpected, revise the definition file and repeat that
|
||||
cycle; never patch results by hand.
|
||||
- Each run of a step is archived self-contained under the
|
||||
destination directory given explicitly on the `run-llm`
|
||||
command line (by convention `runs/<step>/run<N>/`):
|
||||
prompt snapshot, raw output, and `meta.json` (model ID,
|
||||
parameters, timestamps, batch ID). The two runs of a step
|
||||
are two separate invocations of `run-llm`. An arbitration
|
||||
pass is a step of its own with its own archive. Replacing
|
||||
parameters, timestamps, batch ID). The runs of a step are
|
||||
that many separate invocations of `run-llm`. Replacing
|
||||
an existing run archive requires an explicit flag;
|
||||
superseded runs live in git history. Deterministic steps
|
||||
archive under `runs/<step>/` with no `run<N>` level.
|
||||
@@ -51,7 +51,7 @@
|
||||
|
||||
## Documents
|
||||
|
||||
- `results/` holds final arbitrated tables (what the paper
|
||||
- `results/` holds the final tallied tables (what the paper
|
||||
cites); `runs/` holds raw audit records. The paper cites
|
||||
`results/` only.
|
||||
- Any change to a definition file, the codebook, or the plan is
|
||||
|
||||
Reference in New Issue
Block a user