Require valid JSON with escaped quotes in the definition files

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/decision-log.md

@@ -99,8 +99,8 @@
 
 ## 2026-08-02
 
-- **`import-lyrics` 不設為子命令，pilot 歌詞改以私人腳本
-  匯入**。理由：pilot 捕捉檔不隨論文發布，子命令
+- **`import-lyrics` 不設為子命令，pilot 歌詞改以 `excludes/`
+  私人腳本匯入**。理由：pilot 捕捉檔不隨論文發布，子命令
   形式會在發布的 CLI 裡留下讀者無法執行的死命令——要交待的
   是「沿用 pilot 捕捉」的事實（記於 lyrics-provenance.csv 與
   論文方法節），不是工具本身；工具移出專案，發布管線即
@@ -178,8 +178,8 @@
   來源公開可稽核，快照乾淨、overrides 縮小；論文方法節揭露
   「捕捉前經研究者查證補完」。性別以公開自我認同為準，
   推測不確定且無佐證者列疑慮清單；查證屬資料策展而非
-  分析，以 Claude Code 輔助、不走分析 API；QS 批次以
-  私人工作檔留存，不隨論文發布。
+  分析，以 Claude Code 輔助、不走分析 API；QS 批次存
+  `excludes/`（私人工作檔，不隨論文發布）。
 
 ## 2026-08-04
 
@@ -227,7 +227,7 @@
 - **Markdown 檔名一律以 dash 連接**（decision-log.md、
   research-plan.md、project-structure.md 等；與 data/ 層
   CSV 檔名慣例一致），定義檔命名慣例同步改為
-  `prompts/<task>-v<N>.md`（如 screen-v1.md），版本庫外的
+  `prompts/<task>-v<N>.md`（如 screen-v1.md），`excludes/`
   私人工作檔一併改名；`conference_abstract.md` 改名並搬入
   `paper/`——它是本次年會實際送出的摘要，與全文同屬投稿
   血脈，不是 `docs/` 的內部工作文件。全 repo 指涉同步更新。
@@ -374,3 +374,25 @@
   層；(3) 複雜仲裁機械無以自證其益（見前條）。驗證改為
   確定性格式檢查（完整分割、組名唯一、cap ≤ 50），違規
   依協定修訂定義檔重跑。
+
+## 2026-08-05
+
+- **歌詞捕捉加浮水印／誤碼正規化**：全語料普查（程式窮舉
+  非 ASCII）發現歌詞站浮水印與 cp1252 誤碼——同形字（西里爾
+  е 160 處／78 檔、希臘 ό 6 處）、異體空格（U+2005 211 處、
+  U+205F 80 處、U+200A 6 處）、零寬字元（U+200B 3 處）、C1
+  誤碼（U+0091/92/93/94/97 共 58 處；先前手修之 U+0085 同
+  類）；受影響 81/883 檔全部來自 pilot 匯入，新抓取檔乾淨。
+  正規化規則四條（C1 依 cp1252 本義還原、同形字還原、異體
+  空格→ASCII 空格、零寬刪除）實作於 fetch-lyrics 的
+  normalize_lyrics()，寫入時套用；既有語料以同一函式一次性
+  套用。倒放行等歌曲本體內容不動——策展界線：清傳輸雜質，
+  不改內容。
+- **Song 775 拒答機制查明與裁定為無法分析**：tag 首跑僅
+  此一首遭模型拒答。以隔離子代理做消融測試（四變體）
+  證實觸發源是歌詞中的倒放拼寫行（觸發混淆文字防護），
+  與歌詞中的種族蔑稱無關、與零寬字元亦無關（移除
+  U+200B 而保留倒放行仍拒答）。裁定：倒放行是歌曲的
+  忠實內容，保留不動——語料忠實優先於可分析性；該首
+  視為無法分析，缺漏歸因於 Anthropic 平台防護，非本
+  研究語料的問題。

docs/project-structure.md

@@ -73,7 +73,7 @@ pop-fem-audit/
 │   └── methodology.md         # 方法細節（全文方法節底稿；
 │                              #   映射分析方法須在看結果前寫定）
 └── paper/
-    ├── abstract.md            # 摘要
+    ├── conference-abstract.md # pilot 摘要（投稿版）
     └── full-paper.md          # 全文
 ```
 

docs/research-plan.md

@@ -7,7 +7,7 @@
 
 - **研究敘事**：全文為獨立完整的正式研究（2016–2025 十年語料
   共 1000 筆榜單紀錄，凍結程序後全量執行），**不與先導研究
-  做比較**；全文數字一律以正式研究
+  （paper/conference-abstract.md）做比較**；全文數字一律以正式研究
   結果為準。先導研究僅作為 codebook v0 與假說的內部來源，
   記於決策日誌，不進入論文敘事。
 - **執行原則**：主會話只做討論；所有分析由 deterministic script
@@ -72,8 +72,8 @@
   候選歌曲的歌手另做深度背景（族裔以公開自我認同為準、音樂
   場景），script 輔助、人工核定。
 - **Pilot 歌詞沿用（私人匯入，不進發布管線）**：先導研究
-  捕捉檔（lyrics.json，684 首，2018–2025）以私人腳本
-  匯入歌詞快取——只取識別欄位與歌詞本文，以
+  捕捉檔（lyrics.json，684 首，2018–2025）以 `excludes/` 的
+  私人腳本匯入歌詞快取——只取識別欄位與歌詞本文，以
   (year, rank) 精確匹配 song_id。匯入工具不屬於專案交付物
   （讀者拿不到其輸入），讀者的重現路徑純粹是 `fetch-lyrics`；
   沿用之**事實**記於 `data/captures/lyrics-provenance.csv`（進 git）：
@@ -165,10 +165,14 @@
   與「貶低其他女性／以男性指標定義權力」。
 - 流程：v0 → 校準樣本試編 → 修訂 → 凍結 v1 → 全量編碼；
   中途修訂須記決策日誌並回檢已編歌曲。
+- Positionality statement 寫入論文（53 歲、長年婦運者、資深
+  工程師、離開學術圈多年、因整理榜單而起）。
 
 ## 其他已定案事項
 
 - 歌詞受版權保護：完整歌詞不進 git（`data/captures/lyrics/` gitignored），
   論文與 repo 只留分類所引摘錄。
+- 工程支援（寫 script、style check、審稿）用公司訂閱的
+  Claude Code；進論文的分析 token 由個人 API 帳號支付。
 - Fable 5 不用於 pipeline：成本高、thinking 無法關閉且不可稽核、
   無 temperature 控制，且會混淆「盲點是提示問題」的核心主張。

prompts/01-01-01-tag.md

@@ -14,6 +14,10 @@ Rules:
 - For each keyword, quote 1 to 3 verbatim lines of the lyrics
   that ground it; do not give a keyword you cannot ground.
 
+Do not wrap the output in a Markdown code fence.
+The output must be valid JSON; escape any double quote
+inside a string as \".
+
 Output a single JSON object mapping each keyword to its list
 of quotes, and nothing else:
 

prompts/01-02-01-merge.md

@@ -17,6 +17,10 @@ Rules:
   by hyphens, that best names the theme it gathers.  Group
   names must be unique.
 
+Do not wrap the output in a Markdown code fence.
+The output must be valid JSON; escape any double quote
+inside a string as \".
+
 Output a single JSON object mapping each group name to the
 list of its member keywords, and nothing else:
 

prompts/01-03-01-cap.md

@@ -20,6 +20,10 @@ Rules:
   by hyphens, that best names the theme it gathers.  Group
   names must be unique.
 
+Do not wrap the output in a Markdown code fence.
+The output must be valid JSON; escape any double quote
+inside a string as \".
+
 Output a single JSON object mapping each group name to the
 list of its member keywords, and nothing else:
 

prompts/01-04-01-code.md

@@ -21,6 +21,10 @@ Rules:
   the lyrics that ground it; do not assign a keyword you
   cannot ground.
 
+Do not wrap the output in a Markdown code fence.
+The output must be valid JSON; escape any double quote
+inside a string as \".
+
 Output a single JSON object mapping each assigned keyword to
 its list of quotes (an empty object when no keyword applies),
 and nothing else:

prompts/01-04-02-code-arb.md

@@ -30,6 +30,10 @@ Rules:
   lines of the lyrics; quote them.
 - Do not add any keyword that is not listed.
 
+Do not wrap the output in a Markdown code fence.
+The output must be valid JSON; escape any double quote
+inside a string as \".
+
 Output a single JSON object mapping each kept keyword to your
 list of quotes, and nothing else.  A dropped keyword is simply
 left out; output an empty object when nothing is kept:

prompts/02-01-01-screen.md

@@ -12,6 +12,10 @@ Rules:
   lines of the lyrics that ground the judgment; if you cannot
   ground it in a quote, the answer is no.
 
+Do not wrap the output in a Markdown code fence.
+The output must be valid JSON; escape any double quote
+inside a string as \".
+
 Output a single JSON array and nothing else: the quoted
 lines when the song expresses the theme, or an empty array
 when it does not:

prompts/02-01-02-screen-arb.md

@@ -21,6 +21,10 @@ Rules:
   lines of the lyrics that ground the judgment; if you cannot
   ground it in a quote, the answer is no.
 
+Do not wrap the output in a Markdown code fence.
+The output must be valid JSON; escape any double quote
+inside a string as \".
+
 Output a single JSON array and nothing else: the quoted
 lines when the song expresses the theme, or an empty array
 when it does not:

tools/src/pop_fem_audit_tools/commands/fetch_lyrics.py

@@ -57,6 +57,49 @@ SLEEP_SECONDS: float = 1.0
 """The delay between consecutive HTTP requests, in seconds."""
 
 
+def _build_normalization() -> dict[int, str | None]:
+    """Build the lyrics normalization translation table.
+
+    :return: The codepoint-to-replacement mapping, a replacement
+        of None meaning removal.
+    """
+    table: dict[int, str | None] = {}
+    codepoint: int
+    for codepoint in range(0x80, 0xa0):
+        try:
+            table[codepoint] = bytes([codepoint]).decode("cp1252")
+        except UnicodeDecodeError:
+            table[codepoint] = None
+    table[0x0435] = "e"
+    table[0x03cc] = "ó"
+    for codepoint in (0x2005, 0x205f, 0x200a):
+        table[codepoint] = " "
+    for codepoint in (0x200b, 0x200c, 0x200d, 0xfeff):
+        table[codepoint] = None
+    return table
+
+
+NORMALIZATION: dict[int, str | None] = _build_normalization()
+"""The codepoint-to-replacement mapping applied to fetched
+lyrics: cp1252-mojibake restoration for U+0080-U+009F (with the
+five byte values undefined in cp1252 removed), homoglyph
+restoration for the Cyrillic "e" and the Greek "o" with tonos,
+ASCII-space restoration for exotic space variants, and removal
+of zero-width characters.  A replacement of None removes the
+codepoint."""
+
+
+def normalize_lyrics(text: str) -> str:
+    """Restore or remove watermark and mojibake characters.
+
+    :param text: The lyrics text as fetched from an API.
+    :return: The text with the codepoints in
+        :data:`NORMALIZATION` replaced or removed; every other
+        character is unchanged.
+    """
+    return text.translate(NORMALIZATION)
+
+
 def parse_args(argv: list[str] | None) -> argparse.Namespace:
     """Parse the command-line arguments.
 
@@ -187,6 +230,9 @@ def save_lyrics(lyrics_dir: Path, song_id: int,
 
     The cache directory is created when missing.
 
+    The lyrics text is normalized with :func:`normalize_lyrics`
+    before being written.
+
     :param lyrics_dir: The lyrics cache directory.
     :param song_id: The song ID.
     :param lyrics: The lyrics text.
@@ -195,7 +241,7 @@ def save_lyrics(lyrics_dir: Path, song_id: int,
     """
     lyrics_dir.mkdir(parents=True, exist_ok=True)
     (lyrics_dir / f"{song_id}.txt").write_text(
-        lyrics, encoding="utf-8")
+        normalize_lyrics(lyrics), encoding="utf-8")
 
 
 def append_provenance(path: Path, song_id: int,

tools/tests/test_fetch_lyrics.py

@@ -317,6 +317,79 @@ class TestFetchLyrics(unittest.TestCase):
         self.assertEqual([x[0] for x in rows[1:]],
                          ["1", "2", "2"])
 
+    def test_normalize_cp1252_mojibake(self) -> None:
+        """Test that cp1252 mojibake codepoints are restored."""
+        self.assertEqual(
+            fetch_lyrics.normalize_lyrics("wait…"),
+            "wait…")
+        self.assertEqual(
+            fetch_lyrics.normalize_lyrics(
+                "‘quote’"),
+            "‘quote’")
+        self.assertEqual(
+            fetch_lyrics.normalize_lyrics(
+                "“quote”"),
+            "“quote”")
+        self.assertEqual(
+            fetch_lyrics.normalize_lyrics("dash—line"),
+            "dash—line")
+
+    def test_normalize_undefined_cp1252_removed(self) -> None:
+        """Test that undefined cp1252 byte values are removed."""
+        text: str = ("abcdef")
+        self.assertEqual(
+            fetch_lyrics.normalize_lyrics(text), "abcdef")
+
+    def test_normalize_homoglyphs(self) -> None:
+        """Test that watermark homoglyphs are restored."""
+        self.assertEqual(
+            fetch_lyrics.normalize_lyrics("likе that"),
+            "like that")
+        self.assertEqual(
+            fetch_lyrics.normalize_lyrics("lό que soy"),
+            "ló que soy")
+
+    def test_normalize_space_variants(self) -> None:
+        """Test that exotic space variants become ASCII space."""
+        self.assertEqual(
+            fetch_lyrics.normalize_lyrics(
+                "a b c d"),
+            "a b c d")
+
+    def test_normalize_zero_width_removed(self) -> None:
+        """Test that zero-width characters are removed."""
+        text: str = (
+            "a​b‌c‍de")
+        self.assertEqual(
+            fetch_lyrics.normalize_lyrics(text), "abcde")
+
+    def test_normalize_ascii_unchanged(self) -> None:
+        """Test that plain ASCII text passes through unchanged."""
+        text: str = "Hello, it's me\n"
+        self.assertEqual(
+            fetch_lyrics.normalize_lyrics(text), text)
+
+    def test_normalize_legitimate_non_ascii_unchanged(self) -> None:
+        """Test that legitimate non-ASCII content is unchanged."""
+        text: str = "¿cómo estás? 안녕하세요\n"
+        self.assertEqual(
+            fetch_lyrics.normalize_lyrics(text), text)
+
+    def test_fetched_lyrics_saved_normalized(self) -> None:
+        """Test that a fetched lyric is normalized before saving."""
+        self.__seed([("Hello", "Adele")])
+        with mock.patch(
+                "urllib.request.urlopen",
+                side_effect=[self.__response(
+                    {"lyrics": "wait… likе"
+                               " that now​\n"})]):
+            status: int = self.__run_fetch()[0]
+        self.assertEqual(status, 0)
+        self.assertEqual(
+            (self.__lyrics / "1.txt")
+            .read_text(encoding="utf-8"),
+            "wait… like that now\n")
+
     def test_no_store_fails(self) -> None:
         """Test that a missing working store fails the run."""
         urlopen: mock.Mock