diff --git a/README.md b/README.md index eccda66..1d21574 100644 --- a/README.md +++ b/README.md @@ -24,8 +24,7 @@ Hot 100(2016–2025)為例的內容分析。 ../data/source/yearend_hot100_2016_2025.csv ../data/derived --lyrics-dir ../data/captures/lyrics - --wikidata-csv ../data/captures/artists-wikidata.csv - --overrides-csv ../data/manual/artists_overrides.csv`)。 + --wikidata-csv ../data/captures/artists-wikidata.csv`)。 LLM 步驟使用 `claude-sonnet-4-6`、temperature=0、 thinking 關閉;每步驟獨立執行兩次後由仲裁步驟合併 (「2+1」協定)。 diff --git a/docs/decision_log.md b/docs/decision_log.md index 78f5561..37a3a98 100644 --- a/docs/decision_log.md +++ b/docs/decision_log.md @@ -180,3 +180,18 @@ 推測不確定且無佐證者列疑慮清單;查證屬資料策展而非 分析,以 Claude Code 輔助、不走分析 API;QS 批次以 私人工作檔留存,不隨論文發布。 + +## 2026-08-04 + +- **移除人工 overrides 層**:`build-db` 刪去 + `--overrides-csv` 選項與套用邏輯,文件同步移除 + `data/manual/artists_overrides.csv`。理由:「先補完 + Wikidata、再捕捉快照」工作流實跑後,469 位歌手全數 + 在上游查證補齊,快照即完整,本地覆蓋層已無存在事實; + `data/manual/` 層保留(供日後黃金標準編碼)。 +- **歌手型態刪去 mixed 值**:`ArtistType` 只留 solo/group。 + mixed 是先導研究「男/女/混合團體」單一欄位的殘留, + 正式設計拆成 gender+type 後從未定義其指涉;署名一律 + 拆成個人後,男女混合是歌曲層(song_artists+各歌手 + gender)可推導的事實,不屬歌手實體。非人非團體者 + (Pinkfong)type 留空由人工判定,維持現狀。 diff --git a/docs/project_structure.md b/docs/project_structure.md index 90c90d8..df0a8e5 100644 --- a/docs/project_structure.md +++ b/docs/project_structure.md @@ -21,7 +21,7 @@ pop-fem-audit/ │ │ └── lyrics/ # 歌詞 .txt 快取 │ │ # (gitignored,版權) │ ├── manual/ # 人工著作:只由研究者手寫 -│ │ └── artists_overrides.csv # 人工核定 / 深度背景 +│ │ # (黃金標準編碼等) │ └── derived/ # 衍生:只由 build-db 寫入 │ ├── songs.csv # 歌曲報表(人讀;進 git) │ └── artists.csv # 歌手報表(人讀;進 git) diff --git a/docs/research_plan.md b/docs/research_plan.md index 07363e5..40f97e5 100644 --- a/docs/research_plan.md +++ b/docs/research_plan.md @@ -33,7 +33,7 @@ ——一律以文字格式 commit。格式跟著層次走,不跟著偏好走。 - **分層**: - 源頭:原始榜單 CSV(進 git)。 - - 捕捉:Wikidata 快照 CSV、人工 overrides CSV、你的編碼 + - 捕捉:Wikidata 快照 CSV、你的編碼 CSV、`runs/` JSONL(皆進 git);歌詞 `.txt` 快取 (版權因素 gitignored,為已知的稽核缺口)。 - 工作儲存:SQLite 單檔(`tools/instance/`,generated、 @@ -79,7 +79,7 @@ | 階段 | 內容 | 方式 | 時程 | |---|---|---|---| | 0 | 基礎建設:git init、目錄結構、.gitignore、決策日誌、runner script(含 Batch API)、codebook v0 骨架 | script + 討論 | 7/30–7/31 | -| 1 | 資料準備:`run_llm` 改走統一設定 → `build-db`(解析榜單成 songs/chart_entries/artists/song_artists)→ `import-lyrics`(pilot 2018–2025)→ `fetch-lyrics`(2016–17 與缺漏,Lyrics.ovh / LRCLIB)→ `fetch-artists`(Wikidata 快照 + 人工 overrides)→ `export-llm-input` | 子命令 | 7/31–8/3 | +| 1 | 資料準備:`run_llm` 改走統一設定 → `build-db`(解析榜單成 songs/chart_entries/artists/song_artists)→ `import-lyrics`(pilot 2018–2025)→ `fetch-lyrics`(2016–17 與缺漏,Lyrics.ovh / LRCLIB)→ `fetch-artists`(Wikidata 快照)→ `export-llm-input` | 子命令 | 7/31–8/3 | | 2 | 候選篩選:全部唯一歌曲高召回 women-power 候選篩選(寧可多抓,人工剔除) | API 2+1 | 8/2–8/3 | | 3 | 黃金標準:依 codebook 人工逐首判定 genuine/peripheral/fake,附引用歌詞證據表(LLM 只做摘錄,不給判定建議);先以 10–15 首校準樣本試編並修訂 codebook 後凍結;同批校準樣本實測 Sonnet 4.6 vs Opus 5 一致率 | 人工 + script 輔助 | 8/4–8/8 | | 4 | 受控比較(盲點實驗):條件 A(詞彙層提示)vs 條件 B(框架感知提示),各 2+1,對照黃金標準計算假陽/假陰率 | API | 8/6–8/9 | diff --git a/tools/src/pop_fem_audit_tools/build_db.py b/tools/src/pop_fem_audit_tools/build_db.py index b546ba0..a8cabcb 100644 --- a/tools/src/pop_fem_audit_tools/build_db.py +++ b/tools/src/pop_fem_audit_tools/build_db.py @@ -8,8 +8,8 @@ Rebuilds the working store from scratch out of the committed inputs: the year-end chart CSV and the output directory for the review CSV files, given as the two positional command-line arguments, and the optional capture inputs, each given as an -option: the lyrics cache directory, the Wikidata artist snapshot -CSV, and the manual artist overrides CSV. An omitted option +option: the lyrics cache directory and the Wikidata artist +snapshot CSV. An omitted option leaves its capture layer unloaded; a given option whose path does not exist fails the build. Missing tables are created on a fresh store; existing tables are never altered, @@ -188,9 +188,6 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace: parser.add_argument( "--wikidata-csv", type=Path, default=None, help="the Wikidata artist snapshot CSV file to apply") - parser.add_argument( - "--overrides-csv", type=Path, default=None, - help="the manual artist override CSV file to apply") return parser.parse_args(argv) @@ -215,8 +212,7 @@ def parse_artist_credit(credit: str) -> list[tuple[str, Role]]: word or punctuation. Known limitation: a compound act name that contains one of the - delimiters, other than the protected names, is over-split; - such cases are corrected later via the human override layer. + delimiters, other than the protected names, is over-split. :param credit: The combined artist credit string. :return: The (name, role) pairs in credit order, primary side @@ -407,8 +403,7 @@ def apply_artist_csv(session: Session, path: Path) -> None: """Apply an artist attribute CSV onto the artist rows. Artists match by exact name. Only the non-empty cells are - applied, so a later CSV overrides an earlier one field by - field. The note column is ignored. + applied, field by field. The note column is ignored. :param session: The database session, with the artists flushed. @@ -727,8 +722,6 @@ def main(argv: list[str] | None = None) -> int: load_lyrics(session, args.lyrics_dir) if args.wikidata_csv is not None: apply_artist_csv(session, args.wikidata_csv) - if args.overrides_csv is not None: - apply_artist_csv(session, args.overrides_csv) session.flush() violations: list[str] = find_violations( session, YEARS, RANKS_PER_YEAR) diff --git a/tools/src/pop_fem_audit_tools/fetch_artists.py b/tools/src/pop_fem_audit_tools/fetch_artists.py index 8308b80..a1c9f50 100644 --- a/tools/src/pop_fem_audit_tools/fetch_artists.py +++ b/tools/src/pop_fem_audit_tools/fetch_artists.py @@ -106,9 +106,6 @@ class ArtistType(enum.StrEnum): """A solo artist: a human.""" GROUP = "group" """A musical ensemble.""" - MIXED = "mixed" - """A mixed act, assigned manually via the overrides; never - derived by the fetcher.""" @dataclass diff --git a/tools/src/pop_fem_audit_tools/models.py b/tools/src/pop_fem_audit_tools/models.py index 71a03c0..a75ea09 100644 --- a/tools/src/pop_fem_audit_tools/models.py +++ b/tools/src/pop_fem_audit_tools/models.py @@ -78,7 +78,7 @@ class Artist(Base): gender: Mapped[str | None] """The gender of the artist.""" type: Mapped[str | None] - """The artist type: solo, group, or mixed.""" + """The artist type: solo or group.""" genre: Mapped[str | None] """The music genre of the artist.""" country: Mapped[str | None] diff --git a/tools/tests/test_build_db.py b/tools/tests/test_build_db.py index 7e2fb56..937da1c 100644 --- a/tools/tests/test_build_db.py +++ b/tools/tests/test_build_db.py @@ -258,8 +258,6 @@ class TestBuildDB(unittest.TestCase): self.__lyrics: Path = self.__dir / "lyrics" self.__wikidata: Path = \ self.__dir / "artists_wikidata.csv" - self.__overrides: Path = \ - self.__dir / "artists_overrides.csv" self.__write_chart(self.CHART_CSV) url: str = f"sqlite:///{self.__dir}/store.sqlite3" config.set_settings(config.Settings( @@ -573,42 +571,6 @@ class TestBuildDB(unittest.TestCase): list(session.scalars(sa.select(ChartEntry))), []) self.assertEqual(list(session.scalars(sa.select(Song))), []) - def test_overrides_apply_over_wikidata(self) -> None: - """Test that the overrides win over the Wikidata snapshot.""" - self.__wikidata.write_text( - "name,qid,gender,type,genre,country,note\n" - "Adele,Q2831,female,solo,pop,GB,\n", - encoding="utf-8") - self.__overrides.write_text( - "name,qid,gender,type,genre,country,note\n" - "Adele,,,,soul,,manually checked\n", - encoding="utf-8") - self.assertEqual(self.__run_build( - "--wikidata-csv", str(self.__wikidata), - "--overrides-csv", str(self.__overrides))[0], 0) - session: Session = self.__session() - artist: Artist | None = session.scalar( - sa.select(Artist).where(Artist.name == "Adele")) - assert artist is not None - self.assertEqual(artist.genre, "soul") - self.assertEqual(artist.gender, "female") - self.assertEqual(artist.wikidata_qid, "Q2831") - self.assertEqual(artist.country, "GB") - - def test_unknown_override_name_fails(self) -> None: - """Test that an unknown override name fails the build.""" - self.__overrides.write_text( - "name,qid,gender,type,genre,country,note\n" - "Adel,,female,,,,typo\n", encoding="utf-8") - status: int - stderr: str - status, stderr = self.__run_build( - "--overrides-csv", str(self.__overrides)) - self.assertNotEqual(status, 0) - self.assertIn("Adel", stderr) - session: Session = self.__session() - self.assertEqual(list(session.scalars(sa.select(Song))), []) - def test_lyrics_loaded(self) -> None: """Test loading the lyrics cache into the songs.""" self.__lyrics.mkdir() @@ -676,18 +638,6 @@ class TestBuildDB(unittest.TestCase): session: Session = self.__session() self.assertEqual(list(session.scalars(sa.select(Song))), []) - def test_missing_overrides_csv_fails(self) -> None: - """Test that a given but missing override CSV fails.""" - status: int - stderr: str - status, stderr = self.__run_build( - "--overrides-csv", str(self.__overrides)) - self.assertNotEqual(status, 0) - self.assertIn("error:", stderr) - self.assertIn(str(self.__overrides), stderr) - session: Session = self.__session() - self.assertEqual(list(session.scalars(sa.select(Song))), []) - REVIEW_CHART_CSV: str = ( "year,rank,title,artist\n" "2016,1,banana,Artist B\n"