Compare commits
16
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
195605826f | ||
|
|
e569b3481c | ||
|
|
f0547208ed | ||
|
|
eef434acf7 | ||
|
|
7a625d50fc | ||
|
|
e1e2c86661 | ||
|
|
2fa55f29d7 | ||
|
|
676d7788e8 | ||
|
|
6d055c1ee5 | ||
|
|
bfbbe1875e | ||
|
|
a5fd86deb6 | ||
|
|
85d776da8b | ||
|
|
e3cca7e538 | ||
|
|
245b38c306 | ||
|
|
d470600eb3 | ||
|
|
e7e6787927 |
@@ -1,68 +0,0 @@
|
||||
# Project Conventions
|
||||
|
||||
## Analysis pipeline
|
||||
|
||||
- LLM analysis runs via Python scripts calling the Anthropic
|
||||
Messages API, Batch API where possible. Steps 1 and 3 run
|
||||
on `claude-sonnet-4-6` with `temperature=0` and thinking
|
||||
disabled; steps 4 and 5 run on `claude-fable-5`, which
|
||||
accepts neither parameter -- step 4 absorbs its sampling
|
||||
variance by the majority vote, step 5 by consolidating the
|
||||
three readings.
|
||||
- Prompt definition files live in
|
||||
`prompts/<step>-<substep>-<task>.md` (e.g. 01-tag.md; no
|
||||
version suffix -- versions live in git history) and are
|
||||
passed verbatim as the system prompt. The number names a
|
||||
step of the research procedure, not the file: the
|
||||
deterministic vocabulary step (step 2) has no definition
|
||||
file yet holds its own number. Zero padding is for
|
||||
sorting only -- prose says "step 1", "step 3".
|
||||
- Itemwise LLM judgments (per-song coding in step 3,
|
||||
per-keyword group selection in step 4) run the same
|
||||
definition file three times, independently, over the same
|
||||
input; a deterministic tally then assigns an item (a
|
||||
(song, keyword) or (group, keyword) pair) when at least
|
||||
two of the three runs assign it ("3 runs + majority
|
||||
vote"). Free-generation steps run
|
||||
twice and both outputs are pooled. The step-5 qualitative
|
||||
readings are neither: three independent readings per song,
|
||||
consolidated per song and synthesized across songs by
|
||||
their own definition files -- a qualitative protocol, not
|
||||
a vote (see docs/methodology.md). The vocabulary is
|
||||
built by a deterministic subcommand (embedding +
|
||||
clustering), not by an LLM. If a validation outcome is
|
||||
unexpected, revise the definition file and repeat that
|
||||
cycle; never patch results by hand.
|
||||
- Each run of a step is archived self-contained under the
|
||||
destination directory given explicitly on the `run-llm`
|
||||
command line (by convention `runs/<step>/run<N>/`):
|
||||
prompt snapshot, raw output, and `meta.json` (model ID,
|
||||
parameters, timestamps, batch ID). The runs of a step are
|
||||
that many separate invocations of `run-llm`. Replacing
|
||||
an existing run archive requires an explicit flag;
|
||||
superseded runs live in git history. Deterministic steps
|
||||
archive under `runs/<step>/` with no `run<N>` level.
|
||||
- Token usage and cost of every `run-llm` execution are
|
||||
recorded in `docs/run-costs.md` in the same commit as the
|
||||
run archive.
|
||||
- Scripts read the API key from the `ANTHROPIC_API_KEY`
|
||||
environment variable (`.env`, gitignored).
|
||||
|
||||
## Data rules
|
||||
|
||||
- `data/source/` holds the immutable hand-placed raw files;
|
||||
`data/captures/` is written only by the fetch commands and the
|
||||
private import script; `data/manual/` is written only by the
|
||||
user's own hand; `data/derived/` is written only by the
|
||||
`build-db` subcommand.
|
||||
- Full lyrics are copyrighted: they stay in `data/captures/lyrics/`
|
||||
(gitignored) and must never be committed or reproduced in
|
||||
full anywhere in the repo.
|
||||
|
||||
## Documents
|
||||
|
||||
- `results/` holds the final tallied tables (what the paper
|
||||
cites); `runs/` holds raw audit records. The paper cites
|
||||
`results/` only.
|
||||
- Any change to a definition file, the codebook, or the plan is
|
||||
recorded in `docs/decision-log.md` with date and reason.
|
||||
@@ -0,0 +1,396 @@
|
||||
Attribution 4.0 International
|
||||
|
||||
=======================================================================
|
||||
|
||||
Creative Commons Corporation ("Creative Commons") is not a law firm and
|
||||
does not provide legal services or legal advice. Distribution of
|
||||
Creative Commons public licenses does not create a lawyer-client or
|
||||
other relationship. Creative Commons makes its licenses and related
|
||||
information available on an "as-is" basis. Creative Commons gives no
|
||||
warranties regarding its licenses, any material licensed under their
|
||||
terms and conditions, or any related information. Creative Commons
|
||||
disclaims all liability for damages resulting from their use to the
|
||||
fullest extent possible.
|
||||
|
||||
Using Creative Commons Public Licenses
|
||||
|
||||
Creative Commons public licenses provide a standard set of terms and
|
||||
conditions that creators and other rights holders may use to share
|
||||
original works of authorship and other material subject to copyright
|
||||
and certain other rights specified in the public license below. The
|
||||
following considerations are for informational purposes only, are not
|
||||
exhaustive, and do not form part of our licenses.
|
||||
|
||||
Considerations for licensors: Our public licenses are
|
||||
intended for use by those authorized to give the public
|
||||
permission to use material in ways otherwise restricted by
|
||||
copyright and certain other rights. Our licenses are
|
||||
irrevocable. Licensors should read and understand the terms
|
||||
and conditions of the license they choose before applying it.
|
||||
Licensors should also secure all rights necessary before
|
||||
applying our licenses so that the public can reuse the
|
||||
material as expected. Licensors should clearly mark any
|
||||
material not subject to the license. This includes other CC-
|
||||
licensed material, or material used under an exception or
|
||||
limitation to copyright. More considerations for licensors:
|
||||
wiki.creativecommons.org/Considerations_for_licensors
|
||||
|
||||
Considerations for the public: By using one of our public
|
||||
licenses, a licensor grants the public permission to use the
|
||||
licensed material under specified terms and conditions. If
|
||||
the licensor's permission is not necessary for any reason--for
|
||||
example, because of any applicable exception or limitation to
|
||||
copyright--then that use is not regulated by the license. Our
|
||||
licenses grant only permissions under copyright and certain
|
||||
other rights that a licensor has authority to grant. Use of
|
||||
the licensed material may still be restricted for other
|
||||
reasons, including because others have copyright or other
|
||||
rights in the material. A licensor may make special requests,
|
||||
such as asking that all changes be marked or described.
|
||||
Although not required by our licenses, you are encouraged to
|
||||
respect those requests where reasonable. More considerations
|
||||
for the public:
|
||||
wiki.creativecommons.org/Considerations_for_licensees
|
||||
|
||||
=======================================================================
|
||||
|
||||
Creative Commons Attribution 4.0 International Public License
|
||||
|
||||
By exercising the Licensed Rights (defined below), You accept and agree
|
||||
to be bound by the terms and conditions of this Creative Commons
|
||||
Attribution 4.0 International Public License ("Public License"). To the
|
||||
extent this Public License may be interpreted as a contract, You are
|
||||
granted the Licensed Rights in consideration of Your acceptance of
|
||||
these terms and conditions, and the Licensor grants You such rights in
|
||||
consideration of benefits the Licensor receives from making the
|
||||
Licensed Material available under these terms and conditions.
|
||||
|
||||
|
||||
Section 1 -- Definitions.
|
||||
|
||||
a. Adapted Material means material subject to Copyright and Similar
|
||||
Rights that is derived from or based upon the Licensed Material
|
||||
and in which the Licensed Material is translated, altered,
|
||||
arranged, transformed, or otherwise modified in a manner requiring
|
||||
permission under the Copyright and Similar Rights held by the
|
||||
Licensor. For purposes of this Public License, where the Licensed
|
||||
Material is a musical work, performance, or sound recording,
|
||||
Adapted Material is always produced where the Licensed Material is
|
||||
synched in timed relation with a moving image.
|
||||
|
||||
b. Adapter's License means the license You apply to Your Copyright
|
||||
and Similar Rights in Your contributions to Adapted Material in
|
||||
accordance with the terms and conditions of this Public License.
|
||||
|
||||
c. Copyright and Similar Rights means copyright and/or similar rights
|
||||
closely related to copyright including, without limitation,
|
||||
performance, broadcast, sound recording, and Sui Generis Database
|
||||
Rights, without regard to how the rights are labeled or
|
||||
categorized. For purposes of this Public License, the rights
|
||||
specified in Section 2(b)(1)-(2) are not Copyright and Similar
|
||||
Rights.
|
||||
|
||||
d. Effective Technological Measures means those measures that, in the
|
||||
absence of proper authority, may not be circumvented under laws
|
||||
fulfilling obligations under Article 11 of the WIPO Copyright
|
||||
Treaty adopted on December 20, 1996, and/or similar international
|
||||
agreements.
|
||||
|
||||
e. Exceptions and Limitations means fair use, fair dealing, and/or
|
||||
any other exception or limitation to Copyright and Similar Rights
|
||||
that applies to Your use of the Licensed Material.
|
||||
|
||||
f. Licensed Material means the artistic or literary work, database,
|
||||
or other material to which the Licensor applied this Public
|
||||
License.
|
||||
|
||||
g. Licensed Rights means the rights granted to You subject to the
|
||||
terms and conditions of this Public License, which are limited to
|
||||
all Copyright and Similar Rights that apply to Your use of the
|
||||
Licensed Material and that the Licensor has authority to license.
|
||||
|
||||
h. Licensor means the individual(s) or entity(ies) granting rights
|
||||
under this Public License.
|
||||
|
||||
i. Share means to provide material to the public by any means or
|
||||
process that requires permission under the Licensed Rights, such
|
||||
as reproduction, public display, public performance, distribution,
|
||||
dissemination, communication, or importation, and to make material
|
||||
available to the public including in ways that members of the
|
||||
public may access the material from a place and at a time
|
||||
individually chosen by them.
|
||||
|
||||
j. Sui Generis Database Rights means rights other than copyright
|
||||
resulting from Directive 96/9/EC of the European Parliament and of
|
||||
the Council of 11 March 1996 on the legal protection of databases,
|
||||
as amended and/or succeeded, as well as other essentially
|
||||
equivalent rights anywhere in the world.
|
||||
|
||||
k. You means the individual or entity exercising the Licensed Rights
|
||||
under this Public License. Your has a corresponding meaning.
|
||||
|
||||
|
||||
Section 2 -- Scope.
|
||||
|
||||
a. License grant.
|
||||
|
||||
1. Subject to the terms and conditions of this Public License,
|
||||
the Licensor hereby grants You a worldwide, royalty-free,
|
||||
non-sublicensable, non-exclusive, irrevocable license to
|
||||
exercise the Licensed Rights in the Licensed Material to:
|
||||
|
||||
a. reproduce and Share the Licensed Material, in whole or
|
||||
in part; and
|
||||
|
||||
b. produce, reproduce, and Share Adapted Material.
|
||||
|
||||
2. Exceptions and Limitations. For the avoidance of doubt, where
|
||||
Exceptions and Limitations apply to Your use, this Public
|
||||
License does not apply, and You do not need to comply with
|
||||
its terms and conditions.
|
||||
|
||||
3. Term. The term of this Public License is specified in Section
|
||||
6(a).
|
||||
|
||||
4. Media and formats; technical modifications allowed. The
|
||||
Licensor authorizes You to exercise the Licensed Rights in
|
||||
all media and formats whether now known or hereafter created,
|
||||
and to make technical modifications necessary to do so. The
|
||||
Licensor waives and/or agrees not to assert any right or
|
||||
authority to forbid You from making technical modifications
|
||||
necessary to exercise the Licensed Rights, including
|
||||
technical modifications necessary to circumvent Effective
|
||||
Technological Measures. For purposes of this Public License,
|
||||
simply making modifications authorized by this Section 2(a)
|
||||
(4) never produces Adapted Material.
|
||||
|
||||
5. Downstream recipients.
|
||||
|
||||
a. Offer from the Licensor -- Licensed Material. Every
|
||||
recipient of the Licensed Material automatically
|
||||
receives an offer from the Licensor to exercise the
|
||||
Licensed Rights under the terms and conditions of this
|
||||
Public License.
|
||||
|
||||
b. No downstream restrictions. You may not offer or impose
|
||||
any additional or different terms or conditions on, or
|
||||
apply any Effective Technological Measures to, the
|
||||
Licensed Material if doing so restricts exercise of the
|
||||
Licensed Rights by any recipient of the Licensed
|
||||
Material.
|
||||
|
||||
6. No endorsement. Nothing in this Public License constitutes or
|
||||
may be construed as permission to assert or imply that You
|
||||
are, or that Your use of the Licensed Material is, connected
|
||||
with, or sponsored, endorsed, or granted official status by,
|
||||
the Licensor or others designated to receive attribution as
|
||||
provided in Section 3(a)(1)(A)(i).
|
||||
|
||||
b. Other rights.
|
||||
|
||||
1. Moral rights, such as the right of integrity, are not
|
||||
licensed under this Public License, nor are publicity,
|
||||
privacy, and/or other similar personality rights; however, to
|
||||
the extent possible, the Licensor waives and/or agrees not to
|
||||
assert any such rights held by the Licensor to the limited
|
||||
extent necessary to allow You to exercise the Licensed
|
||||
Rights, but not otherwise.
|
||||
|
||||
2. Patent and trademark rights are not licensed under this
|
||||
Public License.
|
||||
|
||||
3. To the extent possible, the Licensor waives any right to
|
||||
collect royalties from You for the exercise of the Licensed
|
||||
Rights, whether directly or through a collecting society
|
||||
under any voluntary or waivable statutory or compulsory
|
||||
licensing scheme. In all other cases the Licensor expressly
|
||||
reserves any right to collect such royalties.
|
||||
|
||||
|
||||
Section 3 -- License Conditions.
|
||||
|
||||
Your exercise of the Licensed Rights is expressly made subject to the
|
||||
following conditions.
|
||||
|
||||
a. Attribution.
|
||||
|
||||
1. If You Share the Licensed Material (including in modified
|
||||
form), You must:
|
||||
|
||||
a. retain the following if it is supplied by the Licensor
|
||||
with the Licensed Material:
|
||||
|
||||
i. identification of the creator(s) of the Licensed
|
||||
Material and any others designated to receive
|
||||
attribution, in any reasonable manner requested by
|
||||
the Licensor (including by pseudonym if
|
||||
designated);
|
||||
|
||||
ii. a copyright notice;
|
||||
|
||||
iii. a notice that refers to this Public License;
|
||||
|
||||
iv. a notice that refers to the disclaimer of
|
||||
warranties;
|
||||
|
||||
v. a URI or hyperlink to the Licensed Material to the
|
||||
extent reasonably practicable;
|
||||
|
||||
b. indicate if You modified the Licensed Material and
|
||||
retain an indication of any previous modifications; and
|
||||
|
||||
c. indicate the Licensed Material is licensed under this
|
||||
Public License, and include the text of, or the URI or
|
||||
hyperlink to, this Public License.
|
||||
|
||||
2. You may satisfy the conditions in Section 3(a)(1) in any
|
||||
reasonable manner based on the medium, means, and context in
|
||||
which You Share the Licensed Material. For example, it may be
|
||||
reasonable to satisfy the conditions by providing a URI or
|
||||
hyperlink to a resource that includes the required
|
||||
information.
|
||||
|
||||
3. If requested by the Licensor, You must remove any of the
|
||||
information required by Section 3(a)(1)(A) to the extent
|
||||
reasonably practicable.
|
||||
|
||||
4. If You Share Adapted Material You produce, the Adapter's
|
||||
License You apply must not prevent recipients of the Adapted
|
||||
Material from complying with this Public License.
|
||||
|
||||
|
||||
Section 4 -- Sui Generis Database Rights.
|
||||
|
||||
Where the Licensed Rights include Sui Generis Database Rights that
|
||||
apply to Your use of the Licensed Material:
|
||||
|
||||
a. for the avoidance of doubt, Section 2(a)(1) grants You the right
|
||||
to extract, reuse, reproduce, and Share all or a substantial
|
||||
portion of the contents of the database;
|
||||
|
||||
b. if You include all or a substantial portion of the database
|
||||
contents in a database in which You have Sui Generis Database
|
||||
Rights, then the database in which You have Sui Generis Database
|
||||
Rights (but not its individual contents) is Adapted Material; and
|
||||
|
||||
c. You must comply with the conditions in Section 3(a) if You Share
|
||||
all or a substantial portion of the contents of the database.
|
||||
|
||||
For the avoidance of doubt, this Section 4 supplements and does not
|
||||
replace Your obligations under this Public License where the Licensed
|
||||
Rights include other Copyright and Similar Rights.
|
||||
|
||||
|
||||
Section 5 -- Disclaimer of Warranties and Limitation of Liability.
|
||||
|
||||
a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
|
||||
EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
|
||||
AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
|
||||
ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
|
||||
IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
|
||||
WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
|
||||
PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
|
||||
ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
|
||||
KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
|
||||
ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
|
||||
|
||||
b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
|
||||
TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
|
||||
NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
|
||||
INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
|
||||
COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
|
||||
USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
|
||||
ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
|
||||
DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
|
||||
IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
|
||||
|
||||
c. The disclaimer of warranties and limitation of liability provided
|
||||
above shall be interpreted in a manner that, to the extent
|
||||
possible, most closely approximates an absolute disclaimer and
|
||||
waiver of all liability.
|
||||
|
||||
|
||||
Section 6 -- Term and Termination.
|
||||
|
||||
a. This Public License applies for the term of the Copyright and
|
||||
Similar Rights licensed here. However, if You fail to comply with
|
||||
this Public License, then Your rights under this Public License
|
||||
terminate automatically.
|
||||
|
||||
b. Where Your right to use the Licensed Material has terminated under
|
||||
Section 6(a), it reinstates:
|
||||
|
||||
1. automatically as of the date the violation is cured, provided
|
||||
it is cured within 30 days of Your discovery of the
|
||||
violation; or
|
||||
|
||||
2. upon express reinstatement by the Licensor.
|
||||
|
||||
For the avoidance of doubt, this Section 6(b) does not affect any
|
||||
right the Licensor may have to seek remedies for Your violations
|
||||
of this Public License.
|
||||
|
||||
c. For the avoidance of doubt, the Licensor may also offer the
|
||||
Licensed Material under separate terms or conditions or stop
|
||||
distributing the Licensed Material at any time; however, doing so
|
||||
will not terminate this Public License.
|
||||
|
||||
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
|
||||
License.
|
||||
|
||||
|
||||
Section 7 -- Other Terms and Conditions.
|
||||
|
||||
a. The Licensor shall not be bound by any additional or different
|
||||
terms or conditions communicated by You unless expressly agreed.
|
||||
|
||||
b. Any arrangements, understandings, or agreements regarding the
|
||||
Licensed Material not stated herein are separate from and
|
||||
independent of the terms and conditions of this Public License.
|
||||
|
||||
|
||||
Section 8 -- Interpretation.
|
||||
|
||||
a. For the avoidance of doubt, this Public License does not, and
|
||||
shall not be interpreted to, reduce, limit, restrict, or impose
|
||||
conditions on any use of the Licensed Material that could lawfully
|
||||
be made without permission under this Public License.
|
||||
|
||||
b. To the extent possible, if any provision of this Public License is
|
||||
deemed unenforceable, it shall be automatically reformed to the
|
||||
minimum extent necessary to make it enforceable. If the provision
|
||||
cannot be reformed, it shall be severed from this Public License
|
||||
without affecting the enforceability of the remaining terms and
|
||||
conditions.
|
||||
|
||||
c. No term or condition of this Public License will be waived and no
|
||||
failure to comply consented to unless expressly agreed to by the
|
||||
Licensor.
|
||||
|
||||
d. Nothing in this Public License constitutes or may be interpreted
|
||||
as a limitation upon, or waiver of, any privileges and immunities
|
||||
that apply to the Licensor or You, including from the legal
|
||||
processes of any jurisdiction or authority.
|
||||
|
||||
|
||||
=======================================================================
|
||||
|
||||
Creative Commons is not a party to its public
|
||||
licenses. Notwithstanding, Creative Commons may elect to apply one of
|
||||
its public licenses to material it publishes and in those instances
|
||||
will be considered the “Licensor.” The text of the Creative Commons
|
||||
public licenses is dedicated to the public domain under the CC0 Public
|
||||
Domain Dedication. Except for the limited purpose of indicating that
|
||||
material is shared under a Creative Commons public license or as
|
||||
otherwise permitted by the Creative Commons policies published at
|
||||
creativecommons.org/policies, Creative Commons does not authorize the
|
||||
use of the trademark "Creative Commons" or any other trademark or logo
|
||||
of Creative Commons without its prior written consent including,
|
||||
without limitation, in connection with any unauthorized modifications
|
||||
to any of its public licenses or any other arrangements,
|
||||
understandings, or agreements concerning use of licensed material. For
|
||||
the avoidance of doubt, this paragraph does not form part of the
|
||||
public licenses.
|
||||
|
||||
Creative Commons may be contacted at creativecommons.org.
|
||||
|
||||
@@ -1,40 +1,39 @@
|
||||
# pop-fem-audit
|
||||
# 流行音樂中「女性力量」語彙的挪用與污染——以 Billboard Year-End Hot 100(2016-2025)為例的內容分析
|
||||
|
||||
流行音樂中「女性力量」語彙的挪用與污染——以 Billboard Year-End
|
||||
Hot 100(2016–2025)為例的內容分析。
|
||||
這是研究論文《流行音樂中「女性力量」語彙的挪用與污染——以 Billboard Year-End Hot 100(2016-2025)為例的內容分析》的專案資料,包括論文本身、文件、紀錄、資料、工具程式、LLM提示詞,等等。
|
||||
|
||||
台灣女性學學會 2026 年會論文之研究資料與分析程式。
|
||||
## 論文和摘要
|
||||
|
||||
## 目錄結構
|
||||
研究論文全文和摘要,請參閱 paper/ 資料夾。
|
||||
|
||||
見 `docs/project-structure.md`。研究步驟規劃見
|
||||
`docs/research-plan.md`;方法細節見 `docs/methodology.md`;
|
||||
人工編碼手冊見 `docs/codebook.md`;決策日誌見
|
||||
`docs/decision-log.md`。
|
||||
## 文件說明
|
||||
|
||||
## 重現方式
|
||||
研究方法、記錄等文件,請參閱 docs/ 資料夾。
|
||||
|
||||
1. 準備 Python 3.14+ 環境,安裝分析管線套件:
|
||||
`pip install -e tools/`。
|
||||
2. 自 `tools/.env.example` 建立 `tools/.env`,寫入
|
||||
Anthropic API 金鑰。
|
||||
3. 依 `docs/research-plan.md` 的階段順序,於 `tools/` 目錄下
|
||||
執行子命令,必要輸入以位置引數、選擇性輸入以選項給定(如
|
||||
`pop-fem-audit-tools build-db
|
||||
../data/source/yearend_hot100_2016_2025.csv
|
||||
../data/derived
|
||||
--lyrics-dir ../data/captures/lyrics
|
||||
--wikidata-csv ../data/captures/artists-wikidata.csv`)。
|
||||
LLM 步驟使用 `claude-sonnet-4-6`、temperature=0、
|
||||
thinking 關閉;輸出可逐項比對的步驟獨立執行三次,由
|
||||
程式取三票多數決定案(「三次執行+多數決」協定),
|
||||
自由生成步驟兩次執行進池。
|
||||
4. 每次執行的完整紀錄(定義檔快照、原始輸出、參數)存於
|
||||
`runs/`,可逐筆稽核。論文引用的最終資料表在 `results/`。
|
||||
## LLM提示
|
||||
|
||||
注意:歌詞受版權保護,`data/captures/lyrics/` 不隨 repo 發布,須自行
|
||||
以 `tools/` 中的抓取程式重建。
|
||||
LLM提示請參閱 prompts/ 資料夾。
|
||||
|
||||
## 輔助工具程式
|
||||
|
||||
輔助工具程式請參閱 tools/ 資料夾。
|
||||
|
||||
## 研究資料
|
||||
|
||||
研究資料請參閱 data/ 資料夾。
|
||||
|
||||
## 授權
|
||||
|
||||
(待定)
|
||||
版權所有 © 2026 楊士青。
|
||||
|
||||
除另有註明外,本專案之內容均依「CC 姓名標示 4.0 國際」授權條款提供。
|
||||
|
||||
排除物件:
|
||||
1. 本專案引用之第三方歌詞,相應權利歸各該權利人所有;
|
||||
2. tools/ 資料夾內之工具程式,依該工具程式授權條款提供。
|
||||
|
||||
## 作者
|
||||
|
||||
楊士青<br>
|
||||
imacat@mail.imacat.idv.tw<br>
|
||||
2026/8/18<br>
|
||||
|
||||
|
Can't render this file because it is too large.
|
@@ -0,0 +1,69 @@
|
||||
# 常設工作規範
|
||||
|
||||
本專案的常設工作規範。原專案 `CLAUDE.md`;移置於此,
|
||||
使 Claude Code subagent 不將其繼承入 context(盲判型
|
||||
agent 不得見之)。凡於本專案工作的主會話,動手管線、
|
||||
資料或文件之前,先讀本檔。
|
||||
|
||||
## 分析管線
|
||||
|
||||
管線已跑畢;本規範適用於任何重跑或擴充。
|
||||
|
||||
- LLM 分析以 Python 腳本呼叫 Anthropic Messages API
|
||||
執行,能用 Batch API 處即用之。步驟 1 與步驟 3 以
|
||||
`claude-sonnet-4-6` 執行,`temperature=0`、thinking
|
||||
停用;步驟 4 與步驟 5 以 `claude-fable-5` 執行,該
|
||||
模型兩個參數皆不受理——步驟 4 的取樣變異由多數決
|
||||
吸收,步驟 5 由整合三份閱讀吸收。
|
||||
- 定義檔置於 `prompts/<步><次步>-<task>.md`(如
|
||||
1-tag.md、5a-read.md;次步以字母標示,與論文正文的
|
||||
步驟編號一致;不帶版本號——版本即 git 歷史),逐字
|
||||
作為 system prompt。
|
||||
- 逐項的 LLM 判斷(步驟 3 的逐首編碼、步驟 4 的逐碼
|
||||
入群判斷)以同一份定義檔、同一份輸入獨立執行三次;
|
||||
再由確定性計票將三次執行中至少兩次指派的項目(一個
|
||||
(歌,關鍵字)或(群,關鍵字)配對)收入定案
|
||||
(「三次執行+多數決」)。自由生成步驟執行兩次,
|
||||
兩份輸出進池。步驟 5 的質性閱讀兩者皆非:逐首三次
|
||||
獨立閱讀,由各自的定義檔逐首整合、跨首統整——質性
|
||||
協定,不是投票(見 docs/methodology.md)。詞彙表由
|
||||
確定性子命令建構(嵌入+分群),不經 LLM。驗證結果
|
||||
不符預期時,修訂定義檔並重複該循環;絕不手改結果。
|
||||
- 一步的每次執行皆自我完備歸檔於 `run-llm` 命令列上
|
||||
明示指定的目的目錄下(慣例為
|
||||
`data/runs/<步驟>/run<N>/`):定義檔快照、原始輸出與
|
||||
`meta.json`(model ID、參數、時間戳、batch ID)。
|
||||
一步的 N 次執行即 N 次各自的 `run-llm` 呼叫。覆蓋
|
||||
既有執行歸檔須明示旗標;被取代的執行留在 git 歷史。
|
||||
確定性步驟歸檔於 `data/runs/<步驟>/`,不分 `run<N>` 層。
|
||||
- 每次 `run-llm` 執行的 token 用量與費用記入
|
||||
`docs/run-costs.md`,與執行歸檔同一 commit。
|
||||
- 腳本自環境變數 `ANTHROPIC_API_KEY` 讀取 API key
|
||||
(`.env`,gitignored)。
|
||||
|
||||
## 資料規則
|
||||
|
||||
- `data/source/` 存手放後不動的原始檔;
|
||||
`data/captures/` 只由 fetch 命令與私人匯入腳本
|
||||
寫入;`data/manual/` 只由研究者親手寫入;
|
||||
`data/derived/` 只由 `build-db` 子命令寫入;
|
||||
`data/runs/` 存 LLM 執行的原始歸檔,只由執行程序
|
||||
寫入;`data/results/` 存論文引用的定案表,只由
|
||||
計票程序寫入。
|
||||
- 歌詞全文有版權:一律置於 `data/captures/lyrics/`
|
||||
(gitignored),絕不 commit,亦絕不於 repo 任何處
|
||||
全文重現。
|
||||
|
||||
## 文件
|
||||
|
||||
- `data/results/` 存計票後的定案表(論文所引);
|
||||
`data/runs/` 存原始稽核紀錄。論文只引 `data/results/`。
|
||||
- **Commit 判準**:凡能由「committed 的輸入+committed
|
||||
的程式」決定性再生者不 commit;凡不能者一律以文字
|
||||
格式 commit,格式跟著上文「資料規則」一節所定的
|
||||
層次走。例外:`data/results/` 定案表與
|
||||
`data/derived/` 人讀報表雖可再生仍 commit——理由是
|
||||
引用穩定性、審稿人零門檻、撰稿期數字變動可 diff;
|
||||
兩者皆與工作儲存同一動作產出,稽核鏈無中間空缺。
|
||||
- 凡定義檔或研究規劃之更動,皆記入
|
||||
`docs/decision-log.md`,註明日期與原因。
|
||||
@@ -899,3 +899,48 @@
|
||||
展延一至二日與剩餘撰寫工作。仍然有效的原則(先導不
|
||||
比較、提示只定格式、歌手背景防火牆、commit 判準、
|
||||
版權規則)保留。
|
||||
|
||||
## 2026-08-17
|
||||
|
||||
- **先導研究的地位另立常設紀錄 `docs/pilot-study.md`**:論文
|
||||
交件後重讀投稿摘要,認清先導研究係由當時協作的 Claude Code
|
||||
設計並執行,研究者檢視的是研究問題與答案,未逐步檢視中間
|
||||
過程;摘要中的具體數字未經稽核,亦無歸檔可回溯。此地位
|
||||
先前僅零星散見於本日誌各條(歌詞沿用、women-power 來歷
|
||||
考據、粒度繼承、簿記容量),投稿摘要本身則只是產物,不足以
|
||||
記述過程。裁定:另立 `docs/pilot-study.md`,完整記述先導
|
||||
研究的執行方式、沿用與棄用的成果,以及它如何促成正式研究
|
||||
的可稽核設計;投稿摘要不於樹中另存副本,其內容留在 git
|
||||
歷史(`paper/abstract.md` 的前身)。
|
||||
|
||||
## 2026-08-18
|
||||
|
||||
- **工序編號改與論文正文一致**:論文正文將五個步驟的細分
|
||||
工序以字母標示(表三「步驟3a」、表四「步驟5d」),repo
|
||||
的檔名與歸檔目錄卻沿用 `05-01`、`05-02` 這種數字次步,
|
||||
兩邊對不上。裁定:改為 `1-tag`、`2-cluster`、`3a-code`、
|
||||
`4-group`、`5a-read`、`5b-consolidate`、`5c-synthesize`、
|
||||
`5d-annotate`(`prompts/` 與 `runs/` 同步,以 `git mv`
|
||||
保留歷史),補零一併取消。`runs/*/meta.json`
|
||||
內記的 `prompt_path` 不追改——它是執行當下的實況記錄;
|
||||
本日誌的既有條目同理,維持當時的名稱。`run-costs.md` 則
|
||||
相反:它的「步驟」欄是查閱歸檔的索引,故一律換為現行
|
||||
編號(`01-01-01-tag`→`1-tag`、`03-01-code` 與
|
||||
`03-code`→`3a-code`),並新增末欄「原步驟名」保留執行
|
||||
當時的名稱——索引要指得到現在的歸檔,紀錄要留得住當時
|
||||
的形狀,兩者以兩欄並存解決。已刪除的工序(步驟 2 的
|
||||
LLM 合併嘗試 `01-02-01-merge`、步驟 3 的仲裁
|
||||
`03-02-arbitration`)其歸檔已不存在,無現行編號可指,
|
||||
不另發新名,步驟欄逕標「已廢棄」,原名存於末欄。
|
||||
|
||||
- **CLAUDE.md 移為 `docs/conventions.md`**:專案 CLAUDE.md 會
|
||||
全文自動注入每一個 Claude Code subagent 的 context(2026-07-30
|
||||
實測條),步驟 6 的 883 個盲判 agent 因此都看到了工作規範。
|
||||
規範本身無研究語意(無 women-power、無假說),污染有限,但
|
||||
「防脈絡污染」的原則應貫徹到自家工具鏈:docs/ 底下的檔案
|
||||
不會自動注入,搬移後 subagent 除非主動翻閱否則不會看到,
|
||||
盲判 agent 的定義檔並明令不得查閱專案材料。代價是主會話
|
||||
也不再自動載入,須自行先讀 `docs/conventions.md`。內容
|
||||
順帶微調:註明管線已完成、規範適用於重跑;刪去已不存在
|
||||
的 codebook 一詞。進論文的數據一律走 Anthropic API(無
|
||||
檔案系統,機制上不可能讀到),此層防線不受本次搬移影響。
|
||||
|
||||
+101
-202
@@ -1,12 +1,12 @@
|
||||
# 方法細節
|
||||
|
||||
(全文方法節底稿。演算法在執行前寫定;任何修訂記入
|
||||
`decision-log.md`。定義檔全文見 `prompts/`,執行紀錄見
|
||||
`runs/`。)
|
||||
`decision-log.md`。)
|
||||
|
||||
## 自然編碼管線總覽
|
||||
|
||||
五個步驟:步驟 1 自由標註(兩次執行進池)→ 步驟 2 詞彙表
|
||||
五個步驟:步驟 1 自由標註(`claude-sonnet-4-6`,
|
||||
temperature=0、thinking 關閉,兩次執行進池)→ 步驟 2 詞彙表
|
||||
建構(詞向量分群,確定性)→ 步驟 3 全量編碼(三次執行+
|
||||
多數決)→ 步驟 4 語意編碼群(三次執行+多數決)→ 步驟 5
|
||||
女性主義問題之質性深讀(三次閱讀+逐首整合+樣態統整)。
|
||||
@@ -14,13 +14,6 @@
|
||||
不接觸歌詞,步驟 2 亦不呼叫 LLM。設計原則見
|
||||
`research-plan.md`;本檔記載可重現的演算法細節。
|
||||
|
||||
編號的所指為**研究程序的工序**,不是定義檔:步驟 1、
|
||||
步驟 3、步驟 4 與步驟 5 有定義檔(`prompts/`;步驟 5 依
|
||||
子工序有三份),步驟 2 沒有——它是單一確定性計算,由
|
||||
`cluster-keywords` 一個子命令完成。有無定義檔的區別即
|
||||
「該步是否為 LLM 判斷」,由 `prompts/` 是否存在同號檔案
|
||||
直接可見。
|
||||
|
||||
## 步驟 2 詞彙表建構——詞向量分群
|
||||
|
||||
詞彙表由確定性程序產生,不經 LLM。完整分割(每個關鍵字
|
||||
@@ -30,8 +23,7 @@
|
||||
### 進池
|
||||
|
||||
兩次標註執行的全部關鍵字取聯集、逐字串精確去重、字典序
|
||||
排列。失敗與拒答的記錄跳過(其歌曲不貢獻關鍵字);解析
|
||||
時偵測重複鍵,違規即失敗。
|
||||
排列。失敗與拒答的記錄跳過(其歌曲不貢獻關鍵字)。
|
||||
|
||||
### 分群
|
||||
|
||||
@@ -39,14 +31,10 @@
|
||||
(釘定 revision),關鍵字的連字號先還原為空格再編碼,
|
||||
輸出 768 維向量並 L2 正規化。
|
||||
- **分群**:階層式聚合分群(Ward linkage),k=100。
|
||||
向量既已正規化,歐氏距離與餘弦相似度單調對應;三種
|
||||
linkage 實測比較,Ward 於各個 k 的組內一致性均最高
|
||||
(average 與 complete 皆產生吞噬半數語料的巨大異質
|
||||
組)。
|
||||
向量既已正規化,歐氏距離與餘弦相似度單調對應。
|
||||
- **組數的取捨**:k 太小則壓縮比過高,樹上層被迫併入
|
||||
不相干的詞,組雖大而無主題(k=30 最大組 491 詞、
|
||||
組內一致性 0.41,成員橫跨籃球、海灘、外星人綁架);
|
||||
k 太大則人工難以通覽。定於 100,理由是實測顯示雜物櫃
|
||||
不相干的詞,組雖大而無主題;k 太大則人工難以通覽。
|
||||
定於 100,理由是實測顯示雜物櫃
|
||||
組於此始裂解為有主題的組,且編碼實測未見碼數過多的
|
||||
副作用——全量三次執行下 101 個碼全數用到;模型另行
|
||||
造出的碼共 13 筆,佔 44,149 筆標籤指派的 0.03%。
|
||||
@@ -55,39 +43,23 @@
|
||||
自己產出過的關鍵字,非任何人事後撰寫。已知限制:
|
||||
組越大越異質時,medoid 只是折衷詞,可能代表不了組內
|
||||
內容(實測 `mutual-individuality` 組內一致 0.70 而
|
||||
編碼從未使用);此類碼於結果中呈現為零使用,據實
|
||||
報告,不事後改名。
|
||||
- **取捨紀錄**:曾以 LLM 單發收斂(merge/cap 兩步)
|
||||
實作本步,四種模型六次執行全部無法維持完整分割,
|
||||
已棄用(詳見 `decision-log.md` 2026-08-05;棄用的
|
||||
定義檔止於 git 歷史,見 `git log -- prompts/`)。
|
||||
- **產物**:五份,前綴分別標示來源與結果。
|
||||
`source-keywords.txt`(進池後的關鍵字,一行一個)
|
||||
記錄進來的是什麼;`result-keywords.txt`(組名,一行
|
||||
一個)與 `groups.csv`(欄位 Group、Keyword,一列一個
|
||||
成員)記錄算出來的分割;`keywords-to-merge.json`
|
||||
(`{"keywords": [...]}`)是實際交給模型的碼,即組名
|
||||
加上先驗主題詞——五份中只有這一份含研究者的介入。
|
||||
`meta.json` 記錄執行本身:進池的兩份執行歸檔與其有效
|
||||
筆數、嵌入模型與釘定 revision、分群參數與組數、外加
|
||||
的先驗詞、關鍵字總數,以及產生數字的套件版本。凡命令
|
||||
列上的選擇與環境事實皆在此,不記時間戳與輸入雜湊
|
||||
——前者使同環境重跑逐位元組可再生,後者只會重述 git
|
||||
已保證的事。
|
||||
編碼從未使用);此類碼於結果中呈現為零使用。
|
||||
- **取捨紀錄**:詞彙表分群曾比較的替代法與棄用理由,見
|
||||
`decision-log.md` 2026-08-05 條。
|
||||
- **產物**:分群輸出中,只有實際交給模型的碼表——組名
|
||||
加上先驗主題詞——含研究者的介入;其餘皆為分群過程本身
|
||||
的機械紀錄。
|
||||
- **可重現性**:同一輸入、同一釘定模型、同一參數逐次
|
||||
重現。不同 CPU/BLAS 實作的浮點尾數差異可能使邊界
|
||||
詞的歸屬翻動,屬已揭露的限制;論文所用碼表逐字
|
||||
commit,引用單位為該份定案檔案。
|
||||
詞的歸屬翻動,屬已揭露的限制。
|
||||
|
||||
### women-power 的注入
|
||||
|
||||
定案詞彙表為 100 個分群組名再加上 `women-power` 一詞,
|
||||
共 101 個碼。`women-power` 是研究者任意決定的先驗主題(即本
|
||||
論文的主題本身),不由資料產生,屬揭露的儀器介入。該詞
|
||||
於執行時以 `--extra-keyword` 明示加入,不寫死在程式裏
|
||||
——研究者的介入因此每次都出現在重現命令上,而非無聲
|
||||
發生;分群結果的兩份產物不含它,只有交給模型的碼表含
|
||||
它。
|
||||
的加入於每次執行皆為明示、可稽核的介入,而非無聲內建於
|
||||
判斷邏輯;分群結果本身不含它,只有交給模型的碼表含它。
|
||||
|
||||
注入而非另設篩選軌的理由:讓研究者的主題詞與模型自己
|
||||
收斂出的類別(分群已自行長出 `female-empowerment` 等組)
|
||||
@@ -97,24 +69,17 @@
|
||||
|
||||
## 步驟 3 編碼的三次執行與多數決
|
||||
|
||||
- **模型**:`claude-sonnet-4-6`,temperature=0、thinking 關閉。
|
||||
- **三次執行**:同一份定義檔、同一份輸入檔,獨立執行
|
||||
三次,三份歸檔並列(`runs/03-code/run1`、`run2`、
|
||||
`run3`),彼此無先後主從之別。
|
||||
三次,三份歸檔並列,彼此無先後主從之別。
|
||||
- **多數決**:一首歌的一個標籤,三次執行中至少兩次標出
|
||||
即收入定案編碼。三票不平手,裁決規則因此無例外條款,
|
||||
計票由確定性子命令完成(見交接契約)。
|
||||
- **第三票取全量**:只對前兩次分歧的標籤補問第三票,
|
||||
計票結果相同;仍採全量執行——全部歌曲、全部關鍵字
|
||||
——使三票在同一條件下取得。
|
||||
計票由確定性程序完成(見交接契約)。
|
||||
- **第三票取全量**:第三次執行同為全量——全部歌曲、
|
||||
全部關鍵字——使三票在同一條件下取得。
|
||||
- **對邊緣標籤的作用**:兩次執行只分得出「兩次皆標」與
|
||||
「僅一次標」;三次執行還分得出 3-0 與 2-1,故「定案
|
||||
編碼中有多少比例僅以一票之差成立」成為可報告的量。
|
||||
至於判定本身,兩次執行相左的標籤在何種協定下都由第三
|
||||
個判斷定奪,票數不使不確定性消失:某標籤於單次執行被
|
||||
標出的傾向若恰為一半,任何票數皆為擲幣。三票之效在
|
||||
傾向偏離一半處——多數決將判定推向該傾向本身(單次
|
||||
0.7 者為 0.78,0.9 者為 0.97),程序重跑的一致性因而
|
||||
高於單次執行,唯獨恰半處無從改善。
|
||||
|
||||
## 步驟 4 語意編碼群
|
||||
|
||||
@@ -125,103 +90,79 @@
|
||||
由 LLM 依編碼名的字面語意判斷。
|
||||
- **任務**:每筆輸入為一個群名加 101 個編碼的字母序
|
||||
清單,輸出為入選編碼的單層 JSON 陣列;定義檔
|
||||
`prompts/04-group.md` 只定格式,不含任何群的語意定義。
|
||||
- **模型**:`claude-fable-5`(步驟 1、3 為
|
||||
`claude-sonnet-4-6`)。該模型不受理 `temperature` 與
|
||||
`thinking` 參數,兩者均不送出;取樣變異由多數決吸收。
|
||||
只定格式,不含任何群的語意定義。
|
||||
- **模型**:`claude-fable-5`;取樣變異由多數決吸收;
|
||||
模型裁定的理由與對照實驗見決策日誌。
|
||||
- **三次執行**:同一份定義檔、同一份輸入檔,獨立執行
|
||||
三次,歸檔並列(`runs/04-group/run1`、`run2`、`run3`)。
|
||||
三次,歸檔並列。
|
||||
- **多數決**:一個(群,編碼)配對,三次執行中至少兩次
|
||||
入選即屬該群;不在 101 碼詞彙表內的輸出項無效,每筆
|
||||
丟棄印於標準錯誤。計票由確定性子命令 `tally-groups`
|
||||
完成:`tally-groups <執行歸檔 1> <執行歸檔 2> <執行歸檔
|
||||
3> <合法碼清單> <輸出 CSV>`,合法碼清單之產法同步驟 3。
|
||||
定案分群寫入 `results/groups.csv`,欄位 `Group`、
|
||||
`Keyword`、`Votes`,列序先依群名、再依編碼,一律以
|
||||
Unicode 碼位比較,換行為 CRLF。
|
||||
- **工作儲存**:`build-db --groups <定案分群 CSV>` 將定案
|
||||
分群逐欄照存入 `groups` 資料表(群、編碼、票數),供
|
||||
群層次查詢。
|
||||
入選即屬該群;不在 101 碼詞彙表內的輸出項無效,逐筆
|
||||
記錄後丟棄。計票由確定性計票程序完成,合法碼
|
||||
清單之產法同步驟 3,結果為定案分群表。
|
||||
|
||||
## 步驟 5 女性主義問題之質性深讀
|
||||
|
||||
本步驟之 5-1 至 5-3 為**質性閱讀,非編碼**:輸出為自由
|
||||
文字的問題閱讀報告,無可逐項機械比對的單位,故不適用
|
||||
三票多數決與仲裁;
|
||||
三次獨立閱讀為分析者三角檢核,逐首整合為整合而非裁決,
|
||||
跨首統整之產出為草稿,終審與詮釋由研究者為之。論文引用
|
||||
本步驟之 5a 至 5c 為**質性閱讀,非編碼**:輸出為自由
|
||||
文字的問題閱讀報告,無可逐項機械比對的單位,故不適用
|
||||
三票多數決與仲裁;
|
||||
三次獨立閱讀為分析者三角檢核,逐首整合為整合而非裁決,
|
||||
跨首統整之產出為草稿,終審與詮釋由研究者為之。論文引用
|
||||
本步驟時不作次數宣稱。
|
||||
|
||||
- **對象**:定案編碼含 `women-power` 或
|
||||
- **對象**:定案編碼含 `women-power` 或
|
||||
`female-empowerment` 的 145 首歌。
|
||||
- **5-1 逐首閱讀**:每筆輸入為一首歌的完整歌詞逐字全文,
|
||||
不含歌名與演唱者(盲讀);定義檔
|
||||
`prompts/05-01-read.md`。同一份定義檔、同一份輸入檔,
|
||||
獨立執行三次,歸檔並列(`runs/05-01-read/run1`、
|
||||
`run2`、`run3`)。
|
||||
- **5-2 逐首整合**:每筆輸入為該首歌的三份閱讀報告
|
||||
(不含歌詞);以問題機制為單位保守合併,標收斂註記
|
||||
((3/3)、(2/3)),主清單僅列兩讀以上提出者,單讀發現
|
||||
以一行存目;定義檔 `prompts/05-02-consolidate.md`,
|
||||
執行一次,歸檔 `runs/05-02-consolidate/run1`。
|
||||
- **5-3 樣態統整**:單筆輸入為一批整合報告;歸納問題
|
||||
**樣態**——問題呈現與運作的重複形態,非問題分類,
|
||||
代表引句僅取自主清單;定義檔
|
||||
`prompts/05-03-synthesize.md`。四種輸入範圍各執行
|
||||
一次:全 145 首之基底統整(歸檔
|
||||
`runs/05-03-synthesize/run1`),及依 `performer_gender`
|
||||
(演唱聲音之性別)切分之三個發話脈絡統整——男聲
|
||||
(male,歸檔 `run2`)、女聲(female,歸檔 `run3`)、
|
||||
混合(mixed,歸檔 `run4`);基底看橫貫各脈絡之樣態,
|
||||
分組看各權力脈絡下之樣態。genderfluid 與 non-binary
|
||||
共 3 首不設群——樣本數不支持歸納——僅入基底統整,
|
||||
由研究者以個案閱讀。同一輸入不重複執行:自由歸納之
|
||||
產出無機械合併可言,其變異由 5-1 三讀、5-2 整合與
|
||||
草稿地位承接,四份草稿互為對照,由研究者終審裁決。
|
||||
- **5-4 樣態標註**:以 (歌, 樣態) 對為可逐項機械比對之
|
||||
單位,回歸「三次執行+多數決」協定。對象為「有問題」
|
||||
的歌——三讀中至多一個「無」(多數決精神;恰兩「無」
|
||||
者其整合報告主清單必為空,與 5-2 主清單規則自洽),
|
||||
計 111 首(男聲 12、女聲 69、混合 29、genderfluid 1)。
|
||||
樣態表為三份分組統整草稿原文:男聲 13 條(M1–M13)、
|
||||
女聲 14 條(F1–F14)、混合 16 條(X1–X16);基底 15 條
|
||||
- **5a 逐首閱讀**:每筆輸入為一首歌的完整歌詞逐字全文,
|
||||
不含歌名與演唱者(盲讀)。同一份定義檔、同一份輸入檔,
|
||||
獨立執行三次,歸檔並列。
|
||||
- **5b 逐首整合**:每筆輸入為該首歌的三份閱讀報告
|
||||
(不含歌詞);以問題機制為單位保守合併,標收斂註記
|
||||
((3/3)、(2/3)),主清單僅列兩讀以上提出者,單讀發現
|
||||
以一行存目;定義檔,執行一次。
|
||||
- **5c 樣態統整**:單筆輸入為一批整合報告;歸納問題
|
||||
**樣態**——問題呈現與運作的重複形態,非問題分類,
|
||||
代表引句僅取自主清單。四種輸入範圍各執行
|
||||
一次:全 145 首之基底統整,及依演唱聲音之性別切分
|
||||
之三個發話脈絡統整——男聲(male)、女聲(female)、
|
||||
混合(mixed);基底看橫貫各
|
||||
脈絡之樣態,分組看各權力脈絡下之樣態。genderfluid
|
||||
與 non-binary 共 3 首不設群——樣本數不支持歸納——
|
||||
僅入基底統整。同一輸入不重複
|
||||
執行:自由歸納之產出無機械合併可言,其變異由 5a
|
||||
三讀、5b 整合與草稿地位承接,四份草稿互為對照,由
|
||||
研究者終審裁決。
|
||||
- **5d 樣態標註**:以(歌,樣態)對為可逐項機械比對之
|
||||
單位,回歸「三次執行+多數決」協定。對象為「有問題」
|
||||
的歌——三讀中至多一個「無」(多數決精神;恰兩「無」
|
||||
者其整合報告主清單必為空,與 5b 主清單規則自洽),
|
||||
計 111 首(男聲 12、女聲 69、混合 29、genderfluid 1)。
|
||||
樣態表為三份分組統整草稿原文:男聲 13 條(M1–M13)、
|
||||
女聲 14 條(F1–F14)、混合 16 條(X1–X16);基底 15 條
|
||||
不入矩陣——全體歸納之一條樣態可能疊合不同方向的
|
||||
權力關係(男對女、女對男),實為多個樣態共用一名。
|
||||
檢驗範圍:男聲樣態不檢驗純女聲歌、女聲樣態不檢驗
|
||||
純男聲歌(發話位置範疇錯置,檢查無意義);混合樣態
|
||||
檢驗全部(其男女聲部無系統化切分方式,無意義之標註
|
||||
容忍之);genderfluid 歌三套全查(無自身透鏡,發話
|
||||
位置無法先驗決定),其歸屬引用維持個案地位。每筆
|
||||
輸入為一首歌之整合報告(「僅單獨提及」行於組裝時
|
||||
剝除,標註僅依主清單)與該首適用之樣態表;定義檔
|
||||
`prompts/05-04-annotate.md`,獨立執行三次
|
||||
(`runs/05-04-annotate/run1`~`run3`),(歌, 樣態) 對
|
||||
權力關係(男對女、女對男),實為多個樣態共用一名。
|
||||
檢驗範圍:男聲樣態不檢驗純女聲歌、女聲樣態不檢驗
|
||||
純男聲歌(發話位置範疇錯置,檢查無意義);混合樣態
|
||||
檢驗全部(其男女聲部無系統化切分方式,無意義之標註
|
||||
容忍之);genderfluid 歌三套全查(無自身透鏡,發話
|
||||
位置無法先驗決定),其歸屬引用維持個案地位。每筆
|
||||
輸入為一首歌之整合報告(「僅單獨提及」行於組裝時
|
||||
剝除,標註僅依主清單)與該首適用之樣態表;定義檔
|
||||
獨立執行三次,(歌,樣態)對
|
||||
得兩票以上者定案。
|
||||
- **模型**:`claude-fable-5`(與先導深讀同儀器;
|
||||
`temperature` 與 `thinking` 參數不適用,均不送出)。
|
||||
- **輸入組裝**:確定性行內腳本。5-1:145 首依歌曲 ID
|
||||
升序,`content` 為歌詞逐字全文;5-2:每筆
|
||||
`{"reports": [run1 輸出, run2 輸出, run3 輸出]}`;
|
||||
5-3:單筆以 `song-<ID>` 為鍵、整合報告為值之 JSON
|
||||
物件,鍵集合為該次統整之範圍(基底為全 145 首,
|
||||
分組依工作庫 `performer_gender` 切分);5-4:每筆
|
||||
`{"report": 主清單, "patterns": [{"id", "name",
|
||||
"description"}]}`,樣態條目自分組統整草稿機械切出,
|
||||
代表引句不隨附——引句出自特定歌曲,判該曲時形同
|
||||
預答。各輸入檔之 SHA-256 記入該步 meta。
|
||||
- **模型**:`claude-fable-5`。
|
||||
- **輸入組裝**:各步輸入檔由確定性程序自上游產物組裝。
|
||||
5d 之樣態條目自分組統整草稿機械切出,代表引句不
|
||||
隨附——引句出自特定歌曲,判該曲時形同預答。
|
||||
|
||||
## 女性力量候選集
|
||||
|
||||
候選集為兩類歌曲的合集:定案編碼含 `women-power` 者,
|
||||
以及定案編碼含研究者指認之女性力量概念域分群組者。
|
||||
指認於詞彙表定案後、黃金標準編碼開始前完成,指認清單
|
||||
與理由記入決策日誌。
|
||||
候選集為定案編碼含 `women-power` 或 `female-empowerment`
|
||||
(步驟 4 女性力量群的兩個編碼)之歌曲聯集:wp 66 首、
|
||||
fe 144 首,聯集 145 首。
|
||||
|
||||
## 軌跡對映(診斷用)
|
||||
|
||||
沿收斂軌跡的機械對映:原始關鍵字 →(兩份標註執行歸檔的
|
||||
`output.jsonl`)歌曲、原始關鍵字 →(分群)組,純程式查表,
|
||||
原始輸出)歌曲、原始關鍵字 →(分群)組,純機械查表,
|
||||
決定性。以其結果與步驟 3 直接編碼的差異率作為「收斂軌跡
|
||||
扭曲」的診斷量,不作主結果。
|
||||
|
||||
@@ -230,78 +171,36 @@
|
||||
每一步的輸出如何變成下一步的輸入,皆為確定性程序,規則
|
||||
明定如下:
|
||||
|
||||
- **歌詞輸入檔(步驟 1)**:`export-llm-input` 自工作
|
||||
儲存產出,每筆 `{"id": "song-<ID>", "content": <歌詞>}`,
|
||||
依歌曲 ID 升序。步驟 3 的輸入由同一子命令、同一工作
|
||||
儲存產出(見下),兩步的語料同一性由此成立;各步
|
||||
輸入檔的 SHA-256 記入該步 meta。
|
||||
- **步驟 1 → 2**:`cluster-keywords` 讀兩份執行歸檔的
|
||||
`output.jsonl`(一律以換行字元 `\n` 切行——歌詞含
|
||||
U+0085 等控制字元時,`str.splitlines()` 類的通用切行
|
||||
會截斷 JSON 字串,實測踩中),進池後直接分群,一次
|
||||
產出上列五份檔案。
|
||||
- **步驟 2 → 3 輸入檔**:`export-llm-input --extras
|
||||
<定案碼表>` 自工作儲存產出步驟 3 的輸入,每筆
|
||||
`{"id": "song-<ID>", "content": <字串>}`,`content` 為
|
||||
固定鍵序序列化的 `{"lyrics": …, "keywords": [...]}`,
|
||||
依歌曲 ID 升序。碼表以參數傳入而非填進定義檔——定義
|
||||
- **歌詞輸入檔(步驟 1)**:由確定性的匯出程序自工作
|
||||
儲存產出,一筆一首歌,依歌曲 ID 升序。步驟 3 的輸入
|
||||
由同一匯出程序、同一工作儲存產出(見下),兩步的語料
|
||||
同一性由此成立。
|
||||
- **步驟 1 → 2**:確定性的分群程序讀兩份執行歸檔的
|
||||
執行紀錄,進池後直接分群,產出詞彙表與交給
|
||||
模型的碼表。
|
||||
- **步驟 2 → 3 輸入檔**:同一匯出程序自工作儲存產出
|
||||
步驟 3 的輸入,一筆一首歌,兼含歌詞與定案碼表,依
|
||||
歌曲 ID 升序。碼表以參數傳入而非填進定義檔——定義
|
||||
檔只規定任務形狀,換詞彙表、換演算法都不必改它。
|
||||
- **步驟 3 定案**:`tally-codings <執行歸檔 1> <執行歸檔
|
||||
2> <執行歸檔 3> <輸出 CSV> --corrections <更正表>
|
||||
--valid-keywords <合法碼清單>` 讀三份執行歸檔的
|
||||
`output.jsonl`,依序套用更正表、驗證所有標籤皆在合法碼
|
||||
- **步驟 3 定案**:確定性的計票程序讀三份執行歸檔的
|
||||
執行紀錄,依序套用更正表、驗證所有標籤皆在合法碼
|
||||
清單之內、計票。
|
||||
- **更正表**:`data/manual/coding-corrections.csv`,研究者
|
||||
逐列校定的人工著作,欄位 `Song ID`、`Run`、`Type`、
|
||||
`To Be Replaced`、`Correct Term`。`Type` 為 `keyword`
|
||||
或 `evidence`,分別更正標籤與引述;`Correct Term` 為
|
||||
替代字串,或 `**REMOVE**` 表示刪去該筆標籤指派(`keyword`)
|
||||
或該句引述(`evidence`)。一筆 `evidence` 更正套用於該
|
||||
首歌該次執行的所有出現處。兩個文字欄以歌詞慣例「 / 」
|
||||
表示換行(與載入後的執行紀錄同一表示法,逐字比對、不再
|
||||
轉換),故一列一行,純文字工具可逐列處理。表中任一列若
|
||||
在資料中找不到對應者,即中止;校定的判準記於
|
||||
`decision-log.md`。
|
||||
- **合法碼清單**:純文字、一行一個碼,自詞彙表產出:
|
||||
`{ cat runs/02-cluster/result-keywords.txt; echo
|
||||
women-power; } | sort`。
|
||||
- **定案表**:`results/codings.csv`,欄位 `Song`、
|
||||
`Artist Credit`、`Keyword`、`Quote`,一列一個標籤。歌名
|
||||
與演出者名銜逐首查工作儲存取得,故本子命令須在
|
||||
`build-db` 之後執行。`Quote` 為該標籤在計票中各份執行
|
||||
所引的歌詞行:各份的引述串接後逐字去重,按 Unicode
|
||||
碼位排序,以單一 `|` 相接(三份執行彼此無先後主從之
|
||||
別,引述之序取決於引述本身);引述內的換行於執行紀錄
|
||||
載入時一次換成歌詞慣例「 / 」,此後更正表、定案表與
|
||||
工作儲存全鏈路同一表示法,不再還原。「 / 」的無歧義性
|
||||
是語料事實而非結構保證:全 883 首歌詞經窮舉查核不含
|
||||
「 / 」;換語料須重查。列序依印出的前三欄依序排:
|
||||
歌名、演出者名銜、
|
||||
標籤,一律以 Unicode 碼位比較,換行為 CRLF(同專案
|
||||
其他 CSV)。
|
||||
- **序列化通則**:所有中間檔為 UTF-8,欄序、鍵序與元素
|
||||
序皆依上列規則明定,無時間戳、無隨機成分;JSON 解析
|
||||
一律偵測重複鍵,違規即失敗。人讀為主的產物採純文字或
|
||||
CSV(CSV 依 RFC 4180,標題列字首大寫),機器交接檔採
|
||||
JSON。給定相同的 LLM 執行輸出,全部交接產物逐位元組
|
||||
可再生。
|
||||
- **更正表**:研究者逐列校定的人工著作,逐筆更正標籤
|
||||
或引述——以替代字串取代,或刪去該筆標籤指派或該句
|
||||
引述。一筆引述更正套用於該首歌該次執行的所有出現處。
|
||||
表中任一列若在資料中找不到對應者,即中止;校定的
|
||||
判準記於 `decision-log.md`。
|
||||
- **合法碼清單**:自詞彙表產出:分群組名加上
|
||||
`women-power`。
|
||||
- **定案編碼表**:歌名與演出者名銜逐首查工作儲存取得。
|
||||
每個定案標籤隨附其在計票中各份執行所引的
|
||||
歌詞行,供逐碼查核(三份執行彼此無先後主從之別)。
|
||||
|
||||
## 執行與稽核
|
||||
|
||||
- LLM 步驟以 `run-llm <定義檔> <輸入檔> <歸檔目錄>`
|
||||
執行;一步的 N 次執行=重現命令清單上的 N 行命令,
|
||||
各自歸檔(`runs/<步驟>/run1`、`run2`,三票制步驟另有
|
||||
`run3`)。
|
||||
- 確定性步驟(進池、分群、計票、對映)為子命令,其
|
||||
輸入輸出檔同隨 `runs/` 歸檔;因無執行變異,歸檔目錄
|
||||
下不分 `run<N>` 層。
|
||||
- LLM 步驟以批次執行程序執行,一份定義檔配一份
|
||||
輸入檔;一步的 N 次執行為 N 次各自獨立的呼叫,各自
|
||||
歸檔自我完備。
|
||||
- Batch API 的每筆請求自含全部脈絡且互不可見(平台
|
||||
契約),歌與歌之間的獨立性由此成立;各次執行的獨立
|
||||
性由「一次呼叫、一個批次、一份歸檔」的執行結構自明。
|
||||
- 每次 `run-llm` 執行的 token 用量與費用記入
|
||||
`run-costs.md`,被取代的執行一併保留供總支出核算。
|
||||
|
||||
## 映射分析方法
|
||||
|
||||
(依 2026-07-30 決策,於看到結果前寫定;待黃金標準
|
||||
編碼展開前補入。)
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# LLM 輸出的契約查核
|
||||
|
||||
(2026-08-06 量測。對象為步驟 3 的三份執行歸檔
|
||||
`runs/03-code/run1`–`run3`,共 883 首歌、44,149 筆標籤
|
||||
`data/runs/3a-code/run1`–`run3`,共 883 首歌、44,149 筆標籤
|
||||
指派、44,146 句引述。歌詞以模型實際看到的那一份為準,即
|
||||
`tools/instance/llm-input-code.jsonl`。)
|
||||
|
||||
定義檔 `prompts/03-code.md` 對輸出下了四條明確要求:只用
|
||||
定義檔 `prompts/3a-code.md` 對輸出下了四條明確要求:只用
|
||||
給定的碼且拼寫照給、每個標出的碼恰引一行歌詞、引述逐字、
|
||||
輸出為合法 JSON 且無其他文字。本檔記錄這四條各自被遵守到
|
||||
什麼程度。
|
||||
@@ -46,9 +46,8 @@ song-750 run3: -use=[] -abuse=["I got a thing for the hard
|
||||
liquor on ice"]
|
||||
```
|
||||
|
||||
模型寫錯後綴、已輸出的 token 收不回,遂以空陣列收束該鍵,
|
||||
再於正確的鍵補上引述。song-750 的 run1 與 run2 則整筆使用
|
||||
錯拼的鍵並附上引述,故該首的三票分裂於兩種拼寫之間。
|
||||
song-750 的 run1 與 run2 則整筆使用錯拼的鍵並附上引述,
|
||||
故該首的三票分裂於兩種拼寫之間。
|
||||
|
||||
## 引述的存在
|
||||
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
# 先導研究的來歷與地位
|
||||
|
||||
(2026-08-17 認清並記錄。本檔記述正式研究之前的先導研究:
|
||||
它做了什麼、研究者檢視到哪一層、哪些成果被沿用、哪些被
|
||||
棄用,以及它如何促成正式研究的設計。)
|
||||
|
||||
## 一、先導研究是什麼
|
||||
|
||||
正式研究之前,研究者曾以 Claude Code 對 Billboard Year-End
|
||||
Hot 100(2018–2025、684 首)的歌詞做過一輪探索性分析,檢視
|
||||
「女性力量」語彙的使用狀況,並附帶分析 pussy 一詞作為女性
|
||||
代稱的修辭。2026 年 4 月投出的研討會摘要即根據該輪分析撰寫。
|
||||
|
||||
## 二、它實際的執行方式
|
||||
|
||||
**研究問題與方向由研究者提出**:女性力量語彙被父權框架取用、
|
||||
pussy 作為代稱的跨性別使用,都是研究者長期關注的問題。
|
||||
|
||||
**設計與執行由 Claude Code 進行**:分析管線、分類架構(genuine
|
||||
/peripheral/fake 三分類)、五種「假女性力量」類型、以及
|
||||
frame-aware 提示修正實驗,均由當時協作的 Claude Code 設計並
|
||||
執行。
|
||||
|
||||
**研究者檢視的是問題與答案,不是中間過程**:研究者確認研究
|
||||
問題的方向正確、答案值得追究,即據以投稿;分類類目如何產生、
|
||||
提示詞如何下、數字如何算出,並未逐步檢視。投稿摘要中的具體
|
||||
數字(如「44% 屬於假女性力量」、62 首的三分類、53 首的 pussy
|
||||
修辭分類、假率由 44% 降至 5% 以下)研究者未曾稽核,其產生
|
||||
過程亦無歸檔可回溯。
|
||||
|
||||
**當時的判斷**:投稿未上即罷;若上,正式研究由研究者自行
|
||||
設計與執行。事實上正式研究即是如此重做的。
|
||||
|
||||
## 三、被沿用的成果
|
||||
|
||||
- **歌詞捕捉檔**:先導研究蒐集的 lyrics.json(684 首,
|
||||
2018–2025)以私人腳本(不入版本庫)匯入歌詞快取,只取
|
||||
識別欄位與歌詞本文,先導的分析欄位一概不匯入;出處記於
|
||||
`data/captures/lyrics-provenance.csv`
|
||||
(見 `decision-log.md` 2026-07-31、08-02 條)。
|
||||
- **假說方向**:女性力量語彙的挪用與污染,成為正式研究的
|
||||
研究問題。
|
||||
- **粒度選擇**:正式研究鎖定 thematic keywords 這一粒度,
|
||||
係繼承先導研究三種粒度的比較結果(keywords 過碎、themes
|
||||
過早抽象)。
|
||||
- **`women-power` 一詞的來歷**:考據先導研究的 local agent
|
||||
存檔可知,其第一步指令含數十個範例 thematic keywords,
|
||||
其中即有 women-power——為當時協作的 Claude Code 依研究者
|
||||
長期表達的關注主動加入(研究者端播種,非明示指定);先導
|
||||
的標籤 `women-power-and-empowerment` 則是第三步強制合併
|
||||
兩個關鍵字的管線人工產物(見 `decision-log.md` 2026-08-04
|
||||
條)。
|
||||
- **簿記容量的教訓**:先導研究九百餘詞可以在單一回應內完成
|
||||
分組;此法未沿用,詞彙表改由確定性程序產生(見
|
||||
`decision-log.md` 2026-08-06 條)。
|
||||
|
||||
## 四、被棄用的成果
|
||||
|
||||
以下先導研究的產物未進入正式研究,論文亦未引用:
|
||||
|
||||
- **genuine/peripheral/fake 三分類與「44% 假女性力量」**:
|
||||
構念與分母均未經稽核回溯,未沿用。
|
||||
- **五種「假女性力量」類型(A–E)**:未沿用。
|
||||
- **pussy 一詞的修辭分類**:未沿用。
|
||||
- **frame-aware(框架感知)提示修正法**:先導研究以框架判準
|
||||
寫進提示以提高準確率;未沿用。
|
||||
- **三個獨立 LLM subagent+人工仲裁的流程**:未沿用,原因見
|
||||
`decision-log.md` 2026-07-30 條。
|
||||
|
||||
## 五、它促成了可稽核的設計
|
||||
|
||||
先導研究的根本限制不在結論對錯,而在**不可稽核**:沒有定義檔
|
||||
快照、沒有原始輸出歸檔、沒有參數紀錄,因此任何一個數字都無法
|
||||
回溯到產生它的那一次執行。正式研究的可稽核設計,正是針對這一點
|
||||
而立。
|
||||
|
||||
換言之,正式研究對 LLM 輸出所採取的「不信任、須查核」立場,
|
||||
其第一個案例就是先導研究本身。
|
||||
|
||||
## 六、現存的紀錄
|
||||
|
||||
- 投稿時的先導研究摘要:已於 git 歷史中(`paper/abstract.md`
|
||||
的前身),樹中不另存副本——摘要只是產物,不足以記述過程,
|
||||
故另立本檔。
|
||||
- 先導研究的歌詞捕捉檔沿用事實:`data/captures/lyrics-provenance.csv`。
|
||||
@@ -1,114 +0,0 @@
|
||||
# 專案目錄結構
|
||||
|
||||
(2026-07-30 討論定案;2026-07-31 更新為 tools/ 子專案與
|
||||
SQLite 工作儲存架構)
|
||||
|
||||
```
|
||||
pop-fem-audit/
|
||||
├── README.md # 專案說明、重現步驟
|
||||
├── CLAUDE.md # 極簡工作規範(subagent 會讀到,
|
||||
│ # 絕不放理論、codebook、預期結果)
|
||||
├── .gitignore # captures/lyrics/、.env、scratch
|
||||
├── data/ # 依生命週期分層(文字格式)
|
||||
│ ├── source/ # 源頭:手放後不動
|
||||
│ │ └── yearend_hot100_2016_2025.csv # 原始榜單
|
||||
│ ├── captures/ # 外部捕捉:只由 fetch 命令與
|
||||
│ │ │ # 私人匯入腳本寫入
|
||||
│ │ ├── artists-wikidata.csv # Wikidata 快照
|
||||
│ │ ├── lyrics-provenance.csv # 歌詞出處
|
||||
│ │ └── lyrics/ # 歌詞 .txt 快取
|
||||
│ │ # (gitignored,版權)
|
||||
│ ├── manual/ # 人工著作:只由研究者手寫
|
||||
│ │ # (黃金標準編碼等)
|
||||
│ └── derived/ # 衍生:只由 build-db 寫入
|
||||
│ ├── songs.csv # 歌曲報表(人讀;進 git)
|
||||
│ └── artists.csv # 歌手報表(人讀;進 git)
|
||||
├── prompts/ # LLM 定義檔(逐字作為 system prompt)
|
||||
│ └── <步>-<次步>-<task>.md # 01-tag.md、03-code.md
|
||||
│ # (步內僅一個執行時省略次步)
|
||||
│ # 不帶版本號,版本即 git 歷史
|
||||
│ # (編號的所指是工序:確定性
|
||||
│ # 的步驟 2 無定義檔仍佔一號;
|
||||
│ # 補零只為排序)
|
||||
├── tools/ # 輔助工具子專案(src-layout)
|
||||
│ ├── pyproject.toml # 發行名 pop-fem-audit-tools;
|
||||
│ │ # pip install -e tools/ 安裝
|
||||
│ ├── README.rst LICENSE MANIFEST.in .env.example .gitignore
|
||||
│ ├── docs/ # Sphinx API 文件
|
||||
│ ├── instance/ # SQLite 工作儲存(generated、
|
||||
│ │ # gitignored;含歌詞全文)
|
||||
│ ├── src/pop_fem_audit_tools/
|
||||
│ │ ├── __main__.py # 套件 CLI 進入點(分派子命令)
|
||||
│ │ ├── commands/ # CLI 子命令模組(登記於 __init__)
|
||||
│ │ │ ├── build_db.py # build the SQLite working store
|
||||
│ │ │ │ # from the inputs
|
||||
│ │ │ ├── export_llm_input.py # export the LLM input JSONL
|
||||
│ │ │ │ # (lyrics only) from the
|
||||
│ │ │ │ # working store
|
||||
│ │ │ ├── fetch_artists.py # fetch artist metadata from
|
||||
│ │ │ │ # Wikidata into the snapshot CSV
|
||||
│ │ │ ├── fetch_lyrics.py # fetch missing lyrics from the
|
||||
│ │ │ │ # public APIs into the lyrics dir
|
||||
│ │ │ ├── cluster_keywords.py # pool the tagging runs'
|
||||
│ │ │ │ # keywords and cluster them
|
||||
│ │ │ │ # into the codes (step 2)
|
||||
│ │ │ └── run_llm.py # API 執行器:一份定義檔+一份輸入
|
||||
│ │ │ # →歸檔至指定目錄(Batch API);
|
||||
│ │ │ # 多次執行的計票由獨立子命令承擔
|
||||
│ │ ├── config.py # pydantic-settings 設定(.env)
|
||||
│ │ ├── database.py # SQLAlchemy engine / session / Base
|
||||
│ │ ├── models.py # SQLAlchemy ORM 資料模型
|
||||
│ │ └── utils.py # 共用工具(format_duration)
|
||||
│ └── tests/ # 單元測試(unittest)
|
||||
├── runs/ # 現行執行的完整稽核紀錄(進 git;
|
||||
│ │ # 重跑同一 run 須明示 --replace)
|
||||
│ ├── <步驟名>/ # 一步一個目錄(如 03-code)
|
||||
│ │ └── run<N>/ # LLM 步驟:每個 run 一份自我
|
||||
│ │ ├── prompt.md # 完備歸檔(定義檔快照)
|
||||
│ │ ├── output.jsonl # 該次執行原始輸出
|
||||
│ │ └── meta.json # model ID、temperature、時間戳、
|
||||
│ │ # batch ID、token 用量
|
||||
│ └── 02-cluster/ # 確定性步驟:無執行變異,
|
||||
│ # 不分 run<N> 層
|
||||
├── results/ # 論文引用的報表 CSV(export 產出;
|
||||
│ # 「可再生仍 commit」的唯一例外)
|
||||
├── docs/
|
||||
│ ├── research-plan.md # 研究步驟規劃(本檔之姊妹篇)
|
||||
│ ├── project-structure.md # 本檔
|
||||
│ ├── codebook.md # 人工編碼手冊(版本由 git 管理)
|
||||
│ ├── decision-log.md # 決策日誌:每次改定義檔的原因
|
||||
│ ├── run-costs.md # 每次執行的 token 用量與費用
|
||||
│ └── methodology.md # 方法細節(全文方法節底稿;
|
||||
│ # 映射分析方法須在看結果前寫定)
|
||||
└── paper/
|
||||
├── abstract.md # 摘要
|
||||
└── 流行音樂中「女性力量」….odt # 全文
|
||||
```
|
||||
|
||||
## 設計理由
|
||||
|
||||
- **`runs/` 自我完備**:每個執行目錄含定義檔快照 + 原始輸出 +
|
||||
meta,讀者不需 git 考古即可稽核任一筆結果。
|
||||
- **`runs/`(原始稽核資料)與 `results/`(最終表)分離**:
|
||||
論文只引 `results/`,其來源可回溯至 `runs/`。
|
||||
- **`prompts/` 檔名不帶版本號**:版本即 git 歷史,失敗的
|
||||
版本不保留;論文引用的單位是 `runs/` 內隨執行保存的定義檔
|
||||
快照(每個執行目錄自我完備),不需檔名可指的版本名。
|
||||
- **工作儲存的資料表**:`songs`、`chart_entries`、`artists`、
|
||||
`song_artists`、`codings`(定案編碼:一歌一標籤一列,`quotes`
|
||||
存該標籤所據的歌詞引述,多句以 `|` 相接)。定案表
|
||||
`results/codings.csv` 經 `build-db --codings` 匯入,與其餘資料
|
||||
同一交易,儲存不會半建;詳見 `research-plan.md`「資料儲存與
|
||||
模型」。
|
||||
- **Commit 判準**:能由「committed 輸入+程式」決定性再生者不
|
||||
commit(SQLite 工作儲存、LLM 輸入檔);源頭、捕捉、人工著作
|
||||
一律以文字 commit。「可再生仍 commit」的例外有二:
|
||||
`results/` 報表(引用穩定性、審稿人零門檻、撰稿期可 diff)
|
||||
與 `data/derived/` 人讀報表(與工作儲存同一動作產出,稽核
|
||||
鏈無中間空缺)。詳見 `research-plan.md`「資料儲存與模型」。
|
||||
- **設定**經 pydantic-settings 統一:`.env`(gitignored,範本
|
||||
`tools/.env.example`)供應 `SQLALCHEMY_DATABASE_URL` 與
|
||||
`ANTHROPIC_API_KEY`,絕不寫入 repo。
|
||||
- **CLAUDE.md 極簡**:實測證實 Claude Code subagent 會繼承專案
|
||||
CLAUDE.md 全文,故其中只放工作流程規則,領域知識一律放
|
||||
`docs/`(subagent 不會自動讀到)。
|
||||
@@ -0,0 +1,150 @@
|
||||
# 信度:量測方式與結果
|
||||
|
||||
(2026-08-16 量測。對象為步驟 3a 與步驟 5d 各三次執行的
|
||||
原始輸出;數字由原始輸出直接計算。)
|
||||
|
||||
## 一、本研究的信度是什麼
|
||||
|
||||
**信度(reliability)** 問的是:同一個量測重複做,會不會得到
|
||||
同樣的結果。它管的是**一致性**,不管對錯。一把每次都少兩公斤
|
||||
的秤,信度很好、效度很差。**效度(validity)** 問的則是:量到
|
||||
的是不是想量的東西。
|
||||
|
||||
Krippendorff 依產生資料的設計,把信度分成三型
|
||||
(Krippendorff, 2004;Hayes & Krippendorff, 2007):
|
||||
|
||||
- **穩定性(stability)**:同一位觀察者、同樣的材料,重複
|
||||
量測是否一致(test–retest,即 intra-observer)。
|
||||
- **可複製性(reproducibility)**:不同觀察者各自獨立量測
|
||||
是否一致,即一般所稱的**編碼者間信度**
|
||||
(inter-coder reliability)。Krippendorff 認為這是內容
|
||||
分析中最強、也最可行的一型,因為它排除了「這些類目只是
|
||||
某一個人特異的讀法」的疑慮。
|
||||
- **準確性(accuracy)**:與已知標準比對。
|
||||
|
||||
**本研究三次獨立執行量到的是第一型。**同一個模型、同一份
|
||||
定義檔、同一批輸入,重複三次,量到的是這個編碼者自己前後
|
||||
一不一致。第二型在本設計中**結構上不存在**:編碼者只有一位,
|
||||
不同的模型版本(Sonnet 4.6、Fable 5)也不是「不同的編碼者」,
|
||||
而是不同的儀器。第三型**未量測**:沒有黃金標準可比。
|
||||
|
||||
以工程的語言說更精確:這裏量的是**儀器的重複性
|
||||
(repeatability)**,亦即**隨機性有多大**;計量學上稱之為
|
||||
**精密度(precision)**,而**準確度(accuracy)** 未經量測。
|
||||
若儀器是確定性的,重複性無須量測;但 LLM 不是——同樣的提示、
|
||||
同樣的輸入,三次會給出不同結果,因此重複性是必須報告的儀器
|
||||
規格。
|
||||
|
||||
## 二、三個常用指標
|
||||
|
||||
以「這首歌有沒有這個碼」這種是非題為例:
|
||||
|
||||
- **百分比一致率**:兩次判斷相同的格子佔全部格子的比例。
|
||||
最直觀,但會被「碰巧同意」灌水——當九成五的格子都是
|
||||
「無」,兩次隨便判也會有九成以上一致。
|
||||
- **Cohen's κ(kappa)**:把「碰巧同意」的部分扣掉之後的
|
||||
一致度,兩位編碼者用。0 表示與隨機無異,1 表示完全一致。
|
||||
慣例上 0.61–0.80 稱為 substantial、0.81 以上稱為
|
||||
almost perfect(Landis & Koch 1977 的分級)。
|
||||
- **Krippendorff's α**:可處理兩位以上編碼者、缺漏值與
|
||||
不同尺度,是內容分析文獻最常被要求的指標。慣例門檻為
|
||||
α ≥ 0.800 可作結論、0.667–0.800 只能作暫定結論
|
||||
(Krippendorff 的建議)。
|
||||
|
||||
三者裏,**百分比一致率會高估、κ 與 α 才是可比較的數字**。
|
||||
|
||||
補充一個常被誤解的方向:κ 的著名「悖論」
|
||||
(Feinstein & Cicchetti, 1990)是**類別分佈極不平衡時,
|
||||
高一致率反而算出很低的 κ**——偏斜會壓低 κ,不會抬高它。
|
||||
本研究的正例只佔 16.5%(89,183 格中約 14,700 格有碼),
|
||||
在這種偏斜下 κ 仍達 0.93,屬於保守估計,不是被大量「雙方
|
||||
都判無」灌出來的。
|
||||
|
||||
## 三、實測結果
|
||||
|
||||
### 步驟 3a(編碼:883 首 × 101 個碼 = 89,183 格,執行三次)
|
||||
|
||||
| 兩次執行 | 百分比一致率 | Cohen's κ | Jaccard |
|
||||
|---|---:|---:|---:|
|
||||
| run1 × run2 | 98.10% | 0.931 | 0.891 |
|
||||
| run1 × run3 | 98.09% | 0.931 | 0.891 |
|
||||
| run2 × run3 | 98.26% | 0.937 | 0.900 |
|
||||
|
||||
**Krippendorff's α = 0.933**(三次執行合計)。
|
||||
|
||||
三票計票的分佈:三票全同 13,500 格、兩票 1,163 格、
|
||||
僅一票 1,310 格(未達門檻而剔除)。**定案的 14,663 個編碼
|
||||
中,92.1% 是三次全數同意的**。
|
||||
|
||||
(此處由原始輸出計算,未套用 936 筆人工校對表;論文引用的
|
||||
定案數 14,664 為校對後的結果,差 1 筆。)
|
||||
|
||||
### 步驟 5d(樣態標註:111 首 × 各自適用樣態 = 3,708 格,執行三次)
|
||||
|
||||
| 兩次執行 | 百分比一致率 | Cohen's κ | Jaccard |
|
||||
|---|---:|---:|---:|
|
||||
| run1 × run2 | 97.87% | 0.933 | 0.898 |
|
||||
| run1 × run3 | 98.17% | 0.942 | 0.911 |
|
||||
| run2 × run3 | 97.87% | 0.933 | 0.898 |
|
||||
|
||||
**Krippendorff's α = 0.936**。三票全同 675 格、兩票 62 格、
|
||||
僅一票 51 格;**定案的 737 筆歸屬中,91.6% 三次全同**。
|
||||
|
||||
### 怎麼讀這些數字
|
||||
|
||||
κ 與 α 都在 0.93 上下,以慣例分級屬於 almost perfect,
|
||||
也高於 α ≥ 0.800 的門檻。用白話說:**這個編碼者重複三次,
|
||||
判斷幾乎不變;會左右搖擺的,是那一成左右的邊緣案例,而
|
||||
三票多數決正是為它們設計的**。
|
||||
|
||||
Jaccard(只看「有標到」的格子,忽略雙方都沒標的)約 0.89–0.91,
|
||||
比百分比一致率低而更誠實——因為 82% 的格子是雙方都判「無」,
|
||||
那些一致並不費力。
|
||||
|
||||
## 四、隨機性的規模
|
||||
|
||||
信度數字回答的實際問題是:**這台儀器的隨機性,大到會不會
|
||||
改變結論?**
|
||||
|
||||
步驟 3a 的 89,183 格中,搖擺的格子共 2,473 格(兩票 1,163、
|
||||
一票 1,310),佔 2.8%。若只執行一次即定案,這批格子當中約
|
||||
一半會成為誤收、另一半會成為漏收。三票多數決把兩票以上者
|
||||
收入、一票者剔除,處理的正是這一批。
|
||||
|
||||
以此規模判斷,結論層的三條帶狀結構(女性力量與陽剛群共現、
|
||||
與脆弱群互斥、與厭女群獨立)不可能由這個量級的雜訊翻轉。
|
||||
反過來說,若一致率只有 0.6,同一組結論就不能採信——信度
|
||||
數字的用途在此,而不在滿足慣例。
|
||||
|
||||
## 五、這些數字證明了什麼、不能證明什麼
|
||||
|
||||
**能證明**:量測是穩定的,結果不是單次抽樣的偶然;
|
||||
三票多數決確實只在少數邊緣格子上發揮作用(約 3%)。
|
||||
|
||||
**不能證明**:判斷是正確的。三次共用同一個模型與同一份
|
||||
先驗,能濾掉隨機噪音,濾不掉系統性偏誤。本研究已發現三個
|
||||
三次一致的錯誤可作實例:Sonnet 4.6 穩定地將 Women Power
|
||||
讀為 Women+Power;深讀階段將〈Cowgirls〉中握韁繩者判反、
|
||||
將〈Happen To Me〉分屬兩段的引文接成同一動機。一把每次都
|
||||
量出同樣偏差的尺,重複性滿分,準確度為零。
|
||||
|
||||
**結構上沒有**:編碼者間信度。要有它,需要第二個獨立的
|
||||
觀察者——另一個模型,或研究者人工編一小批對照。本研究
|
||||
未做,列為限制。
|
||||
|
||||
## 六、計算方式與依據
|
||||
|
||||
- **分析單位**:每一個「(歌曲, 編碼)」格為一個單位,是非題
|
||||
(有標/無標)。步驟 3a 為 883 首 × 101 碼 = 89,183 格;
|
||||
步驟 5d 為各歌適用樣態數合計 3,708 格。
|
||||
- **Cohen's κ**:兩次執行的 2×2 表,κ=(p_o−p_e)/(1−p_e),
|
||||
p_e 由兩次各自的邊際比例相乘求得。
|
||||
- **Krippendorff's α(名目、三位「編碼者」、無缺漏)**:
|
||||
α=1−D_o/D_e;某單位若有 v 次標為「有」,其不一致配對數為
|
||||
v(3−v),D_o 為其總和除以總配對數,D_e 由整體「有/無」的
|
||||
邊際比例求得。
|
||||
- **門檻依據**:κ 的分級為 Landis & Koch (1977)
|
||||
(0.61–0.80 substantial、0.81–1.00 almost perfect);
|
||||
α 的門檻為 Krippendorff 的建議(≥0.800 可作結論、
|
||||
0.667–0.800 僅能作暫定結論)。
|
||||
- 數字由原始執行輸出直接計算,未套用 936 筆人工校對表。
|
||||
+23
-83
@@ -11,81 +11,33 @@
|
||||
**不與先導研究做比較**;
|
||||
全文數字一律以正式研究結果為準。先導研究僅作為假說的
|
||||
內部來源,記於決策日誌,不進入論文敘事。
|
||||
- **執行原則**:主會話只做討論;所有分析由 deterministic
|
||||
script 執行。LLM 步驟以 Python script 呼叫 Anthropic
|
||||
Messages API(個人 Console 帳號、Batch API 五折),定義
|
||||
檔逐字作為 system prompt。可逐項機械比對的判斷(步驟 3
|
||||
編碼、步驟 4 選群、步驟 5-4 樣態標註)採「同一定義檔
|
||||
獨立執行三次+多數決」,計票由確定性子命令完成。自由
|
||||
生成(步驟 1 自由標註)兩次執行全數進池。步驟 5-1 至
|
||||
5-3 為質性閱讀協定:三次獨立閱讀為分析者三角檢核、
|
||||
逐首整合、樣態統整產出草稿,不適用投票與仲裁。詞彙表
|
||||
不經 LLM,由詞向量嵌入+確定性分群產生。驗證結果不符
|
||||
預期則修訂定義檔重跑該循環,絕不手改結果。
|
||||
- **執行原則**:主會話只做討論,分析一律由確定性程序
|
||||
執行。可逐項機械比對的判斷(步驟 3 編碼、步驟 4 選群、
|
||||
步驟 5d 樣態標註)採「三次獨立執行+多數決」定案,
|
||||
自由生成(步驟 1 自由標註)採兩次執行全數進池,步驟
|
||||
5a 至 5c 定為質性閱讀協定;程序細節見 `methodology.md`,
|
||||
工作規範見 `conventions.md`。
|
||||
- **提示詞只定格式、不定語意**:研究對象是通用 LLM 以其
|
||||
網路語料知識背景所做的自然編碼與閱讀,其結果本身是
|
||||
批判對象。定義檔只規定任務形狀(輸入、數量範圍、輸出
|
||||
格式),不給任何主題的定義、判準或範例。LLM 判斷一律
|
||||
要求逐項引述歌詞原句,作為檢視偏差的依據。
|
||||
- **模型**:步驟 1、3 用 `claude-sonnet-4-6`(temperature=0、
|
||||
thinking 關閉);步驟 4、5 用 `claude-fable-5`(兩參數
|
||||
不適用,均不送出;取樣變異由多數決或整合吸收)——實測
|
||||
發現 sonnet 將「Women Power」拆讀為 women+power 的組合
|
||||
語意,fable-5 讀為詞彙化概念,語意層任務因此換用
|
||||
fable-5(經過見決策日誌)。
|
||||
- **模型**:步驟 1、3 用 `claude-sonnet-4-6`;步驟 4、5
|
||||
用 `claude-fable-5`——實測發現 sonnet 將「Women Power」
|
||||
拆讀為 women+power 的組合語意,fable-5 讀為詞彙化
|
||||
概念,語意層任務因此換用 fable-5(經過見決策日誌)。
|
||||
- **信度與效度**:三次獨立執行量測穩定性(intra-rater
|
||||
reliability,可報告兩兩一致率);封閉母體全量檢查取代
|
||||
抽樣防衛。可重現性定義為「程序透明+可稽核」:公開定義
|
||||
檔、記錄 model ID 與執行時間、保存全部原始輸出。
|
||||
|
||||
## 資料儲存與模型
|
||||
## 範圍性定案
|
||||
|
||||
- **Commit 判準**:凡能由「committed 的輸入+committed 的
|
||||
程式」決定性再生者,不 commit;凡不能者——源頭資料、
|
||||
外部世界的捕捉(Wikidata 快照、LLM 原始輸出)、人工
|
||||
著作——一律以文字格式 commit。格式跟著層次走。
|
||||
- **分層**:
|
||||
- 源頭:原始榜單 CSV(進 git)。
|
||||
- 捕捉:Wikidata 快照 CSV、`runs/` JSONL(皆進 git);
|
||||
歌詞 `.txt` 快取(版權因素 gitignored,為已知的稽核
|
||||
缺口)。
|
||||
- 人工:`data/manual/`(僅研究者親手寫入:編碼修正、
|
||||
演唱者性別修正)。
|
||||
- 工作儲存:SQLite 單檔(`tools/instance/`,generated、
|
||||
不進 git),SQLAlchemy 2.0 typed ORM 定義 schema,
|
||||
設定經 pydantic-settings(`.env` 供應
|
||||
`SQLALCHEMY_DATABASE_URL` 與 `ANTHROPIC_API_KEY`)。
|
||||
歌詞全文入 DB(不進 git 故無版權疑慮)。
|
||||
- 衍生:`build-db` 建置工作儲存的同一動作產出人讀報表
|
||||
(`data/derived/`,進 git),與 SQLite 同交易語意。
|
||||
- 報表:論文引用的定案表進 `results/`——
|
||||
`codings.csv`(步驟 3 定案編碼)、`groups.csv`
|
||||
(步驟 4 定案編碼群)、`patterns.csv`(步驟 5-3 定案
|
||||
樣態表)、`annotations.csv`(步驟 5-4 定案歌×樣態
|
||||
矩陣)。衍生與報表為「可再生仍 commit」的例外,理由:
|
||||
引用穩定性、審稿人零門檻、撰稿期數字變動可 diff。
|
||||
- **資料模型**:`songs`(含 lyrics、`performer_gender`——
|
||||
演唱聲音之性別,由署名藝人之 Wikidata 性別推導後套用
|
||||
`data/manual/performer-gender-corrections.csv` 手工修正)、
|
||||
`chart_entries`、`artists`、`song_artists`、`codings`、
|
||||
`groups`(語意編碼群)、`patterns`(深讀樣態)、
|
||||
`annotations`(歌×樣態定案矩陣)。領域不變量(恰 1000
|
||||
筆榜單、每歌至少一 primary 歌手等)檢查內建於
|
||||
`build-db`,違規即建置失敗。
|
||||
- **歌手背景防火牆**:歌手背景資料只進人工解讀階段,
|
||||
**絕不進 LLM 輸入**——LLM 任務的 user message 維持
|
||||
歌詞-only 或管線中間產物-only,避免光環偏誤。
|
||||
- **Pilot 歌詞沿用(私人匯入,不進發布管線)**:先導研究
|
||||
捕捉檔(lyrics.json,684 首,2018–2025)以私人腳本
|
||||
匯入歌詞快取;讀者的重現路徑純粹是
|
||||
`fetch-lyrics`;沿用之事實記於
|
||||
`data/captures/lyrics-provenance.csv`(進 git)。
|
||||
- **子命令**(`pop-fem-audit-tools <cmd>`):`build-db`
|
||||
(`--codings`、`--groups`、`--gender-corrections`、
|
||||
`--patterns`、`--annotations`)、`cluster-keywords`、
|
||||
`export-llm-input`、`fetch-artists`、`fetch-lyrics`、
|
||||
`run-llm`(`--model` 於模型登錄表中擇一)、
|
||||
`tally-codings`、`tally-groups`、`tally-annotations`。
|
||||
- **Pilot 歌詞沿用**:先導歌詞沿用之來歷與細節詳見該檔
|
||||
(`pilot-study.md`)。
|
||||
|
||||
## 分析管線(五步驟,全部完成)
|
||||
|
||||
@@ -96,26 +48,19 @@
|
||||
階層式聚合分群 k=100,組名取 medoid;再併入研究者先驗
|
||||
主題詞 `women-power`(據實揭露的儀器介入),共 101 碼。
|
||||
3. **編碼(步驟 3)**:以定稿詞彙表對全 883 首編碼,逐
|
||||
標籤附引述,×3+多數決(`tally-codings`),定案
|
||||
`results/codings.csv`。「女性力量」候選集(wp 66 首、
|
||||
fe 144 首、wp∪fe 145 首)由此浮現。
|
||||
標籤附引述,×3+多數決,產出定案編碼表。「女性力量」
|
||||
候選集(wp 66 首、fe 144 首、wp∪fe 145 首)由此浮現。
|
||||
4. **語意編碼群(步驟 4)**:LLM 依編碼字面語意將 101 碼
|
||||
選入研究者指定的四個主題群(women-power/misogyny/
|
||||
masculine/vulnerable),×3+多數決(`tally-groups`),
|
||||
定案 `results/groups.csv`;wp/fe 與各編碼、各編碼群之
|
||||
關聯統計(BH-FDR 校正)入論文。
|
||||
masculine/vulnerable),×3+多數決,產出定案分群表;
|
||||
wp/fe 與各編碼、各編碼群之關聯統計(BH-FDR 校正)入
|
||||
論文。
|
||||
5. **女性主義問題之質性深讀(步驟 5)**:對 wp∪fe 145 首
|
||||
——5-1 逐首盲讀(僅歌詞全文)×3;5-2 逐首整合(收斂
|
||||
註記、主清單限兩讀以上);5-3 樣態統整(全體基底+
|
||||
男聲/女聲/混合三個發話脈絡分組);5-4 樣態標註——
|
||||
以分組樣態表逐首標註「有問題」的 111 首,×3+多數決
|
||||
(`tally-annotations`),定案 `results/patterns.csv` 與
|
||||
`results/annotations.csv`。
|
||||
|
||||
定義檔命名 `prompts/<步>-<次步>-<task>.md`,編號的所指是
|
||||
工序而非定義檔(確定性的第 2 步無定義檔仍佔編號);檔名
|
||||
不帶版本號——版本即 git 歷史;每次執行的定義檔快照隨
|
||||
`runs/` 自我完備,token 費用逐筆記於 `docs/run-costs.md`。
|
||||
——5a 逐首盲讀(僅歌詞全文)×3;5b 逐首整合(收斂
|
||||
註記、主清單限兩讀以上);5c 樣態統整(全體基底+
|
||||
男聲/女聲/混合三個發話脈絡分組);5d 樣態標註——
|
||||
以分組樣態表逐首標註「有問題」的 111 首,×3+多數決,
|
||||
產出定案樣態表與歌×樣態定案矩陣。
|
||||
|
||||
## 時程與剩餘工作
|
||||
|
||||
@@ -124,8 +69,3 @@
|
||||
污染」結論(步驟 5 素材已備)、信度說明、fe 與新自由
|
||||
主義敘事之詮釋標註。
|
||||
|
||||
## 其他已定案事項
|
||||
|
||||
- 歌詞受版權保護:完整歌詞不進 git
|
||||
(`data/captures/lyrics/` gitignored),論文與 repo 只留
|
||||
分析所引摘錄。
|
||||
|
||||
+56
-48
@@ -10,53 +10,61 @@ $3/$15、opus-4-6 $5/$25、opus-5 與 fable-5 $10/$50)
|
||||
被 `--replace` 取代的執行以「已取代」標記,數字保留供
|
||||
總支出核算。
|
||||
|
||||
| 日期 | 步驟 | 執行 | 模型 | 批次 ID | 耗時 | input | output | 費用 (USD) | 狀態 |
|
||||
|---|---|---|---|---|---|---:|---:|---:|---|
|
||||
| 2026-08-04 | 01-01-01-tag | run1 | claude-sonnet-4-6 | msgbatch_01PTACDQMr8M6ahnshbedjtB | 6 分 27 秒 | 763,318 | 338,661 | $3.68 | 已取代(浮水印清洗與防圍欄修訂後重跑) |
|
||||
| 2026-08-05 | 01-01-01-tag | run1 | claude-sonnet-4-6 | msgbatch_01TikJNd2pZVxzQ8SaybVthu | 4 分 57 秒 | 774,604 | 323,651 | $3.59 | 已取代(合法 JSON 修訂後重跑) |
|
||||
| 2026-08-05 | 01-01-01-tag | run1 | claude-sonnet-4-6 | msgbatch_01JFBCNqnu1cwXyqmEKLHQYF | 4 分 30 秒 | 790,370 | 328,227 | $3.65 | 已取代(song-288 平台失敗,整批重跑驗證) |
|
||||
| 2026-08-05 | 01-01-01-tag | run1 | claude-sonnet-4-6 | msgbatch_01VSDneWuSbf8mShA32jiWrX | 6 分 15 秒 | 790,370 | 326,193 | $3.63 | 已取代(措辭修訂後全體重跑) |
|
||||
| 2026-08-05 | 01-01-01-tag | run1-rescue-288 | claude-sonnet-4-6 | msgbatch_019Hq6bNVXVmp4DjRZ2cgVda | 2 分 5 秒 | 1,015 | 376 | $0.01 | 已取代(措辭修訂後全體重跑,該首原生通過)|
|
||||
| 2026-08-05 | 01-01-01-tag | run1 | claude-sonnet-4-6 | msgbatch_01VgZ77KAPGWmuu3PnQFqZ7Q | 7 分 1 秒 | 794,913 | 326,435 | $3.64 | 現行 |
|
||||
| 2026-08-05 | 01-01-01-tag | run2 | claude-sonnet-4-6 | msgbatch_01TLFey3L4fimKxcebTZQGYn | 4 分 21 秒 | 794,913 | 328,324 | $3.65 | 現行 |
|
||||
| 2026-08-05 | 01-02-01-merge | run1 | claude-sonnet-4-6 | msgbatch_01BvYMFmH8zrUq9SNxWSNba7 | 9 分 14 秒 | 46,454 | 60,974 | $0.53 | 已取代(完整分割驗證不過:漏 366、重複分派 511、撞名 4) |
|
||||
| 2026-08-05 | 01-02-01-merge | run1 | claude-opus-5 | msgbatch_018TkYpdT3DsZqq2q94FQCUN | — | 0 | 0 | $0.00 | 拒收(temperature 已棄用) |
|
||||
| 2026-08-05 | 01-02-01-merge | run1 | claude-opus-4-8 | msgbatch_018Wuux4adz5mWSvzJjfgLxb | — | 0 | 0 | $0.00 | 拒收(temperature 已棄用) |
|
||||
| 2026-08-05 | 01-02-01-merge | run1 | claude-opus-4-6 | msgbatch_01Lmt2j1Txyof9CctUK7CbHA | 12 分 19 秒 | 46,454 | 54,864 | $0.80 | 已取代(圍欄違規;漏 190、發明 151、重複 11) |
|
||||
| 2026-08-05 | 01-02-01-merge | run1 | claude-opus-5 | msgbatch_012SEfebTnjU5uWN6hhfnhhh | 8 分 45 秒 | 63,368 | 64,000 | $1.92 | 已取代(64k 截斷) |
|
||||
| 2026-08-05 | 01-02-01-merge | run1 | claude-opus-5 | msgbatch_01Wi9xEoQKCy1nZK6myVNg77 | 11 分 34 秒 | 63,368 | 70,175 | $2.07 | 已取代(驗證不過:漏 59、發明 140、重複 3) |
|
||||
| 2026-08-05 | 01-02-01-merge | run1 | claude-fable-5 | msgbatch_01M15hUs8P9FDTqWvAHdc1cA | 32 分 45 秒 | 63,368 | 107,721 | $3.01 | 現行歸檔(JSON 語法毀損,驗證不過) |
|
||||
| 2026-08-05 | 01-02-01-merge | run1 | claude-fable-5 | msgbatch_012stRP28uuQDQwobm41HMLh | — | 0 | 0 | $0.00 | 拒收(thinking.type.enabled 不支援) |
|
||||
| 2026-08-05 | 01-02-01-merge | run1 | claude-fable-5 | msgbatch_01KpS7AMx1zAHTcNujgGsouJ | 26 分 28 秒 | 63,368 | 128,000 | $3.52 | 現行歸檔(effort max:128k 全耗於推理,正文空白) |
|
||||
| 2026-08-05 | 03-01-code | run1 | claude-sonnet-4-6 | msgbatch_01L7VepTxSNz5vruvzzjuL2c | 6 分 36 秒 | 1,247,264 | 492,730 | $5.57 | 36 首輸出遭內容過濾攔阻,待修訂重跑 |
|
||||
| 2026-08-05 | 03-01-code | 引述減量實驗(36 首)| claude-sonnet-4-6 | msgbatch_01Hskdhi2DkudYmgZhhgFts7 | 3 分 51 秒 | 56,458 | 11,074 | $0.17 | 實驗:每碼一行引述,36 首全數通過過濾;歸檔不入 repo |
|
||||
| 2026-08-05 | 03-01-code | run1 | claude-sonnet-4-6 | msgbatch_01GG5Ez9KT1pPwtQaY4sW7tv | 4 分 35 秒 | 1,293,724 | 269,166 | $3.96 | 過濾零攔阻;song-168、song-590 因餘額用盡失敗,song-775 拒答 |
|
||||
| 2026-08-05 | 03-01-code | 101 碼樹狀探測(148 首)| claude-sonnet-4-6 | msgbatch_017jo3E5gWVks9b39iWMTqz3 | 2 小時 11 分 | 385,270 | 79,690 | $0.87 | 實驗:k=100 葉碼+women-power;零違規碼; 歸檔不入 repo |
|
||||
| 2026-08-06 | 命名實驗(100 組)| — | claude-sonnet-4-6 | msgbatch_01XSi1YtWzdVyYUzYh7DQRWg | 3 分 5 秒 | 55,090 | 1,168 | $0.09 | 實驗:LLM 命名對照 medoid,未採用;歸檔不入 repo |
|
||||
| 2026-08-06 | 命名實驗(100 組)| — | claude-fable-5 | msgbatch_01Y8SJj1h1ZuSQRErhqkvgZE | 3 分 2 秒 | 76,211 | 2,049 | $0.43 | 實驗:同上,加禁用 themes;未採用;歸檔不入 repo |
|
||||
| 2026-08-06 | 03-01-code | run1 | claude-sonnet-4-6 | msgbatch_01GS1opvurvsf62oknnQxhtx | 3 分 46 秒 | 1,625,458 | 362,759 | $5.16 | 現行(101 碼;883 首全數有效,零攔阻) |
|
||||
| 2026-08-06 | 03-01-code | run2 | claude-sonnet-4-6 | msgbatch_01CxnwNLWzZbRpZK7UAdpb8i | 5 分 22 秒 | 1,625,458 | 364,098 | $5.17 | 現行(101 碼;883 首全數有效,零攔阻) |
|
||||
| 2026-08-06 | 03-02-arbitration | — | claude-sonnet-4-6 | msgbatch_019xcQXwrbwDGFc9nE5M8AjE | 4 分 0 秒 | 763,193 | 42,389 | $1.46 | 已取代(2 首遭內容過濾攔阻、13 首輸出夾帶散文;定義檔修訂後重跑) |
|
||||
| 2026-08-06 | 03-02-arbitration | — | claude-sonnet-4-6 | msgbatch_01N7bDbXRSfAVUzzaj2thKeR | 4 分 20 秒 | 781,037 | 39,315 | $1.47 | 現行(644 首全數有效,零攔阻;保留 1,481/送裁 1,699) |
|
||||
| 2026-08-06 | 03-code | run3 | claude-sonnet-4-6 | msgbatch_01KnkCaGETnFJrPddrxZTYHA | 6 分 26 秒 | 1,625,458 | 363,840 | $5.17 | 現行(101 碼;883 首全數有效,零攔阻) |
|
||||
| 2026-08-14 | 04-group | run1 | claude-sonnet-4-6 | msgbatch_01UPNedog6feQzJ9WVfSAxBD | 1 分 1 秒 | 3,129 | 470 | $0.01 | 已取代(僅 3 群;改納 women-power 群後重跑) |
|
||||
| 2026-08-14 | 04-group | run1 | claude-sonnet-4-6 | msgbatch_01KntJgxdicStaMjNL12P3zi | 1 分 25 秒 | 4,170 | 558 | $0.01 | 已取代(改以 claude-fable-5 執行;vulnerable 輸出含詞彙表外碼 1 筆;歸檔另行私人備份,不入版本庫) |
|
||||
| 2026-08-14 | 04-group | run1 | claude-fable-5 | msgbatch_01Mr6goBb2Efa4YrqCprbP4U | 55 秒 | 5,553 | 2,049 | $0.08 | 現行(4 群;零違規碼;temperature 與 thinking 參數不適用於本模型,未送出) |
|
||||
| 2026-08-14 | 04-group | run2 | claude-fable-5 | msgbatch_01C17xW3YBefThTYZ83g7KkL | 1 分 21 秒 | 5,553 | 1,891 | $0.08 | 現行(4 群;零違規碼) |
|
||||
| 2026-08-14 | 04-group | run3 | claude-fable-5 | msgbatch_01DveEMYyjCAYe6wxpcCD87V | 2 分 9 秒 | 5,553 | 1,975 | $0.08 | 現行(4 群;零違規碼) |
|
||||
| 2026-08-15 | 05-01-read | run1 | claude-fable-5 | msgbatch_018NJejke75o7kzKyjZeeKwi | 2 分 12 秒 | 189,607 | 181,299 | $5.48 | 現行(144/145 有效;song-444 平台錯誤,單筆補送) |
|
||||
| 2026-08-15 | 05-01-read | run1-rescue-444 | claude-fable-5 | msgbatch_01DgygMm6iYn2qEnyqHkTMfa | 1 分 52 秒 | 1,852 | 1,557 | $0.05 | 現行(run1 之 song-444 單筆補送,成功) |
|
||||
| 2026-08-15 | 05-01-read | run2 | claude-fable-5 | msgbatch_011txWpWDtdkNXNiPA74ZUfQ | 15 分 25 秒 | 191,459 | 185,630 | $5.60 | 現行(145/145 有效) |
|
||||
| 2026-08-15 | 05-01-read | run3 | claude-fable-5 | msgbatch_01Unn7C7WCnuVoLraJRXksoG | 7 分 59 秒 | 191,459 | 190,433 | $5.72 | 現行(145/145 有效) |
|
||||
| 2026-08-15 | 05-02-consolidate | run1 | claude-fable-5 | msgbatch_01WPyH4pCCiieHs4yU3xtHzN | 2 分 28 秒 | 326,287 | 242,785 | $7.70 | 現行(145/145 有效) |
|
||||
| 2026-08-15 | 05-03-synthesize | run1 | claude-fable-5 | msgbatch_01XbNQb1SFrUxhk2ZWF7epPA | 3 分 0 秒 | 103,888 | 9,982 | $0.77 | 現行(15 個樣態;研究者審定用草稿) |
|
||||
| 2026-08-15 | 05-03-synthesize | run2 | claude-fable-5 | msgbatch_01FKqq8U6m5HLLsx7KpQES26 | 3 分 4 秒 | 13,573 | 6,883 | $0.24 | 現行(男聲群 14 首;13 個樣態) |
|
||||
| 2026-08-15 | 05-03-synthesize | run3 | claude-fable-5 | msgbatch_017tfhiRTnaxqCp6YFyoivsu | 4 分 4 秒 | 60,727 | 8,519 | $0.52 | 現行(女聲群 98 首;14 個樣態) |
|
||||
| 2026-08-15 | 05-03-synthesize | run4 | claude-fable-5 | msgbatch_01SNjTLiNPEba4LW2tcUAcTJ | 4 分 3 秒 | 29,958 | 9,793 | $0.39 | 現行(混合群 30 首;16 個樣態) |
|
||||
| 2026-08-15 | 05-04-annotate | run1 | claude-fable-5 | msgbatch_01H9gXXhNbhZG6gzZtvMHxhh | 3 分 9 秒 | 815,492 | 300,901 | $11.60 | 現行(109/111 有效;song-177、song-199 遭 max_tokens 截斷,單筆補送) |
|
||||
| 2026-08-15 | 05-04-annotate | run1-rescue-177-199 | claude-fable-5 | msgbatch_01YEdroZ51b8btPfG1aqPFCD | 3 分 5 秒 | 17,802 | 11,601 | $0.38 | 現行(run1 之 2 筆補送,max_tokens 提為 16000,成功) |
|
||||
| 2026-08-15 | 05-04-annotate | — | claude-fable-5 | msgbatch_01G67STqZsHpnhedat2HRU85 | 3 分 0 秒 | 17,802 | 14,162 | $0.44 | 誤送(補送批次輪詢中斷後誤判死亡而重送;原批次自行完成並歸檔為現行,本批次取消不及、兩筆皆完成,結果棄用) |
|
||||
| 2026-08-15 | 05-04-annotate | run2 | claude-fable-5 | msgbatch_01VVJpsfidNtF66GnHQdWnGR | 9 分 16 秒 | 815,492 | 306,582 | $11.74 | 現行(111/111 有效;max_tokens 16000,零截斷) |
|
||||
| 2026-08-15 | 05-04-annotate | run3 | claude-fable-5 | msgbatch_01AsYvZd8rsTYVfUYLzWfeMK | 4 分 12 秒 | 815,492 | 298,512 | $11.54 | 現行(111/111 有效;max_tokens 16000,零截斷) |
|
||||
「步驟」欄的名稱於 2026-08-18 一律換為現行編號(見
|
||||
`decision-log.md` 當日條目),執行當時的名稱記於末欄
|
||||
「原步驟名」,未曾改名者留空。已刪除的工序——步驟 2 的
|
||||
LLM 合併嘗試(`01-02-01-merge`,詞彙表改採詞向量分群時
|
||||
放棄)與步驟 3 的仲裁(`03-02-arbitration`,改採三票
|
||||
多數決時刪除)——不另發新名,步驟欄標「已廢棄」,僅存
|
||||
本表供總支出核算。
|
||||
|
||||
| 日期 | 步驟 | 執行 | 模型 | 批次 ID | 耗時 | input | output | 費用 (USD) | 狀態 | 原步驟名 |
|
||||
|---|---|---|---|---|---|---:|---:|---:|---|---|
|
||||
| 2026-08-04 | 1-tag | run1 | claude-sonnet-4-6 | msgbatch_01PTACDQMr8M6ahnshbedjtB | 6 分 27 秒 | 763,318 | 338,661 | $3.68 | 已取代(浮水印清洗與防圍欄修訂後重跑) | 01-01-01-tag |
|
||||
| 2026-08-05 | 1-tag | run1 | claude-sonnet-4-6 | msgbatch_01TikJNd2pZVxzQ8SaybVthu | 4 分 57 秒 | 774,604 | 323,651 | $3.59 | 已取代(合法 JSON 修訂後重跑) | 01-01-01-tag |
|
||||
| 2026-08-05 | 1-tag | run1 | claude-sonnet-4-6 | msgbatch_01JFBCNqnu1cwXyqmEKLHQYF | 4 分 30 秒 | 790,370 | 328,227 | $3.65 | 已取代(song-288 平台失敗,整批重跑驗證) | 01-01-01-tag |
|
||||
| 2026-08-05 | 1-tag | run1 | claude-sonnet-4-6 | msgbatch_01VSDneWuSbf8mShA32jiWrX | 6 分 15 秒 | 790,370 | 326,193 | $3.63 | 已取代(措辭修訂後全體重跑) | 01-01-01-tag |
|
||||
| 2026-08-05 | 1-tag | run1-rescue-288 | claude-sonnet-4-6 | msgbatch_019Hq6bNVXVmp4DjRZ2cgVda | 2 分 5 秒 | 1,015 | 376 | $0.01 | 已取代(措辭修訂後全體重跑,該首原生通過)| 01-01-01-tag |
|
||||
| 2026-08-05 | 1-tag | run1 | claude-sonnet-4-6 | msgbatch_01VgZ77KAPGWmuu3PnQFqZ7Q | 7 分 1 秒 | 794,913 | 326,435 | $3.64 | 現行 | 01-01-01-tag |
|
||||
| 2026-08-05 | 1-tag | run2 | claude-sonnet-4-6 | msgbatch_01TLFey3L4fimKxcebTZQGYn | 4 分 21 秒 | 794,913 | 328,324 | $3.65 | 現行 | 01-01-01-tag |
|
||||
| 2026-08-05 | 已廢棄 | run1 | claude-sonnet-4-6 | msgbatch_01BvYMFmH8zrUq9SNxWSNba7 | 9 分 14 秒 | 46,454 | 60,974 | $0.53 | 已取代(完整分割驗證不過:漏 366、重複分派 511、撞名 4) | 01-02-01-merge |
|
||||
| 2026-08-05 | 已廢棄 | run1 | claude-opus-5 | msgbatch_018TkYpdT3DsZqq2q94FQCUN | — | 0 | 0 | $0.00 | 拒收(temperature 已棄用) | 01-02-01-merge |
|
||||
| 2026-08-05 | 已廢棄 | run1 | claude-opus-4-8 | msgbatch_018Wuux4adz5mWSvzJjfgLxb | — | 0 | 0 | $0.00 | 拒收(temperature 已棄用) | 01-02-01-merge |
|
||||
| 2026-08-05 | 已廢棄 | run1 | claude-opus-4-6 | msgbatch_01Lmt2j1Txyof9CctUK7CbHA | 12 分 19 秒 | 46,454 | 54,864 | $0.80 | 已取代(圍欄違規;漏 190、發明 151、重複 11) | 01-02-01-merge |
|
||||
| 2026-08-05 | 已廢棄 | run1 | claude-opus-5 | msgbatch_012SEfebTnjU5uWN6hhfnhhh | 8 分 45 秒 | 63,368 | 64,000 | $1.92 | 已取代(64k 截斷) | 01-02-01-merge |
|
||||
| 2026-08-05 | 已廢棄 | run1 | claude-opus-5 | msgbatch_01Wi9xEoQKCy1nZK6myVNg77 | 11 分 34 秒 | 63,368 | 70,175 | $2.07 | 已取代(驗證不過:漏 59、發明 140、重複 3) | 01-02-01-merge |
|
||||
| 2026-08-05 | 已廢棄 | run1 | claude-fable-5 | msgbatch_01M15hUs8P9FDTqWvAHdc1cA | 32 分 45 秒 | 63,368 | 107,721 | $3.01 | 現行歸檔(JSON 語法毀損,驗證不過) | 01-02-01-merge |
|
||||
| 2026-08-05 | 已廢棄 | run1 | claude-fable-5 | msgbatch_012stRP28uuQDQwobm41HMLh | — | 0 | 0 | $0.00 | 拒收(thinking.type.enabled 不支援) | 01-02-01-merge |
|
||||
| 2026-08-05 | 已廢棄 | run1 | claude-fable-5 | msgbatch_01KpS7AMx1zAHTcNujgGsouJ | 26 分 28 秒 | 63,368 | 128,000 | $3.52 | 現行歸檔(effort max:128k 全耗於推理,正文空白) | 01-02-01-merge |
|
||||
| 2026-08-05 | 3a-code | run1 | claude-sonnet-4-6 | msgbatch_01L7VepTxSNz5vruvzzjuL2c | 6 分 36 秒 | 1,247,264 | 492,730 | $5.57 | 36 首輸出遭內容過濾攔阻,待修訂重跑 | 03-01-code |
|
||||
| 2026-08-05 | 3a-code | 引述減量實驗(36 首)| claude-sonnet-4-6 | msgbatch_01Hskdhi2DkudYmgZhhgFts7 | 3 分 51 秒 | 56,458 | 11,074 | $0.17 | 實驗:每碼一行引述,36 首全數通過過濾;歸檔不入 repo | 03-01-code |
|
||||
| 2026-08-05 | 3a-code | run1 | claude-sonnet-4-6 | msgbatch_01GG5Ez9KT1pPwtQaY4sW7tv | 4 分 35 秒 | 1,293,724 | 269,166 | $3.96 | 過濾零攔阻;song-168、song-590 因餘額用盡失敗,song-775 拒答 | 03-01-code |
|
||||
| 2026-08-05 | 3a-code | 101 碼樹狀探測(148 首)| claude-sonnet-4-6 | msgbatch_017jo3E5gWVks9b39iWMTqz3 | 2 小時 11 分 | 385,270 | 79,690 | $0.87 | 實驗:k=100 葉碼+women-power;零違規碼; 歸檔不入 repo | 03-01-code |
|
||||
| 2026-08-06 | 命名實驗(100 組)| — | claude-sonnet-4-6 | msgbatch_01XSi1YtWzdVyYUzYh7DQRWg | 3 分 5 秒 | 55,090 | 1,168 | $0.09 | 實驗:LLM 命名對照 medoid,未採用;歸檔不入 repo | |
|
||||
| 2026-08-06 | 命名實驗(100 組)| — | claude-fable-5 | msgbatch_01Y8SJj1h1ZuSQRErhqkvgZE | 3 分 2 秒 | 76,211 | 2,049 | $0.43 | 實驗:同上,加禁用 themes;未採用;歸檔不入 repo | |
|
||||
| 2026-08-06 | 3a-code | run1 | claude-sonnet-4-6 | msgbatch_01GS1opvurvsf62oknnQxhtx | 3 分 46 秒 | 1,625,458 | 362,759 | $5.16 | 現行(101 碼;883 首全數有效,零攔阻) | 03-01-code |
|
||||
| 2026-08-06 | 3a-code | run2 | claude-sonnet-4-6 | msgbatch_01CxnwNLWzZbRpZK7UAdpb8i | 5 分 22 秒 | 1,625,458 | 364,098 | $5.17 | 現行(101 碼;883 首全數有效,零攔阻) | 03-01-code |
|
||||
| 2026-08-06 | 已廢棄 | — | claude-sonnet-4-6 | msgbatch_019xcQXwrbwDGFc9nE5M8AjE | 4 分 0 秒 | 763,193 | 42,389 | $1.46 | 已取代(2 首遭內容過濾攔阻、13 首輸出夾帶散文;定義檔修訂後重跑) | 03-02-arbitration |
|
||||
| 2026-08-06 | 已廢棄 | — | claude-sonnet-4-6 | msgbatch_01N7bDbXRSfAVUzzaj2thKeR | 4 分 20 秒 | 781,037 | 39,315 | $1.47 | 現行(644 首全數有效,零攔阻;保留 1,481/送裁 1,699) | 03-02-arbitration |
|
||||
| 2026-08-06 | 3a-code | run3 | claude-sonnet-4-6 | msgbatch_01KnkCaGETnFJrPddrxZTYHA | 6 分 26 秒 | 1,625,458 | 363,840 | $5.17 | 現行(101 碼;883 首全數有效,零攔阻) | 03-code |
|
||||
| 2026-08-14 | 4-group | run1 | claude-sonnet-4-6 | msgbatch_01UPNedog6feQzJ9WVfSAxBD | 1 分 1 秒 | 3,129 | 470 | $0.01 | 已取代(僅 3 群;改納 women-power 群後重跑) | 04-group |
|
||||
| 2026-08-14 | 4-group | run1 | claude-sonnet-4-6 | msgbatch_01KntJgxdicStaMjNL12P3zi | 1 分 25 秒 | 4,170 | 558 | $0.01 | 已取代(改以 claude-fable-5 執行;vulnerable 輸出含詞彙表外碼 1 筆;歸檔另行私人備份,不入版本庫) | 04-group |
|
||||
| 2026-08-14 | 4-group | run1 | claude-fable-5 | msgbatch_01Mr6goBb2Efa4YrqCprbP4U | 55 秒 | 5,553 | 2,049 | $0.08 | 現行(4 群;零違規碼;temperature 與 thinking 參數不適用於本模型,未送出) | 04-group |
|
||||
| 2026-08-14 | 4-group | run2 | claude-fable-5 | msgbatch_01C17xW3YBefThTYZ83g7KkL | 1 分 21 秒 | 5,553 | 1,891 | $0.08 | 現行(4 群;零違規碼) | 04-group |
|
||||
| 2026-08-14 | 4-group | run3 | claude-fable-5 | msgbatch_01DveEMYyjCAYe6wxpcCD87V | 2 分 9 秒 | 5,553 | 1,975 | $0.08 | 現行(4 群;零違規碼) | 04-group |
|
||||
| 2026-08-15 | 5a-read | run1 | claude-fable-5 | msgbatch_018NJejke75o7kzKyjZeeKwi | 2 分 12 秒 | 189,607 | 181,299 | $5.48 | 現行(144/145 有效;song-444 平台錯誤,單筆補送) | 05-01-read |
|
||||
| 2026-08-15 | 5a-read | run1-rescue-444 | claude-fable-5 | msgbatch_01DgygMm6iYn2qEnyqHkTMfa | 1 分 52 秒 | 1,852 | 1,557 | $0.05 | 現行(run1 之 song-444 單筆補送,成功) | 05-01-read |
|
||||
| 2026-08-15 | 5a-read | run2 | claude-fable-5 | msgbatch_011txWpWDtdkNXNiPA74ZUfQ | 15 分 25 秒 | 191,459 | 185,630 | $5.60 | 現行(145/145 有效) | 05-01-read |
|
||||
| 2026-08-15 | 5a-read | run3 | claude-fable-5 | msgbatch_01Unn7C7WCnuVoLraJRXksoG | 7 分 59 秒 | 191,459 | 190,433 | $5.72 | 現行(145/145 有效) | 05-01-read |
|
||||
| 2026-08-15 | 5b-consolidate | run1 | claude-fable-5 | msgbatch_01WPyH4pCCiieHs4yU3xtHzN | 2 分 28 秒 | 326,287 | 242,785 | $7.70 | 現行(145/145 有效) | 05-02-consolidate |
|
||||
| 2026-08-15 | 5c-synthesize | run1 | claude-fable-5 | msgbatch_01XbNQb1SFrUxhk2ZWF7epPA | 3 分 0 秒 | 103,888 | 9,982 | $0.77 | 現行(15 個樣態;研究者審定用草稿) | 05-03-synthesize |
|
||||
| 2026-08-15 | 5c-synthesize | run2 | claude-fable-5 | msgbatch_01FKqq8U6m5HLLsx7KpQES26 | 3 分 4 秒 | 13,573 | 6,883 | $0.24 | 現行(男聲群 14 首;13 個樣態) | 05-03-synthesize |
|
||||
| 2026-08-15 | 5c-synthesize | run3 | claude-fable-5 | msgbatch_017tfhiRTnaxqCp6YFyoivsu | 4 分 4 秒 | 60,727 | 8,519 | $0.52 | 現行(女聲群 98 首;14 個樣態) | 05-03-synthesize |
|
||||
| 2026-08-15 | 5c-synthesize | run4 | claude-fable-5 | msgbatch_01SNjTLiNPEba4LW2tcUAcTJ | 4 分 3 秒 | 29,958 | 9,793 | $0.39 | 現行(混合群 30 首;16 個樣態) | 05-03-synthesize |
|
||||
| 2026-08-15 | 5d-annotate | run1 | claude-fable-5 | msgbatch_01H9gXXhNbhZG6gzZtvMHxhh | 3 分 9 秒 | 815,492 | 300,901 | $11.60 | 現行(109/111 有效;song-177、song-199 遭 max_tokens 截斷,單筆補送) | 05-04-annotate |
|
||||
| 2026-08-15 | 5d-annotate | run1-rescue-177-199 | claude-fable-5 | msgbatch_01YEdroZ51b8btPfG1aqPFCD | 3 分 5 秒 | 17,802 | 11,601 | $0.38 | 現行(run1 之 2 筆補送,max_tokens 提為 16000,成功) | 05-04-annotate |
|
||||
| 2026-08-15 | 5d-annotate | — | claude-fable-5 | msgbatch_01G67STqZsHpnhedat2HRU85 | 3 分 0 秒 | 17,802 | 14,162 | $0.44 | 誤送(補送批次輪詢中斷後誤判死亡而重送;原批次自行完成並歸檔為現行,本批次取消不及、兩筆皆完成,結果棄用) | 05-04-annotate |
|
||||
| 2026-08-15 | 5d-annotate | run2 | claude-fable-5 | msgbatch_01VVJpsfidNtF66GnHQdWnGR | 9 分 16 秒 | 815,492 | 306,582 | $11.74 | 現行(111/111 有效;max_tokens 16000,零截斷) | 05-04-annotate |
|
||||
| 2026-08-15 | 5d-annotate | run3 | claude-fable-5 | msgbatch_01AsYvZd8rsTYVfUYLzWfeMK | 4 分 12 秒 | 815,492 | 298,512 | $11.54 | 現行(111/111 有效;max_tokens 16000,零截斷) | 05-04-annotate |
|
||||
|
||||
累計支出:$125.65。
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
Change Log
|
||||
==========
|
||||
|
||||
|
||||
version 1.0.0
|
||||
-------------
|
||||
|
||||
Released 2026/8/19
|
||||
|
||||
Initial release.
|
||||
@@ -13,6 +13,8 @@ This is a collection of supporting tools for the conference paper "流行音樂
|
||||
:maxdepth: 2
|
||||
:caption: Contents:
|
||||
|
||||
changelog
|
||||
|
||||
|
||||
Indices and tables
|
||||
==================
|
||||
|
||||
@@ -6,5 +6,5 @@
|
||||
"""Tools for A Feminist Audit of Pop Music."""
|
||||
|
||||
|
||||
VERSION: str = "0.0.0"
|
||||
VERSION: str = "1.0.0"
|
||||
"""The package version."""
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -24,8 +24,7 @@ keyword set for ``export-llm-input --extras`` is written as a JSON
|
||||
file holding the group name keywords plus every extra a-priori
|
||||
keyword the caller gives with the repeatable ``--extra-keyword``
|
||||
command-line option, as
|
||||
:attr:`KeywordsToMerge.KEYWORDS_TO_MERGE_JSON`; with no
|
||||
``--extra-keyword``, it holds the group names alone. No default
|
||||
:attr:`KeywordsToMerge.KEYWORDS_TO_MERGE_JSON`. No default
|
||||
extra keyword is ever injected; the caller supplies each one
|
||||
consciously. Finally, the command-line choices and the
|
||||
environment that produced the numbers -- neither recoverable from
|
||||
@@ -160,15 +159,8 @@ class KeywordPooler:
|
||||
if line.strip() == "":
|
||||
continue
|
||||
record: Any = json.loads(line)
|
||||
if not isinstance(record, dict) or "id" not in record:
|
||||
raise ValueError(
|
||||
f"{path}: record without \"id\": {line}")
|
||||
if "error" in record:
|
||||
continue
|
||||
if "text" not in record:
|
||||
raise ValueError(
|
||||
f"{path}: id {record['id']}: record without"
|
||||
" \"text\" or \"error\"")
|
||||
song_id: int = cls.__parse_song_id(record["id"], path)
|
||||
try:
|
||||
keywords: Any = json.loads(
|
||||
@@ -297,7 +289,8 @@ class KeywordGroups:
|
||||
class KeywordClusterer:
|
||||
"""The clusterer of the pooled keywords into coding groups."""
|
||||
|
||||
DEFAULT_MODEL: str = "sentence-transformers/all-mpnet-base-v2"
|
||||
DEFAULT_MODEL: ClassVar[str] \
|
||||
= "sentence-transformers/all-mpnet-base-v2"
|
||||
"""The sentence embedding model used when the caller names
|
||||
none."""
|
||||
|
||||
@@ -668,11 +661,7 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||
parser.add_argument(
|
||||
"output_dir", type=Path,
|
||||
help="the output directory, created if missing, that"
|
||||
f" receives {PooledKeywords.SOURCE_KEYWORDS_TXT},"
|
||||
f" {KeywordGroups.RESULT_KEYWORDS_TXT},"
|
||||
f" {KeywordGroups.RESULT_GROUPS_CSV},"
|
||||
f" {KeywordsToMerge.KEYWORDS_TO_MERGE_JSON},"
|
||||
f" and {RunMeta.META_JSON}")
|
||||
" receives the run's output artifacts")
|
||||
parser.add_argument(
|
||||
"--model", default=model,
|
||||
help=f"the sentence embedding model (default \"{model}\")")
|
||||
@@ -698,16 +687,9 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
"""Pool the two tagging runs' keywords and cluster them.
|
||||
|
||||
Writes the five fixed-named artifacts under the output
|
||||
directory, creating it (with parents) if it does not exist:
|
||||
the pooled keyword text file; then the group membership CSV
|
||||
file, holding the clustering result alone; the group name
|
||||
keyword text file, holding the same group names as a readable
|
||||
list; the coding keyword set JSON file, holding the group
|
||||
names plus every extra keyword given via ``--extra-keyword``;
|
||||
and the run metadata JSON file, recording the command-line
|
||||
choices and the environment. Each file is written as soon as
|
||||
its content is computed, so when the input is rejected, or an
|
||||
Creates the output directory (with parents) if it does not
|
||||
exist. Each output artifact is written as soon as its
|
||||
content is computed, so when the input is rejected, or an
|
||||
extra keyword duplicates a group name or another extra
|
||||
keyword, the output directory holds whatever the steps before
|
||||
the failing one produced, and the error message names what
|
||||
@@ -719,8 +701,8 @@ def main(argv: list[str] | None = None) -> int:
|
||||
"""
|
||||
started: float = time.monotonic()
|
||||
args: argparse.Namespace = parse_args(argv)
|
||||
args.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
args.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
source: PooledKeywords = KeywordPooler(
|
||||
args.run_dir_1, args.run_dir_2, args.output_dir).run()
|
||||
clusters: KeywordGroups = KeywordClusterer(
|
||||
@@ -735,7 +717,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||
f"Done. Clustered {len(source.keywords)} keywords into"
|
||||
f" {len(clusters.names)}. {elapsed} elapsed.",
|
||||
file=sys.stderr)
|
||||
except ClusterError as error:
|
||||
except (ClusterError, OSError) as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
@@ -11,26 +11,17 @@ project's lyrics-only firewall: the output carries only the
|
||||
lyrics text of each song, identified by an opaque song key; no
|
||||
title, artist, or chart data crosses into the LLM input.
|
||||
|
||||
With ``--extras``, each record's ``content`` becomes a JSON
|
||||
object serialized as a string, its ``lyrics`` key holding the
|
||||
song's lyrics followed by the keys of the given extras file in
|
||||
their file order, so a step that needs parameters alongside the
|
||||
lyrics can carry them without this module knowing what they mean.
|
||||
|
||||
With ``--extras-per-id``, the same merge happens per song: the
|
||||
given file maps a song ID to the extra keys of that one song, and
|
||||
the export is restricted to the song IDs the file names, so a step
|
||||
that revisits only some of the songs, each with its own parameters,
|
||||
gets exactly those records. The two options may be given together,
|
||||
in which case a record's keys are ``lyrics``, the shared extras'
|
||||
keys, then that song's own keys, each group in its file order.
|
||||
With ``--extras`` and ``--extras-per-id``, a record's content may
|
||||
carry extra parameters alongside the lyrics, so a step that needs
|
||||
them can get them without this module knowing what they mean; see
|
||||
the exporter's content-building step for how the two merge.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from typing import Any, ClassVar
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -40,6 +31,270 @@ from ..models import Song
|
||||
from ..utils import format_duration
|
||||
|
||||
|
||||
class LlmInputExporter:
|
||||
"""The exporter of the LLM input JSONL file."""
|
||||
|
||||
__LYRICS_KEY: ClassVar[str] = "lyrics"
|
||||
"""The key holding the lyrics in a record's merged content,
|
||||
and the key forbidden in an extras file."""
|
||||
|
||||
def __init__(
|
||||
self, output_jsonl: Path, extras: Path | None = None,
|
||||
extras_per_id: Path | None = None) -> None:
|
||||
"""Set up the exporter of the LLM input JSONL file.
|
||||
|
||||
:param output_jsonl: The JSONL output file.
|
||||
:param extras: The extras JSON file, or None for none.
|
||||
:param extras_per_id: The per-ID extras JSON file, or
|
||||
None for none.
|
||||
"""
|
||||
self.__output_jsonl: Path = output_jsonl
|
||||
"""The JSONL output file."""
|
||||
self.__extras_path: Path | None = extras
|
||||
"""The extras JSON file, or None for none."""
|
||||
self.__extras_per_id_path: Path | None = extras_per_id
|
||||
"""The per-ID extras JSON file, or None for none."""
|
||||
|
||||
def run(self) -> int:
|
||||
"""Export the songs' lyrics to the output JSONL file.
|
||||
|
||||
:return: The number of songs exported.
|
||||
:raises OSError: When a file cannot be read or written.
|
||||
:raises sqlalchemy.exc.SQLAlchemyError: When the working
|
||||
store cannot be read.
|
||||
:raises ValueError: When an extras file is malformed, an
|
||||
exported song has no lyrics, or the per-ID extras
|
||||
name a song the working store does not have.
|
||||
"""
|
||||
session: Session = ds.get_db()
|
||||
try:
|
||||
extras: dict[str, Any] | None = None
|
||||
if self.__extras_path is not None:
|
||||
extras = self.__load_extras(self.__extras_path)
|
||||
extras_per_id: dict[str, dict[str, Any]] | None = None
|
||||
if self.__extras_per_id_path is not None:
|
||||
extras_per_id = self.__load_extras_per_id(
|
||||
self.__extras_per_id_path)
|
||||
lines: list[str] = self.__build_lines(
|
||||
session, extras, extras_per_id)
|
||||
finally:
|
||||
session.close()
|
||||
self.__write_output(lines)
|
||||
return len(lines)
|
||||
|
||||
@staticmethod
|
||||
def __no_duplicate_keys(
|
||||
pairs: list[tuple[str, Any]]) -> dict[str, Any]:
|
||||
"""Build a dict from JSON object pairs, rejecting
|
||||
duplicates.
|
||||
|
||||
:param pairs: The key-value pairs of a JSON object, in
|
||||
file order.
|
||||
:return: The pairs as a dict, in file order.
|
||||
:raises ValueError: When a key appears more than once.
|
||||
"""
|
||||
result: dict[str, Any] = {}
|
||||
key: str
|
||||
value: Any
|
||||
for key, value in pairs:
|
||||
if key in result:
|
||||
raise ValueError(
|
||||
f"duplicate key \"{key}\" in extras")
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
@classmethod
|
||||
def __load_json_object(
|
||||
cls, path: Path, label: str) -> dict[str, Any]:
|
||||
"""Load a single JSON object from a file, in file order.
|
||||
|
||||
:param path: The JSON file.
|
||||
:param label: The kind of file, for the error messages.
|
||||
:return: The object, in file order.
|
||||
:raises OSError: When the file cannot be read.
|
||||
:raises ValueError: When the file is not valid JSON, is
|
||||
not a JSON object, or has duplicate keys.
|
||||
"""
|
||||
with open(path, encoding="utf-8") as file:
|
||||
text: str = file.read()
|
||||
try:
|
||||
data: Any = json.loads(
|
||||
text, object_pairs_hook=cls.__no_duplicate_keys)
|
||||
except json.JSONDecodeError as error:
|
||||
raise ValueError(
|
||||
f"invalid JSON in {label} file {path}: {error}") \
|
||||
from error
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError(
|
||||
f"{label} file {path} must contain a JSON object")
|
||||
return data
|
||||
|
||||
@classmethod
|
||||
def __load_extras(cls, path: Path) -> dict[str, Any]:
|
||||
"""Load the extras object from a JSON file.
|
||||
|
||||
:param path: The extras JSON file.
|
||||
:return: The extras, in file order.
|
||||
:raises OSError: When the file cannot be read.
|
||||
:raises ValueError: When the file is not valid JSON, is
|
||||
not a JSON object, has duplicate keys, or has a
|
||||
"lyrics" key.
|
||||
"""
|
||||
data: dict[str, Any] = cls.__load_json_object(
|
||||
path, "extras")
|
||||
if cls.__LYRICS_KEY in data:
|
||||
raise ValueError(
|
||||
f"extras file {path} must not have a"
|
||||
f" \"{cls.__LYRICS_KEY}\" key")
|
||||
return data
|
||||
|
||||
@classmethod
|
||||
def __load_extras_per_id(
|
||||
cls, path: Path) -> dict[str, dict[str, Any]]:
|
||||
"""Load the per-ID extras object from a JSON file.
|
||||
|
||||
:param path: The per-ID extras JSON file, mapping a song
|
||||
ID, as ``song-<N>``, to the extras of that one song.
|
||||
:return: The extras of each song ID, in file order, every
|
||||
song's own extras in their file order too.
|
||||
:raises OSError: When the file cannot be read.
|
||||
:raises ValueError: When the file is not valid JSON, is
|
||||
not a JSON object, has duplicate keys, has a song
|
||||
whose value is not a JSON object, or has a song with
|
||||
a "lyrics" key.
|
||||
"""
|
||||
data: dict[str, Any] = cls.__load_json_object(
|
||||
path, "per-ID extras")
|
||||
song_id: str
|
||||
extras: Any
|
||||
for song_id, extras in data.items():
|
||||
if not isinstance(extras, dict):
|
||||
raise ValueError(
|
||||
f"per-ID extras file {path}: id {song_id}"
|
||||
" must have a JSON object")
|
||||
if cls.__LYRICS_KEY in extras:
|
||||
raise ValueError(
|
||||
f"per-ID extras file {path}: id {song_id}"
|
||||
f" must not have a \"{cls.__LYRICS_KEY}\""
|
||||
" key")
|
||||
return data
|
||||
|
||||
def __build_lines(
|
||||
self, session: Session,
|
||||
extras: dict[str, Any] | None = None,
|
||||
extras_per_id: dict[str, dict[str, Any]] | None
|
||||
= None) -> list[str]:
|
||||
"""Build the JSONL lines of the exported songs' lyrics.
|
||||
|
||||
Every song is exported, unless per-ID extras are given,
|
||||
in which case only the songs they name are; see
|
||||
:meth:`__build_content` for how the extras merge into a
|
||||
record's content.
|
||||
|
||||
:param session: The database session.
|
||||
:param extras: The extra parameters merged into every
|
||||
record's content alongside the lyrics, in the order
|
||||
they are to appear, or None for none.
|
||||
:param extras_per_id: The extra parameters merged into
|
||||
the content of one record alone, keyed by that
|
||||
record's song ID and in the order they are to appear,
|
||||
restricting the export to the song IDs they name, or
|
||||
None for no such extras and no such restriction.
|
||||
:return: The JSON lines, one per exported song, ordered by
|
||||
song ID.
|
||||
:raises ValueError: When an exported song has no lyrics,
|
||||
or the per-ID extras name a song the working store
|
||||
does not have.
|
||||
"""
|
||||
lines: list[str] = []
|
||||
exported: set[str] = set()
|
||||
song: Song
|
||||
for song in session.scalars(
|
||||
sa.select(Song).order_by(Song.id)):
|
||||
song_id: str = f"song-{song.id}"
|
||||
if extras_per_id is not None \
|
||||
and song_id not in extras_per_id:
|
||||
continue
|
||||
if song.lyrics is None:
|
||||
raise ValueError(
|
||||
f"song {song.id} \"{song.title}\": no lyrics")
|
||||
song_extras: dict[str, Any] | None = None \
|
||||
if extras_per_id is None \
|
||||
else extras_per_id[song_id]
|
||||
content: str = self.__build_content(
|
||||
song.lyrics, extras, song_extras)
|
||||
record: dict[str, str] = {
|
||||
"id": song_id, "content": content}
|
||||
lines.append(json.dumps(record, ensure_ascii=False))
|
||||
exported.add(song_id)
|
||||
if extras_per_id is not None:
|
||||
missing: list[str] = sorted(
|
||||
set(extras_per_id) - exported)
|
||||
if len(missing) > 0:
|
||||
raise ValueError(
|
||||
"the per-ID extras name songs the working"
|
||||
f" store does not have: {', '.join(missing)}")
|
||||
return lines
|
||||
|
||||
@classmethod
|
||||
def __build_content(
|
||||
cls, lyrics: str, extras: dict[str, Any] | None,
|
||||
song_extras: dict[str, Any] | None) -> str:
|
||||
"""Build the content of one exported record.
|
||||
|
||||
Without extras of either kind, a record's content is the
|
||||
bare lyrics string. With ``--extras``, the content
|
||||
becomes a JSON object serialized as a string, its
|
||||
"lyrics" key holding the song's lyrics followed by the
|
||||
keys of the given extras file, in their file order. With
|
||||
``--extras-per-id``, the same merge happens per song: the
|
||||
song's own extra keys follow the lyrics instead. When
|
||||
both are given, a record's keys are "lyrics", the shared
|
||||
extras' keys, then that song's own keys, each group in
|
||||
its file order.
|
||||
|
||||
:param lyrics: The lyrics of the song.
|
||||
:param extras: The extra parameters shared by every
|
||||
record, in the order they are to appear, or None for
|
||||
none.
|
||||
:param song_extras: The extra parameters of this record
|
||||
alone, in the order they are to appear, or None for
|
||||
none.
|
||||
:return: The bare lyrics when there are no extras of
|
||||
either kind, or otherwise a JSON object serialized as
|
||||
a string, whose first key is "lyrics" holding the
|
||||
lyrics, followed by the shared extras' keys and then
|
||||
this record's own keys, each group in its given
|
||||
order.
|
||||
"""
|
||||
if extras is None and song_extras is None:
|
||||
return lyrics
|
||||
payload: dict[str, Any] = {cls.__LYRICS_KEY: lyrics}
|
||||
if extras is not None:
|
||||
payload.update(extras)
|
||||
if song_extras is not None:
|
||||
payload.update(song_extras)
|
||||
return json.dumps(payload, ensure_ascii=False)
|
||||
|
||||
def __write_output(self, lines: list[str]) -> None:
|
||||
"""Write the exported lines to the output JSONL file.
|
||||
|
||||
Creates the parent directory when it does not exist.
|
||||
|
||||
:param lines: The JSONL lines, in the output order.
|
||||
:return: None.
|
||||
:raises OSError: When the file cannot be written.
|
||||
"""
|
||||
self.__output_jsonl.parent.mkdir(
|
||||
parents=True, exist_ok=True)
|
||||
with open(
|
||||
self.__output_jsonl, "w",
|
||||
encoding="utf-8") as file:
|
||||
line: str
|
||||
for line in lines:
|
||||
file.write(line + "\n")
|
||||
|
||||
|
||||
def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||
"""Parse the command-line arguments.
|
||||
|
||||
@@ -56,227 +311,33 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||
parser.add_argument(
|
||||
"--extras", type=Path, default=None,
|
||||
help="a JSON file holding a single JSON object of extra"
|
||||
" parameters; when given, each record's \"content\""
|
||||
" becomes a JSON object string with a \"lyrics\" key"
|
||||
" followed by the extras' keys, instead of the bare"
|
||||
" lyrics string")
|
||||
" parameters merged into every record's content")
|
||||
parser.add_argument(
|
||||
"--extras-per-id", type=Path, default=None,
|
||||
help="a JSON file holding a single JSON object that maps a"
|
||||
" song ID, as \"song-<N>\", to a JSON object of extra"
|
||||
" parameters for that one song; the song's object is"
|
||||
" merged into its \"content\" the same way as with"
|
||||
" --extras, and the export is restricted to the song"
|
||||
" IDs the file names")
|
||||
" parameters for that one song, restricting the"
|
||||
" export to the song IDs the file names")
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def __no_duplicate_keys(
|
||||
pairs: list[tuple[str, Any]]) -> dict[str, Any]:
|
||||
"""Build a dict from JSON object pairs, rejecting duplicates.
|
||||
|
||||
:param pairs: The key-value pairs of a JSON object, in file
|
||||
order.
|
||||
:return: The pairs as a dict, in file order.
|
||||
:raises ValueError: When a key appears more than once.
|
||||
"""
|
||||
result: dict[str, Any] = {}
|
||||
key: str
|
||||
value: Any
|
||||
for key, value in pairs:
|
||||
if key in result:
|
||||
raise ValueError(f"duplicate key \"{key}\" in extras")
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
|
||||
def __load_json_object(path: Path, label: str) -> dict[str, Any]:
|
||||
"""Load a single JSON object from a file, in file order.
|
||||
|
||||
:param path: The JSON file.
|
||||
:param label: The kind of file, for the error messages.
|
||||
:return: The object, in file order.
|
||||
:raises OSError: When the file cannot be read.
|
||||
:raises ValueError: When the file is not valid JSON, is not
|
||||
a JSON object, or has duplicate keys.
|
||||
"""
|
||||
with open(path, encoding="utf-8") as file:
|
||||
text: str = file.read()
|
||||
try:
|
||||
data: Any = json.loads(
|
||||
text, object_pairs_hook=__no_duplicate_keys)
|
||||
except json.JSONDecodeError as error:
|
||||
raise ValueError(
|
||||
f"invalid JSON in {label} file {path}: {error}") \
|
||||
from error
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError(
|
||||
f"{label} file {path} must contain a JSON object")
|
||||
return data
|
||||
|
||||
|
||||
def load_extras(path: Path) -> dict[str, Any]:
|
||||
"""Load the extras object from a JSON file.
|
||||
|
||||
:param path: The extras JSON file.
|
||||
:return: The extras, in file order.
|
||||
:raises OSError: When the file cannot be read.
|
||||
:raises ValueError: When the file is not valid JSON, is not
|
||||
a JSON object, has duplicate keys, or has a "lyrics" key.
|
||||
"""
|
||||
data: dict[str, Any] = __load_json_object(path, "extras")
|
||||
if "lyrics" in data:
|
||||
raise ValueError(
|
||||
f"extras file {path} must not have a \"lyrics\" key")
|
||||
return data
|
||||
|
||||
|
||||
def load_extras_per_id(path: Path) -> dict[str, dict[str, Any]]:
|
||||
"""Load the per-ID extras object from a JSON file.
|
||||
|
||||
:param path: The per-ID extras JSON file, mapping a song ID,
|
||||
as ``song-<N>``, to the extras of that one song.
|
||||
:return: The extras of each song ID, in file order, every
|
||||
song's own extras in their file order too.
|
||||
:raises OSError: When the file cannot be read.
|
||||
:raises ValueError: When the file is not valid JSON, is not
|
||||
a JSON object, has duplicate keys, has a song whose value
|
||||
is not a JSON object, or has a song with a "lyrics" key.
|
||||
"""
|
||||
data: dict[str, Any] = __load_json_object(path, "per-ID extras")
|
||||
song_id: str
|
||||
extras: Any
|
||||
for song_id, extras in data.items():
|
||||
if not isinstance(extras, dict):
|
||||
raise ValueError(
|
||||
f"per-ID extras file {path}: id {song_id} must"
|
||||
" have a JSON object")
|
||||
if "lyrics" in extras:
|
||||
raise ValueError(
|
||||
f"per-ID extras file {path}: id {song_id} must not"
|
||||
" have a \"lyrics\" key")
|
||||
return data
|
||||
|
||||
|
||||
def __build_content(
|
||||
lyrics: str,
|
||||
extras: dict[str, Any] | None,
|
||||
song_extras: dict[str, Any] | None) -> str:
|
||||
"""Build the content of one exported record.
|
||||
|
||||
:param lyrics: The lyrics of the song.
|
||||
:param extras: The extra parameters shared by every record,
|
||||
in the order they are to appear, or None for none.
|
||||
:param song_extras: The extra parameters of this record
|
||||
alone, in the order they are to appear, or None for none.
|
||||
:return: The bare lyrics when there are no extras of either
|
||||
kind, or otherwise a JSON object serialized as a string,
|
||||
whose first key is ``"lyrics"`` holding the lyrics,
|
||||
followed by the shared extras' keys and then this
|
||||
record's own keys, each group in its given order.
|
||||
"""
|
||||
if extras is None and song_extras is None:
|
||||
return lyrics
|
||||
payload: dict[str, Any] = {"lyrics": lyrics}
|
||||
if extras is not None:
|
||||
payload.update(extras)
|
||||
if song_extras is not None:
|
||||
payload.update(song_extras)
|
||||
return json.dumps(payload, ensure_ascii=False)
|
||||
|
||||
|
||||
def build_lines(
|
||||
session: Session,
|
||||
extras: dict[str, Any] | None = None,
|
||||
extras_per_id: dict[str, dict[str, Any]] | None = None) \
|
||||
-> list[str]:
|
||||
"""Build the JSONL lines of the exported songs' lyrics.
|
||||
|
||||
Without extras of either kind, each record's ``content`` is
|
||||
the bare lyrics string. With extras, ``content`` is a JSON
|
||||
object serialized as a string, whose first key is ``"lyrics"``
|
||||
holding the lyrics string, followed by the shared extras' keys
|
||||
and then the song's own per-ID extras' keys, each group in its
|
||||
given order.
|
||||
|
||||
Every song is exported, unless per-ID extras are given, in
|
||||
which case only the songs they name are.
|
||||
|
||||
:param session: The database session.
|
||||
:param extras: The extra parameters merged into every
|
||||
record's content alongside the lyrics, in the order they
|
||||
are to appear, or None for none.
|
||||
:param extras_per_id: The extra parameters merged into the
|
||||
content of one record alone, keyed by that record's song
|
||||
ID and in the order they are to appear, restricting the
|
||||
export to the song IDs they name, or None for no such
|
||||
extras and no such restriction.
|
||||
:return: The JSON lines, one per exported song, ordered by
|
||||
song ID.
|
||||
:raises ValueError: When an exported song has no lyrics, or
|
||||
the per-ID extras name a song the working store does not
|
||||
have.
|
||||
"""
|
||||
lines: list[str] = []
|
||||
exported: set[str] = set()
|
||||
song: Song
|
||||
for song in session.scalars(sa.select(Song).order_by(Song.id)):
|
||||
song_id: str = f"song-{song.id}"
|
||||
if extras_per_id is not None and song_id not in extras_per_id:
|
||||
continue
|
||||
if song.lyrics is None:
|
||||
raise ValueError(
|
||||
f"song {song.id} \"{song.title}\": no lyrics")
|
||||
song_extras: dict[str, Any] | None = None \
|
||||
if extras_per_id is None else extras_per_id[song_id]
|
||||
content: str = __build_content(
|
||||
song.lyrics, extras, song_extras)
|
||||
record: dict[str, str] = {
|
||||
"id": song_id, "content": content}
|
||||
lines.append(json.dumps(record, ensure_ascii=False))
|
||||
exported.add(song_id)
|
||||
if extras_per_id is not None:
|
||||
missing: list[str] = sorted(set(extras_per_id) - exported)
|
||||
if len(missing) > 0:
|
||||
raise ValueError(
|
||||
"the per-ID extras name songs the working store"
|
||||
f" does not have: {', '.join(missing)}")
|
||||
return lines
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
"""Export the LLM input JSONL file from the working store.
|
||||
|
||||
Every song is exported, unless ``--extras-per-id`` is given,
|
||||
in which case only the songs its file names are.
|
||||
|
||||
:param argv: The command-line arguments, or None for
|
||||
``sys.argv``.
|
||||
:return: The exit status: 0 on success, non-zero on failure.
|
||||
"""
|
||||
started: float = time.monotonic()
|
||||
args: argparse.Namespace = parse_args(argv)
|
||||
session: Session = ds.get_db()
|
||||
lines: list[str]
|
||||
try:
|
||||
extras: dict[str, Any] | None = None
|
||||
if args.extras is not None:
|
||||
extras = load_extras(args.extras)
|
||||
extras_per_id: dict[str, dict[str, Any]] | None = None
|
||||
if args.extras_per_id is not None:
|
||||
extras_per_id = load_extras_per_id(args.extras_per_id)
|
||||
lines = build_lines(session, extras, extras_per_id)
|
||||
count: int = LlmInputExporter(
|
||||
args.output_jsonl, args.extras,
|
||||
args.extras_per_id).run()
|
||||
except (OSError, sa.exc.SQLAlchemyError, ValueError) as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 1
|
||||
finally:
|
||||
session.close()
|
||||
args.output_jsonl.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(args.output_jsonl, "w", encoding="utf-8") as file:
|
||||
line: str
|
||||
for line in lines:
|
||||
file.write(line + "\n")
|
||||
elapsed: str = format_duration(time.monotonic() - started)
|
||||
print(f"Done. {len(lines)} songs exported."
|
||||
print(f"Done. {count} songs exported."
|
||||
f" {elapsed} elapsed.", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
@@ -12,12 +12,10 @@ layer. The working store is only read, never written; the
|
||||
``build-db`` subcommand assembles the captured files into the
|
||||
store on the next rebuild.
|
||||
|
||||
Every fetched row is meant for later human verification: the
|
||||
description of the resolved item is recorded in the note column
|
||||
so that a bad match can be spotted. An unresolved artist or an
|
||||
error on one artist is noted on its row and does not fail the
|
||||
run. A row whose name is no longer an artist of the store is
|
||||
dropped from the snapshot and reported on the standard error.
|
||||
An unresolved artist or an error on one artist is noted on its
|
||||
row and does not fail the run. A row whose name is no longer an
|
||||
artist of the store is dropped from the snapshot and reported on
|
||||
the standard error.
|
||||
"""
|
||||
import argparse
|
||||
import csv
|
||||
@@ -33,7 +31,7 @@ import urllib.request
|
||||
from collections.abc import Container, Sequence
|
||||
from dataclasses import asdict, dataclass, field, fields
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal, TextIO
|
||||
from typing import Any, ClassVar, Literal, TextIO
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -43,71 +41,6 @@ from ..database import ds
|
||||
from ..models import Artist, Song, SongArtist
|
||||
from ..utils import format_duration
|
||||
|
||||
API_URL: str = "https://www.wikidata.org/w/api.php"
|
||||
"""The URL of the Wikidata API endpoint."""
|
||||
SPARQL_URL: str = "https://query.wikidata.org/sparql"
|
||||
"""The URL of the Wikidata Query Service SPARQL endpoint."""
|
||||
USER_AGENT: str = (
|
||||
f"pop-fem-audit-tools/{VERSION}"
|
||||
" (https://github.com/imacat/pop-fem-audit;"
|
||||
" mailto:imacat@mail.imacat.idv.tw)")
|
||||
"""The User-Agent header sent on every HTTP request."""
|
||||
TIMEOUT: float = 30.0
|
||||
"""The timeout of an API HTTP request, in seconds."""
|
||||
SPARQL_TIMEOUT: float = 90.0
|
||||
"""The timeout of a SPARQL HTTP request, in seconds.
|
||||
|
||||
Higher than the API timeout: the WDQS server aborts a slow
|
||||
query at 60 seconds, and a lower client timeout would race
|
||||
that server-side abort and misclassify a slow-but-answerable
|
||||
query as a client-side timeout instead of letting the server's
|
||||
own HTTP error response arrive and enter the retry path."""
|
||||
SLEEP_SECONDS: float = 1.0
|
||||
"""The delay between consecutive HTTP requests, in seconds."""
|
||||
MAX_ATTEMPTS: int = 5
|
||||
"""The maximum number of attempts on a transient error."""
|
||||
RETRY_SECONDS: float = 15.0
|
||||
"""The back-off unit on a transient error, in seconds;
|
||||
multiplied by the attempt number already made."""
|
||||
RETRY_STATUSES: frozenset[int] = frozenset({429, 500, 502, 503})
|
||||
"""The HTTP statuses that are retried with a back-off."""
|
||||
MAX_STAGE1_TITLES: int = 3
|
||||
"""The maximum number of charted titles used for the stage-1 song
|
||||
corroboration."""
|
||||
HUMAN_QID: str = "Q5"
|
||||
"""The Wikidata item ID of "human"."""
|
||||
ENSEMBLE_QID: str = "Q2088357"
|
||||
"""The Wikidata item ID of "musical ensemble"."""
|
||||
ORIGINAL_CAST_QID: str = "Q106497009"
|
||||
"""The Wikidata item ID of "original cast"."""
|
||||
GROUP_KEYWORDS: Sequence[str] = ("band", "group", "duo", "trio")
|
||||
"""The label keywords that suggest a musical ensemble, covering
|
||||
labels like "boy band" and "girl group"."""
|
||||
NOTE_NOT_FOUND: str = "not found"
|
||||
"""The note sentinel of an artist without a resolved Wikidata
|
||||
item, written to the snapshot and read back for the
|
||||
classification."""
|
||||
CORPUS_START_YEAR: int = 2016
|
||||
"""The first year of the corpus window: a member who left a group
|
||||
before it never performed a corpus song."""
|
||||
MIXED_GENDER: str = "mixed"
|
||||
"""The gender recorded for a group whose members do not share one
|
||||
gender."""
|
||||
TIME_YEAR_PATTERN: re.Pattern[str] = re.compile(r"^[+-]?\d+")
|
||||
"""The leading year of a Wikidata time value."""
|
||||
PINNED_QIDS: dict[str, str] = {}
|
||||
"""The last-resort pinned item IDs, keyed by the artist name.
|
||||
|
||||
An entry is for an artist the algorithm documented on
|
||||
``ArtistFetcher`` is structurally unable to resolve, with its
|
||||
justification recorded here. Currently empty: the only pin ever
|
||||
needed, "Pinkfong" (typed as a brand, which the type gate
|
||||
excludes by design), became moot when the store's artist entity
|
||||
behind that credit was identified as Hope Segoine.
|
||||
|
||||
A pinned name skips the candidate retrieval and corroboration
|
||||
steps; its item ID is used directly."""
|
||||
|
||||
|
||||
class ArtistType(enum.StrEnum):
|
||||
"""The decided artist type of a snapshot row."""
|
||||
@@ -147,15 +80,14 @@ class ArtistSnapshot:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
SNAPSHOT_FIELDS: Sequence[str] = tuple(
|
||||
x.name for x in fields(ArtistSnapshot))
|
||||
"""The header columns of the Wikidata artist snapshot CSV file."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class GroupMember:
|
||||
"""One has-part member of a Wikidata group item."""
|
||||
|
||||
__CORPUS_START_YEAR: ClassVar[int] = 2016
|
||||
"""The first year of the corpus window: a member who left a
|
||||
group before it never performed a corpus song."""
|
||||
|
||||
qid: str
|
||||
"""The item ID of the member."""
|
||||
start_years: list[int] = field(default_factory=list)
|
||||
@@ -181,7 +113,7 @@ class GroupMember:
|
||||
if len(self.end_years) == 0:
|
||||
return True
|
||||
last_end: int = max(self.end_years)
|
||||
if last_end >= CORPUS_START_YEAR:
|
||||
if last_end >= self.__CORPUS_START_YEAR:
|
||||
return True
|
||||
if len(self.start_years) == 0:
|
||||
return False
|
||||
@@ -229,20 +161,8 @@ class RetryExhausted(Exception):
|
||||
"""
|
||||
|
||||
|
||||
def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||
"""Parse the command-line arguments.
|
||||
|
||||
:param argv: The command-line arguments, or None for
|
||||
``sys.argv``.
|
||||
:return: The parsed arguments.
|
||||
"""
|
||||
parser: argparse.ArgumentParser = argparse.ArgumentParser(
|
||||
description="Fetch the artist metadata from Wikidata"
|
||||
" into the capture layer.")
|
||||
parser.add_argument(
|
||||
"wikidata_csv", type=Path,
|
||||
help="the Wikidata artist snapshot CSV file")
|
||||
return parser.parse_args(argv)
|
||||
NOTE_NOT_FOUND: str = "not found"
|
||||
"""The note marking an artist that could not be resolved."""
|
||||
|
||||
|
||||
class ArtistFetcher:
|
||||
@@ -291,6 +211,71 @@ class ArtistFetcher:
|
||||
in the note.
|
||||
"""
|
||||
|
||||
__API_URL: ClassVar[str] = "https://www.wikidata.org/w/api.php"
|
||||
"""The URL of the Wikidata API endpoint."""
|
||||
__SPARQL_URL: ClassVar[str] \
|
||||
= "https://query.wikidata.org/sparql"
|
||||
"""The URL of the Wikidata Query Service SPARQL endpoint."""
|
||||
__USER_AGENT: ClassVar[str] = (
|
||||
f"pop-fem-audit-tools/{VERSION}"
|
||||
" (https://github.com/imacat/pop-fem-audit;"
|
||||
" mailto:imacat@mail.imacat.idv.tw)")
|
||||
"""The User-Agent header sent on every HTTP request."""
|
||||
__TIMEOUT: ClassVar[float] = 30.0
|
||||
"""The timeout of an API HTTP request, in seconds."""
|
||||
__SPARQL_TIMEOUT: ClassVar[float] = 90.0
|
||||
"""The timeout of a SPARQL HTTP request, in seconds.
|
||||
|
||||
Higher than the API timeout: the WDQS server aborts a slow
|
||||
query at 60 seconds, and a lower client timeout would race
|
||||
that server-side abort and misclassify a slow-but-answerable
|
||||
query as a client-side timeout instead of letting the
|
||||
server's own HTTP error response arrive and enter the retry
|
||||
path."""
|
||||
__SLEEP_SECONDS: ClassVar[float] = 1.0
|
||||
"""The delay between consecutive HTTP requests, in
|
||||
seconds."""
|
||||
__MAX_ATTEMPTS: ClassVar[int] = 5
|
||||
"""The maximum number of attempts on a transient error."""
|
||||
__RETRY_SECONDS: ClassVar[float] = 15.0
|
||||
"""The back-off unit on a transient error, in seconds;
|
||||
multiplied by the attempt number already made."""
|
||||
__RETRY_STATUSES: ClassVar[frozenset[int]] \
|
||||
= frozenset({429, 500, 502, 503})
|
||||
"""The HTTP statuses that are retried with a back-off."""
|
||||
__MAX_STAGE1_TITLES: ClassVar[int] = 3
|
||||
"""The maximum number of charted titles used for the
|
||||
stage-1 song corroboration."""
|
||||
__HUMAN_QID: ClassVar[str] = "Q5"
|
||||
"""The Wikidata item ID of "human"."""
|
||||
__ENSEMBLE_QID: ClassVar[str] = "Q2088357"
|
||||
"""The Wikidata item ID of "musical ensemble"."""
|
||||
__ORIGINAL_CAST_QID: ClassVar[str] = "Q106497009"
|
||||
"""The Wikidata item ID of "original cast"."""
|
||||
__GROUP_KEYWORDS: ClassVar[Sequence[str]] \
|
||||
= ("band", "group", "duo", "trio")
|
||||
"""The label keywords that suggest a musical ensemble,
|
||||
covering labels like "boy band" and "girl group"."""
|
||||
__MIXED_GENDER: ClassVar[str] = "mixed"
|
||||
"""The gender recorded for a group whose members do not
|
||||
share one gender."""
|
||||
__TIME_YEAR_PATTERN: ClassVar[re.Pattern[str]] \
|
||||
= re.compile(r"^[+-]?\d+")
|
||||
"""The leading year of a Wikidata time value."""
|
||||
__PINNED_QIDS: ClassVar[dict[str, str]] = {}
|
||||
"""The last-resort pinned item IDs, keyed by the artist name.
|
||||
|
||||
An entry is for an artist the algorithm documented on
|
||||
``ArtistFetcher`` is structurally unable to resolve, with its
|
||||
justification recorded here. Currently empty: the only pin
|
||||
ever needed, "Pinkfong" (typed as a brand, which the type
|
||||
gate excludes by design), became moot when the store's
|
||||
artist entity behind that credit was identified as Hope
|
||||
Segoine.
|
||||
|
||||
A pinned name skips the candidate retrieval and corroboration
|
||||
steps; its item ID is used directly."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
"""Construct the fetcher."""
|
||||
self.__sent: int = 0
|
||||
@@ -342,8 +327,8 @@ class ArtistFetcher:
|
||||
transient error are exhausted.
|
||||
:raises ValueError: On a JSON decoding error.
|
||||
"""
|
||||
if name in PINNED_QIDS:
|
||||
return PINNED_QIDS[name]
|
||||
if name in self.__PINNED_QIDS:
|
||||
return self.__PINNED_QIDS[name]
|
||||
candidates: list[str] = self.__candidates(name)
|
||||
if len(candidates) == 0:
|
||||
return None
|
||||
@@ -373,11 +358,11 @@ class ArtistFetcher:
|
||||
{{ ?item rdfs:label ?name }}
|
||||
UNION {{ ?item skos:altLabel ?name }}
|
||||
{{
|
||||
?item wdt:P31 wd:{HUMAN_QID}
|
||||
?item wdt:P31 wd:{self.__HUMAN_QID}
|
||||
}} UNION {{
|
||||
?item wdt:P31/wdt:P279* wd:{ENSEMBLE_QID}
|
||||
?item wdt:P31/wdt:P279* wd:{self.__ENSEMBLE_QID}
|
||||
}} UNION {{
|
||||
?item wdt:P31 wd:{ORIGINAL_CAST_QID}
|
||||
?item wdt:P31 wd:{self.__ORIGINAL_CAST_QID}
|
||||
}}
|
||||
}}
|
||||
"""
|
||||
@@ -403,7 +388,7 @@ class ArtistFetcher:
|
||||
transient error are exhausted.
|
||||
:raises ValueError: On a JSON decoding error.
|
||||
"""
|
||||
subset: Sequence[str] = titles[:MAX_STAGE1_TITLES]
|
||||
subset: Sequence[str] = titles[:self.__MAX_STAGE1_TITLES]
|
||||
if len(subset) == 0:
|
||||
return None
|
||||
query: str = f"""
|
||||
@@ -532,7 +517,7 @@ class ArtistFetcher:
|
||||
qid: str
|
||||
for qid in qids:
|
||||
member: MemberClaims = claims.get(qid, MemberClaims())
|
||||
if HUMAN_QID not in member.instance_of_ids:
|
||||
if self.__HUMAN_QID not in member.instance_of_ids:
|
||||
continue
|
||||
if len(member.gender_ids) == 0:
|
||||
return
|
||||
@@ -543,7 +528,7 @@ class ArtistFetcher:
|
||||
[x[1] for x in genders], any_language=True)
|
||||
unique: set[str] = {x[1] for x in genders}
|
||||
snapshot.gender = labels[genders[0][1]] \
|
||||
if len(unique) == 1 else MIXED_GENDER
|
||||
if len(unique) == 1 else self.__MIXED_GENDER
|
||||
basis: str = "gender derived from members: " + "; ".join(
|
||||
f"{x} {labels[y]}" for x, y in genders)
|
||||
snapshot.note = f"{snapshot.note}; {basis}" \
|
||||
@@ -677,7 +662,8 @@ class ArtistFetcher:
|
||||
or not isinstance(value.get("time"), str):
|
||||
continue
|
||||
match: re.Match[str] | None \
|
||||
= TIME_YEAR_PATTERN.match(value["time"])
|
||||
= ArtistFetcher.__TIME_YEAR_PATTERN.match(
|
||||
value["time"])
|
||||
if match is not None:
|
||||
years.append(int(match.group()))
|
||||
return years
|
||||
@@ -806,12 +792,13 @@ class ArtistFetcher:
|
||||
``ArtistType.GROUP`` for a musical ensemble, or the
|
||||
empty string for the human to decide.
|
||||
"""
|
||||
if HUMAN_QID in type_ids:
|
||||
if ArtistFetcher.__HUMAN_QID in type_ids:
|
||||
return ArtistType.SOLO
|
||||
qid: str
|
||||
for qid in type_ids:
|
||||
label: str = labels.get(qid, "").lower()
|
||||
if any(x in label for x in GROUP_KEYWORDS):
|
||||
if any(x in label
|
||||
for x in ArtistFetcher.__GROUP_KEYWORDS):
|
||||
return ArtistType.GROUP
|
||||
return ""
|
||||
|
||||
@@ -827,14 +814,15 @@ class ArtistFetcher:
|
||||
transient error are exhausted.
|
||||
:raises ValueError: On a JSON decoding error.
|
||||
"""
|
||||
url: str = (f"{SPARQL_URL}?"
|
||||
f"{urllib.parse.urlencode({'query': query})}")
|
||||
url: str = (
|
||||
f"{self.__SPARQL_URL}?"
|
||||
f"{urllib.parse.urlencode({'query': query})}")
|
||||
request: urllib.request.Request = urllib.request.Request(
|
||||
url, headers={
|
||||
"User-Agent": USER_AGENT,
|
||||
"User-Agent": self.__USER_AGENT,
|
||||
"Accept": "application/sparql-results+json"})
|
||||
body: bytes = self.__send(
|
||||
request, timeout=SPARQL_TIMEOUT)
|
||||
request, timeout=self.__SPARQL_TIMEOUT)
|
||||
data: Any = json.loads(body)
|
||||
bindings: Any = None
|
||||
if isinstance(data, dict) \
|
||||
@@ -868,13 +856,14 @@ class ArtistFetcher:
|
||||
transient error are exhausted.
|
||||
:raises ValueError: On a JSON decoding error.
|
||||
"""
|
||||
url: str = f"{API_URL}?{urllib.parse.urlencode(params)}"
|
||||
url: str \
|
||||
= f"{self.__API_URL}?{urllib.parse.urlencode(params)}"
|
||||
request: urllib.request.Request = urllib.request.Request(
|
||||
url, headers={"User-Agent": USER_AGENT})
|
||||
url, headers={"User-Agent": self.__USER_AGENT})
|
||||
return json.loads(self.__send(request))
|
||||
|
||||
def __send(self, request: urllib.request.Request,
|
||||
timeout: float = TIMEOUT) -> bytes:
|
||||
timeout: float = __TIMEOUT) -> bytes:
|
||||
"""Send an HTTP request, retrying on a transient error.
|
||||
|
||||
Consecutive requests are separated by a fixed delay. A
|
||||
@@ -892,7 +881,7 @@ class ArtistFetcher:
|
||||
transient error are exhausted.
|
||||
"""
|
||||
if self.__sent > 0:
|
||||
time.sleep(SLEEP_SECONDS)
|
||||
time.sleep(self.__SLEEP_SECONDS)
|
||||
self.__sent += 1
|
||||
attempt: int = 1
|
||||
reason: str | None
|
||||
@@ -905,10 +894,10 @@ class ArtistFetcher:
|
||||
reason = self.__retry_reason(error)
|
||||
if reason is None:
|
||||
raise
|
||||
if attempt >= MAX_ATTEMPTS:
|
||||
if attempt >= self.__MAX_ATTEMPTS:
|
||||
raise RetryExhausted(
|
||||
f"retries exhausted ({reason})") from error
|
||||
time.sleep(RETRY_SECONDS * attempt)
|
||||
time.sleep(self.__RETRY_SECONDS * attempt)
|
||||
attempt += 1
|
||||
|
||||
@staticmethod
|
||||
@@ -923,7 +912,7 @@ class ArtistFetcher:
|
||||
the error is not transient and must not be retried.
|
||||
"""
|
||||
if isinstance(error, urllib.error.HTTPError):
|
||||
if error.code not in RETRY_STATUSES:
|
||||
if error.code not in ArtistFetcher.__RETRY_STATUSES:
|
||||
return None
|
||||
return str(error)
|
||||
if isinstance(error, TimeoutError):
|
||||
@@ -968,92 +957,216 @@ class ArtistFetcher:
|
||||
return uri.rsplit("/", 1)[-1]
|
||||
|
||||
|
||||
def read_snapshot_rows(file: TextIO) -> list[dict[str, str]]:
|
||||
"""Read the current rows of a snapshot CSV file handle.
|
||||
@dataclass(frozen=True)
|
||||
class FetchCounts:
|
||||
"""The outcome counts of one snapshot update run."""
|
||||
|
||||
:param file: The open, seekable snapshot CSV file.
|
||||
:return: The rows, keyed by the column name.
|
||||
:raises OSError: When the file cannot be read.
|
||||
fetched: int
|
||||
"""The number of artists newly resolved."""
|
||||
not_found: int
|
||||
"""The number of artists left unresolved."""
|
||||
errors: int
|
||||
"""The number of artists that ended in an error."""
|
||||
|
||||
|
||||
class ArtistSnapshotUpdater:
|
||||
"""The updater of the Wikidata artist snapshot CSV file.
|
||||
|
||||
Fetches the metadata of every artist of the working store
|
||||
that the snapshot does not resolve yet, appends a row for
|
||||
each to the snapshot as it is fetched, and rewrites the
|
||||
snapshot sorted by artist name with its stale rows dropped.
|
||||
"""
|
||||
file.seek(0)
|
||||
reader: csv.DictReader[str] = csv.DictReader(file)
|
||||
return list(reader)
|
||||
|
||||
__SNAPSHOT_FIELDS: ClassVar[Sequence[str]] = tuple(
|
||||
x.name for x in fields(ArtistSnapshot))
|
||||
"""The header columns of the Wikidata artist snapshot CSV file."""
|
||||
|
||||
def read_artist_titles(session: Session,
|
||||
artist_id: int) -> list[str]:
|
||||
"""Read the charted song titles credited to an artist.
|
||||
def __init__(self, wikidata_csv: Path) -> None:
|
||||
"""Set up the updater.
|
||||
|
||||
:param session: The database session.
|
||||
:param artist_id: The artist ID.
|
||||
:return: The song titles credited to the artist, ordered by
|
||||
the song ID, with the duplicate titles removed.
|
||||
"""
|
||||
titles: Sequence[str] = session.scalars(
|
||||
sa.select(Song.title)
|
||||
.join(SongArtist, SongArtist.song_id == Song.id)
|
||||
.where(SongArtist.artist_id == artist_id)
|
||||
.order_by(Song.id)).all()
|
||||
return list(dict.fromkeys(titles))
|
||||
:param wikidata_csv: The Wikidata artist snapshot CSV
|
||||
file.
|
||||
"""
|
||||
self.__wikidata_csv: Path = wikidata_csv
|
||||
"""The Wikidata artist snapshot CSV file."""
|
||||
|
||||
def run(self) -> FetchCounts:
|
||||
"""Fetch every unresolved artist and update the snapshot.
|
||||
|
||||
def ensure_snapshot_header(file: TextIO) -> None:
|
||||
"""Write the snapshot CSV header row if the file is empty.
|
||||
:return: The counts of the run.
|
||||
:raises OSError: When the snapshot file, or its parent
|
||||
directory, cannot be read or written.
|
||||
:raises sqlalchemy.exc.SQLAlchemyError: When the working
|
||||
store cannot be read.
|
||||
"""
|
||||
session: Session = ds.get_db()
|
||||
try:
|
||||
return self.__run(session)
|
||||
finally:
|
||||
session.close()
|
||||
|
||||
:param file: The open, seekable snapshot CSV file.
|
||||
:return: None.
|
||||
:raises OSError: When the file cannot be written.
|
||||
"""
|
||||
file.seek(0, os.SEEK_END)
|
||||
if file.tell() == 0:
|
||||
csv.writer(file).writerow(SNAPSHOT_FIELDS)
|
||||
def __run(self, session: Session) -> FetchCounts:
|
||||
"""Run the fetch loop with an open database session.
|
||||
|
||||
:param session: The database session.
|
||||
:return: The counts of the run.
|
||||
:raises OSError: When the snapshot file, or its parent
|
||||
directory, cannot be read or written.
|
||||
"""
|
||||
fetcher: ArtistFetcher = ArtistFetcher()
|
||||
fetched: int = 0
|
||||
not_found: int = 0
|
||||
errors: int = 0
|
||||
self.__wikidata_csv.parent.mkdir(
|
||||
parents=True, exist_ok=True)
|
||||
with open(self.__wikidata_csv, "a+", encoding="utf-8",
|
||||
newline="") as csv_file:
|
||||
done: set[str] = {
|
||||
x["name"] for x in
|
||||
self.__read_snapshot_rows(csv_file)
|
||||
if x["gender"] != ""}
|
||||
self.__ensure_snapshot_header(csv_file)
|
||||
names: set[str] = set()
|
||||
artist: Artist
|
||||
for artist in session.scalars(
|
||||
sa.select(Artist).order_by(Artist.id)):
|
||||
names.add(artist.name)
|
||||
if artist.name in done:
|
||||
continue
|
||||
titles: list[str] = self.__read_artist_titles(
|
||||
session, artist.id)
|
||||
snapshot: ArtistSnapshot = fetcher.fetch(
|
||||
artist.name, titles)
|
||||
self.__append_row(csv_file, snapshot)
|
||||
status: str = snapshot.qid
|
||||
if snapshot.note == NOTE_NOT_FOUND:
|
||||
not_found += 1
|
||||
status = NOTE_NOT_FOUND
|
||||
elif snapshot.note.startswith("error: "):
|
||||
errors += 1
|
||||
status = snapshot.note
|
||||
else:
|
||||
fetched += 1
|
||||
print(f"artist \"{artist.name}\": {status}",
|
||||
file=sys.stderr)
|
||||
self.__write_snapshot(csv_file, names)
|
||||
return FetchCounts(
|
||||
fetched=fetched, not_found=not_found, errors=errors)
|
||||
|
||||
@staticmethod
|
||||
def __read_snapshot_rows(file: TextIO) \
|
||||
-> list[dict[str, str]]:
|
||||
"""Read the current rows of a snapshot CSV file handle.
|
||||
|
||||
:param file: The open, seekable snapshot CSV file.
|
||||
:return: The rows, keyed by the column name.
|
||||
:raises OSError: When the file cannot be read.
|
||||
"""
|
||||
file.seek(0)
|
||||
reader: csv.DictReader[str] = csv.DictReader(file)
|
||||
return list(reader)
|
||||
|
||||
@staticmethod
|
||||
def __read_artist_titles(session: Session,
|
||||
artist_id: int) -> list[str]:
|
||||
"""Read the charted song titles credited to an artist.
|
||||
|
||||
:param session: The database session.
|
||||
:param artist_id: The artist ID.
|
||||
:return: The song titles credited to the artist, ordered
|
||||
by the song ID, with the duplicate titles removed.
|
||||
"""
|
||||
titles: Sequence[str] = session.scalars(
|
||||
sa.select(Song.title)
|
||||
.join(SongArtist, SongArtist.song_id == Song.id)
|
||||
.where(SongArtist.artist_id == artist_id)
|
||||
.order_by(Song.id)).all()
|
||||
return list(dict.fromkeys(titles))
|
||||
|
||||
@staticmethod
|
||||
def __ensure_snapshot_header(file: TextIO) -> None:
|
||||
"""Write the snapshot CSV header row if the file is
|
||||
empty.
|
||||
|
||||
:param file: The open, seekable snapshot CSV file.
|
||||
:return: None.
|
||||
:raises OSError: When the file cannot be written.
|
||||
"""
|
||||
file.seek(0, os.SEEK_END)
|
||||
if file.tell() == 0:
|
||||
csv.writer(file).writerow(
|
||||
ArtistSnapshotUpdater.__SNAPSHOT_FIELDS)
|
||||
file.flush()
|
||||
|
||||
@staticmethod
|
||||
def __append_row(file: TextIO,
|
||||
snapshot: ArtistSnapshot) -> None:
|
||||
"""Append a snapshot row to a snapshot CSV file handle.
|
||||
|
||||
:param file: The open snapshot CSV file, opened for
|
||||
append.
|
||||
:param snapshot: The snapshot of an artist.
|
||||
:return: None.
|
||||
:raises OSError: When the file cannot be written.
|
||||
"""
|
||||
csv.DictWriter(
|
||||
file, ArtistSnapshotUpdater.__SNAPSHOT_FIELDS).writerow(
|
||||
snapshot.to_row())
|
||||
file.flush()
|
||||
|
||||
@staticmethod
|
||||
def __write_snapshot(file: TextIO, names: Container[str]) \
|
||||
-> None:
|
||||
"""Rewrite a snapshot CSV file handle sorted by artist
|
||||
name.
|
||||
|
||||
def append_row(file: TextIO, snapshot: ArtistSnapshot) -> None:
|
||||
"""Append a snapshot row to a snapshot CSV file handle.
|
||||
The rows are ordered by the case-folded artist name,
|
||||
matching the convention of the derived ``artists.csv``.
|
||||
An artist keeps one row only, the last one of the file,
|
||||
so that a re-fetched artist replaces its earlier row. A
|
||||
row whose name is not an artist of the store is dropped
|
||||
and reported on the standard error.
|
||||
|
||||
:param file: The open snapshot CSV file, opened for append.
|
||||
:param snapshot: The snapshot of an artist.
|
||||
:return: None.
|
||||
:raises OSError: When the file cannot be written.
|
||||
:param file: The open, seekable snapshot CSV file.
|
||||
:param names: The artist names of the working store.
|
||||
:return: None.
|
||||
:raises OSError: When the file cannot be read or written.
|
||||
"""
|
||||
kept: dict[str, dict[str, str]] = {}
|
||||
row: dict[str, str]
|
||||
for row in ArtistSnapshotUpdater.__read_snapshot_rows(
|
||||
file):
|
||||
if row["name"] not in names:
|
||||
print(f"dropped stale row \"{row['name']}\":"
|
||||
" no such artist in the store",
|
||||
file=sys.stderr)
|
||||
continue
|
||||
kept[row["name"]] = row
|
||||
ordered: list[dict[str, str]] = sorted(
|
||||
kept.values(), key=lambda x: x["name"].casefold())
|
||||
file.seek(0)
|
||||
file.truncate()
|
||||
writer: csv.DictWriter[str] = csv.DictWriter(
|
||||
file, ArtistSnapshotUpdater.__SNAPSHOT_FIELDS)
|
||||
writer.writeheader()
|
||||
writer.writerows(ordered)
|
||||
|
||||
|
||||
def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||
"""Parse the command-line arguments.
|
||||
|
||||
:param argv: The command-line arguments, or None for
|
||||
``sys.argv``.
|
||||
:return: The parsed arguments.
|
||||
"""
|
||||
csv.DictWriter(file, SNAPSHOT_FIELDS).writerow(
|
||||
snapshot.to_row())
|
||||
file.flush()
|
||||
|
||||
|
||||
def write_snapshot(file: TextIO, names: Container[str]) -> None:
|
||||
"""Rewrite a snapshot CSV file handle sorted by artist name.
|
||||
|
||||
The rows are ordered by the case-folded artist name, matching
|
||||
the convention of the derived ``artists.csv``. An artist
|
||||
keeps one row only, the last one of the file, so that a
|
||||
re-fetched artist replaces its earlier row. A row whose name
|
||||
is not an artist of the store is dropped and reported on the
|
||||
standard error.
|
||||
|
||||
:param file: The open, seekable snapshot CSV file.
|
||||
:param names: The artist names of the working store.
|
||||
:return: None.
|
||||
:raises OSError: When the file cannot be read or written.
|
||||
"""
|
||||
kept: dict[str, dict[str, str]] = {}
|
||||
row: dict[str, str]
|
||||
for row in read_snapshot_rows(file):
|
||||
if row["name"] not in names:
|
||||
print(f"dropped stale row \"{row['name']}\":"
|
||||
" no such artist in the store", file=sys.stderr)
|
||||
continue
|
||||
kept[row["name"]] = row
|
||||
ordered: list[dict[str, str]] = sorted(
|
||||
kept.values(), key=lambda x: x["name"].casefold())
|
||||
file.seek(0)
|
||||
file.truncate()
|
||||
writer: csv.DictWriter[str] = csv.DictWriter(
|
||||
file, SNAPSHOT_FIELDS)
|
||||
writer.writeheader()
|
||||
writer.writerows(ordered)
|
||||
parser: argparse.ArgumentParser = argparse.ArgumentParser(
|
||||
description="Fetch the artist metadata from Wikidata"
|
||||
" into the capture layer.")
|
||||
parser.add_argument(
|
||||
"wikidata_csv", type=Path,
|
||||
help="the Wikidata artist snapshot CSV file")
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
@@ -1066,52 +1179,16 @@ def main(argv: list[str] | None = None) -> int:
|
||||
"""
|
||||
started: float = time.monotonic()
|
||||
args: argparse.Namespace = parse_args(argv)
|
||||
fetcher: ArtistFetcher = ArtistFetcher()
|
||||
fetched: int = 0
|
||||
not_found: int = 0
|
||||
errors: int = 0
|
||||
session: Session = ds.get_db()
|
||||
try:
|
||||
args.wikidata_csv.parent.mkdir(
|
||||
parents=True, exist_ok=True)
|
||||
with open(args.wikidata_csv, "a+", encoding="utf-8",
|
||||
newline="") as csv_file:
|
||||
done: set[str] = {x["name"] for x in
|
||||
read_snapshot_rows(csv_file)
|
||||
if x["gender"] != ""}
|
||||
ensure_snapshot_header(csv_file)
|
||||
names: set[str] = set()
|
||||
artist: Artist
|
||||
for artist in session.scalars(
|
||||
sa.select(Artist).order_by(Artist.id)):
|
||||
names.add(artist.name)
|
||||
if artist.name in done:
|
||||
continue
|
||||
titles: list[str] = read_artist_titles(
|
||||
session, artist.id)
|
||||
snapshot: ArtistSnapshot = fetcher.fetch(
|
||||
artist.name, titles)
|
||||
append_row(csv_file, snapshot)
|
||||
status: str = snapshot.qid
|
||||
if snapshot.note == NOTE_NOT_FOUND:
|
||||
not_found += 1
|
||||
status = "not found"
|
||||
elif snapshot.note.startswith("error: "):
|
||||
errors += 1
|
||||
status = snapshot.note
|
||||
else:
|
||||
fetched += 1
|
||||
print(f"artist \"{artist.name}\": {status}",
|
||||
file=sys.stderr)
|
||||
write_snapshot(csv_file, names)
|
||||
counts: FetchCounts \
|
||||
= ArtistSnapshotUpdater(args.wikidata_csv).run()
|
||||
except (OSError, sa.exc.SQLAlchemyError) as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 1
|
||||
finally:
|
||||
session.close()
|
||||
attempted: int = fetched + not_found + errors
|
||||
attempted: int = counts.fetched + counts.not_found \
|
||||
+ counts.errors
|
||||
elapsed: str = format_duration(time.monotonic() - started)
|
||||
print(f"Done. Resolved {fetched}/{attempted} artists."
|
||||
f" {elapsed} elapsed.",
|
||||
print(f"Done. Resolved {counts.fetched}/{attempted}"
|
||||
f" artists. {elapsed} elapsed.",
|
||||
file=sys.stderr)
|
||||
return 0
|
||||
|
||||
@@ -30,8 +30,9 @@ import time
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from typing import Any, ClassVar
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -45,60 +46,6 @@ from ..models import (
|
||||
)
|
||||
from ..utils import format_duration
|
||||
|
||||
PROVENANCE_FIELDS: Sequence[str] = (
|
||||
"song_id", "source", "method", "acquired_at", "note")
|
||||
"""The header columns of the lyrics provenance CSV file."""
|
||||
USER_AGENT: str = ("pop-fem-audit-tools"
|
||||
" (https://github.com/imacat/pop-fem-audit)")
|
||||
"""The User-Agent header sent on every HTTP request."""
|
||||
TIMEOUT: float = 30.0
|
||||
"""The timeout of an HTTP request, in seconds."""
|
||||
SLEEP_SECONDS: float = 1.0
|
||||
"""The delay between consecutive HTTP requests, in seconds."""
|
||||
|
||||
|
||||
def __build_normalization() -> dict[int, str | None]:
|
||||
"""Build the lyrics normalization translation table.
|
||||
|
||||
:return: The codepoint-to-replacement mapping, a replacement
|
||||
of None meaning removal.
|
||||
"""
|
||||
table: dict[int, str | None] = {}
|
||||
codepoint: int
|
||||
for codepoint in range(0x80, 0xa0):
|
||||
try:
|
||||
table[codepoint] = bytes([codepoint]).decode("cp1252")
|
||||
except UnicodeDecodeError:
|
||||
table[codepoint] = None
|
||||
table[0x0435] = "e"
|
||||
table[0x03cc] = "ó"
|
||||
for codepoint in (0x2005, 0x205f, 0x200a):
|
||||
table[codepoint] = " "
|
||||
for codepoint in (0x200b, 0x200c, 0x200d, 0xfeff):
|
||||
table[codepoint] = None
|
||||
return table
|
||||
|
||||
|
||||
NORMALIZATION: dict[int, str | None] = __build_normalization()
|
||||
"""The codepoint-to-replacement mapping applied to fetched
|
||||
lyrics: cp1252-mojibake restoration for U+0080-U+009F (with the
|
||||
five byte values undefined in cp1252 removed), homoglyph
|
||||
restoration for the Cyrillic "e" and the Greek "o" with tonos,
|
||||
ASCII-space restoration for exotic space variants, and removal
|
||||
of zero-width characters. A replacement of None removes the
|
||||
codepoint."""
|
||||
|
||||
|
||||
def normalize_lyrics(text: str) -> str:
|
||||
"""Restore or remove watermark and mojibake characters.
|
||||
|
||||
:param text: The lyrics text as fetched from an API.
|
||||
:return: The text with the codepoints in
|
||||
:data:`NORMALIZATION` replaced or removed; every other
|
||||
character is unchanged.
|
||||
"""
|
||||
return text.translate(NORMALIZATION)
|
||||
|
||||
|
||||
def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||
"""Parse the command-line arguments.
|
||||
@@ -122,6 +69,16 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace:
|
||||
class LyricsFetcher:
|
||||
"""A fetcher of song lyrics from the public lyrics APIs."""
|
||||
|
||||
__USER_AGENT: ClassVar[str] = (
|
||||
"pop-fem-audit-tools"
|
||||
" (https://github.com/imacat/pop-fem-audit)")
|
||||
"""The User-Agent header sent on every HTTP request."""
|
||||
__TIMEOUT: ClassVar[float] = 30.0
|
||||
"""The timeout of an HTTP request, in seconds."""
|
||||
__SLEEP_SECONDS: ClassVar[float] = 1.0
|
||||
"""The delay between consecutive HTTP requests, in
|
||||
seconds."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
"""Construct the fetcher."""
|
||||
self.__sent: int = 0
|
||||
@@ -193,78 +150,216 @@ class LyricsFetcher:
|
||||
network, or decoding error.
|
||||
"""
|
||||
if self.__sent > 0:
|
||||
time.sleep(SLEEP_SECONDS)
|
||||
time.sleep(self.__SLEEP_SECONDS)
|
||||
self.__sent += 1
|
||||
request: urllib.request.Request = urllib.request.Request(
|
||||
url, headers={"User-Agent": USER_AGENT})
|
||||
url, headers={"User-Agent": self.__USER_AGENT})
|
||||
try:
|
||||
with urllib.request.urlopen(
|
||||
request, timeout=TIMEOUT) as response:
|
||||
request, timeout=self.__TIMEOUT) as response:
|
||||
return json.load(response)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def query_artist(session: Session, song_id: int) -> str:
|
||||
"""Find the artist name to query the APIs with.
|
||||
@dataclass(frozen=True)
|
||||
class LyricsFetchCounts:
|
||||
"""The outcome of one run of fetching the missing lyrics."""
|
||||
|
||||
:param session: The database session.
|
||||
:param song_id: The song ID.
|
||||
:return: The name of the primary-role artist with the lowest
|
||||
position.
|
||||
"""
|
||||
name: str | None = session.scalar(
|
||||
sa.select(Artist.name)
|
||||
.join(SongArtist, SongArtist.artist_id == Artist.id)
|
||||
.where(SongArtist.song_id == song_id,
|
||||
SongArtist.role == Role.PRIMARY)
|
||||
.order_by(SongArtist.position)
|
||||
.limit(1))
|
||||
assert name is not None
|
||||
return name
|
||||
fetched: int
|
||||
"""The number of songs newly fetched."""
|
||||
missed: int
|
||||
"""The number of songs every API missed."""
|
||||
|
||||
|
||||
def save_lyrics(lyrics_dir: Path, song_id: int,
|
||||
lyrics: str) -> None:
|
||||
"""Write the lyrics of a song into the cache directory.
|
||||
class LyricsFetchRunner:
|
||||
"""The orchestrator of one run of fetching missing lyrics."""
|
||||
|
||||
The cache directory is created when missing.
|
||||
__PROVENANCE_FIELDS: ClassVar[Sequence[str]] = (
|
||||
"song_id", "source", "method", "acquired_at", "note")
|
||||
"""The header columns of the lyrics provenance CSV file."""
|
||||
|
||||
The lyrics text is normalized with :func:`normalize_lyrics`
|
||||
before being written.
|
||||
@staticmethod
|
||||
def __build_normalization() -> dict[int, str | None]:
|
||||
"""Build the lyrics normalization translation table.
|
||||
|
||||
:param lyrics_dir: The lyrics cache directory.
|
||||
:param song_id: The song ID.
|
||||
:param lyrics: The lyrics text.
|
||||
:return: None.
|
||||
:raises OSError: When the file cannot be written.
|
||||
"""
|
||||
lyrics_dir.mkdir(parents=True, exist_ok=True)
|
||||
(lyrics_dir / f"{song_id}.txt").write_text(
|
||||
normalize_lyrics(lyrics), encoding="utf-8")
|
||||
:return: The codepoint-to-replacement mapping, a
|
||||
replacement of None meaning removal.
|
||||
"""
|
||||
table: dict[int, str | None] = {}
|
||||
codepoint: int
|
||||
for codepoint in range(0x80, 0xa0):
|
||||
try:
|
||||
table[codepoint] = bytes(
|
||||
[codepoint]).decode("cp1252")
|
||||
except UnicodeDecodeError:
|
||||
table[codepoint] = None
|
||||
table[0x0435] = "e"
|
||||
table[0x03cc] = "ó"
|
||||
for codepoint in (0x2005, 0x205f, 0x200a):
|
||||
table[codepoint] = " "
|
||||
for codepoint in (0x200b, 0x200c, 0x200d, 0xfeff):
|
||||
table[codepoint] = None
|
||||
return table
|
||||
|
||||
__NORMALIZATION: ClassVar[dict[int, str | None]] \
|
||||
= __build_normalization()
|
||||
"""The codepoint-to-replacement mapping applied to fetched
|
||||
lyrics: cp1252-mojibake restoration for U+0080-U+009F (with
|
||||
the five byte values undefined in cp1252 removed), homoglyph
|
||||
restoration for the Cyrillic "e" and the Greek "o" with
|
||||
tonos, ASCII-space restoration for exotic space variants, and
|
||||
removal of zero-width characters. A replacement of None
|
||||
removes the codepoint."""
|
||||
|
||||
def append_provenance(path: Path, song_id: int,
|
||||
source: str) -> None:
|
||||
"""Append a provenance row for a fetched lyrics file.
|
||||
def __init__(self, lyrics_dir: Path,
|
||||
provenance_csv: Path) -> None:
|
||||
"""Set up the fetch run.
|
||||
|
||||
The CSV file is created with the header row when missing.
|
||||
:param lyrics_dir: The lyrics cache directory.
|
||||
:param provenance_csv: The lyrics provenance CSV file.
|
||||
"""
|
||||
self.__lyrics_dir: Path = lyrics_dir
|
||||
"""The lyrics cache directory."""
|
||||
self.__provenance_csv: Path = provenance_csv
|
||||
"""The lyrics provenance CSV file."""
|
||||
self.__fetcher: LyricsFetcher = LyricsFetcher()
|
||||
"""The fetcher of the public lyrics APIs."""
|
||||
|
||||
:param path: The lyrics provenance CSV file.
|
||||
:param song_id: The song ID.
|
||||
:param source: The source name of the fetched lyrics.
|
||||
:return: None.
|
||||
:raises OSError: When the file cannot be written.
|
||||
"""
|
||||
is_new: bool = not path.exists()
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(path, "a", encoding="utf-8",
|
||||
newline="") as file:
|
||||
writer: Any = csv.writer(file)
|
||||
if is_new:
|
||||
writer.writerow(PROVENANCE_FIELDS)
|
||||
writer.writerow([song_id, source, "api-fetch",
|
||||
datetime.date.today().isoformat(), ""])
|
||||
def run(self) -> LyricsFetchCounts:
|
||||
"""Fetch the missing lyrics of every song in the store.
|
||||
|
||||
Every song fetched or missed is reported on the standard
|
||||
error as an observable side effect.
|
||||
|
||||
:return: The number of songs fetched and missed.
|
||||
:raises OSError: When a cache file or the provenance CSV
|
||||
cannot be written.
|
||||
:raises sqlalchemy.exc.SQLAlchemyError: On a database
|
||||
error.
|
||||
"""
|
||||
fetched: int = 0
|
||||
missed: int = 0
|
||||
session: Session = ds.get_db()
|
||||
try:
|
||||
song: Song
|
||||
for song in session.scalars(
|
||||
sa.select(Song).order_by(Song.id)):
|
||||
if (self.__lyrics_dir
|
||||
/ f"{song.id}.txt").exists():
|
||||
continue
|
||||
if self.__fetch_one(session, song):
|
||||
fetched += 1
|
||||
else:
|
||||
missed += 1
|
||||
finally:
|
||||
session.close()
|
||||
return LyricsFetchCounts(fetched=fetched, missed=missed)
|
||||
|
||||
def __fetch_one(self, session: Session, song: Song) -> bool:
|
||||
"""Fetch and save the lyrics of one song.
|
||||
|
||||
The song is queried by its primary-role artist name; when
|
||||
every API misses and the song's full artist credit
|
||||
differs from that name, the same APIs are queried again
|
||||
with the artist credit.
|
||||
|
||||
:param session: The database session.
|
||||
:param song: The song to fetch.
|
||||
:return: True when a lyrics text was fetched and saved,
|
||||
False when every API missed on both queries.
|
||||
:raises OSError: When the cache file or the provenance
|
||||
CSV cannot be written.
|
||||
"""
|
||||
artist: str = self.__query_artist(session, song.id)
|
||||
result: tuple[str, str] | None = self.__fetcher.fetch(
|
||||
artist, song.title)
|
||||
if result is None and song.artist_credit != artist:
|
||||
result = self.__fetcher.fetch(
|
||||
song.artist_credit, song.title)
|
||||
if result is None:
|
||||
print(f"song {song.id} \"{song.title}\": miss",
|
||||
file=sys.stderr)
|
||||
return False
|
||||
lyrics: str
|
||||
source: str
|
||||
lyrics, source = result
|
||||
self.__save_lyrics(song.id, lyrics)
|
||||
self.__append_provenance(song.id, source)
|
||||
print(f"song {song.id} \"{song.title}\": {source}",
|
||||
file=sys.stderr)
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def __query_artist(session: Session, song_id: int) -> str:
|
||||
"""Find the artist name to query the APIs with.
|
||||
|
||||
:param session: The database session.
|
||||
:param song_id: The song ID.
|
||||
:return: The name of the primary-role artist with the
|
||||
lowest position.
|
||||
"""
|
||||
name: str | None = session.scalar(
|
||||
sa.select(Artist.name)
|
||||
.join(SongArtist, SongArtist.artist_id == Artist.id)
|
||||
.where(SongArtist.song_id == song_id,
|
||||
SongArtist.role == Role.PRIMARY)
|
||||
.order_by(SongArtist.position)
|
||||
.limit(1))
|
||||
assert name is not None
|
||||
return name
|
||||
|
||||
def __save_lyrics(self, song_id: int, lyrics: str) -> None:
|
||||
"""Write the lyrics of a song into the cache directory.
|
||||
|
||||
The cache directory is created when missing.
|
||||
|
||||
The lyrics text is normalized with
|
||||
:meth:`normalize_lyrics` before being written.
|
||||
|
||||
:param song_id: The song ID.
|
||||
:param lyrics: The lyrics text.
|
||||
:return: None.
|
||||
:raises OSError: When the file cannot be written.
|
||||
"""
|
||||
self.__lyrics_dir.mkdir(parents=True, exist_ok=True)
|
||||
(self.__lyrics_dir / f"{song_id}.txt").write_text(
|
||||
self.normalize_lyrics(lyrics), encoding="utf-8")
|
||||
|
||||
def __append_provenance(self, song_id: int,
|
||||
source: str) -> None:
|
||||
"""Append a provenance row for a fetched lyrics file.
|
||||
|
||||
The CSV file is created with the header row when
|
||||
missing.
|
||||
|
||||
:param song_id: The song ID.
|
||||
:param source: The source name of the fetched lyrics.
|
||||
:return: None.
|
||||
:raises OSError: When the file cannot be written.
|
||||
"""
|
||||
is_new: bool = not self.__provenance_csv.exists()
|
||||
self.__provenance_csv.parent.mkdir(
|
||||
parents=True, exist_ok=True)
|
||||
with open(self.__provenance_csv, "a", encoding="utf-8",
|
||||
newline="") as file:
|
||||
writer: Any = csv.writer(file)
|
||||
if is_new:
|
||||
writer.writerow(self.__PROVENANCE_FIELDS)
|
||||
writer.writerow(
|
||||
[song_id, source, "api-fetch",
|
||||
datetime.date.today().isoformat(), ""])
|
||||
|
||||
@classmethod
|
||||
def normalize_lyrics(cls, text: str) -> str:
|
||||
"""Restore or remove watermark and mojibake characters.
|
||||
|
||||
:param text: The lyrics text as fetched from an API.
|
||||
:return: The text with the codepoints of the
|
||||
normalization table replaced or removed; every other
|
||||
character is unchanged.
|
||||
"""
|
||||
return text.translate(cls.__NORMALIZATION)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
@@ -277,44 +372,15 @@ def main(argv: list[str] | None = None) -> int:
|
||||
"""
|
||||
started: float = time.monotonic()
|
||||
args: argparse.Namespace = parse_args(argv)
|
||||
fetcher: LyricsFetcher = LyricsFetcher()
|
||||
fetched: int = 0
|
||||
missed: int = 0
|
||||
session: Session = ds.get_db()
|
||||
try:
|
||||
song: Song
|
||||
for song in session.scalars(
|
||||
sa.select(Song).order_by(Song.id)):
|
||||
if (args.lyrics_dir / f"{song.id}.txt").exists():
|
||||
continue
|
||||
artist: str = query_artist(session, song.id)
|
||||
result: tuple[str, str] | None = fetcher.fetch(
|
||||
artist, song.title)
|
||||
if result is None and song.artist_credit != artist:
|
||||
result = fetcher.fetch(
|
||||
song.artist_credit, song.title)
|
||||
if result is None:
|
||||
missed += 1
|
||||
print(f"song {song.id} \"{song.title}\": miss",
|
||||
file=sys.stderr)
|
||||
continue
|
||||
lyrics: str
|
||||
source: str
|
||||
lyrics, source = result
|
||||
save_lyrics(args.lyrics_dir, song.id, lyrics)
|
||||
append_provenance(args.provenance_csv, song.id,
|
||||
source)
|
||||
fetched += 1
|
||||
print(f"song {song.id} \"{song.title}\": {source}",
|
||||
file=sys.stderr)
|
||||
counts: LyricsFetchCounts = LyricsFetchRunner(
|
||||
args.lyrics_dir, args.provenance_csv).run()
|
||||
except (OSError, sa.exc.SQLAlchemyError) as error:
|
||||
print(f"error: {error}", file=sys.stderr)
|
||||
return 1
|
||||
finally:
|
||||
session.close()
|
||||
attempted: int = fetched + missed
|
||||
attempted: int = counts.fetched + counts.missed
|
||||
elapsed: str = format_duration(time.monotonic() - started)
|
||||
print(f"Done. Fetched lyrics for {fetched}/{attempted}"
|
||||
f" songs. {elapsed} elapsed.",
|
||||
print(f"Done. Fetched lyrics for {counts.fetched}/"
|
||||
f"{attempted} songs. {elapsed} elapsed.",
|
||||
file=sys.stderr)
|
||||
return 0
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user