I asked for a second rater. I got one plus two LLMs. Here is what the detector check looked like on fresh data.
A few weeks ago I published a check on the objective-projection dataset's applied_rules field: I hand-labeled 120 Turkish scenes blind, compared them to the rule-based detector, and reported that the detector is fine for surface features and unreliable for features that require inference — with the materialized-metaphor rule scoring κ ≈ 0.00, no better than chance. I ended by asking for a second rater.
This is the follow-up. The headline: the unflattering result held up, and it held up against larger models too.
What I did differently
Three changes from the first study.
Fresh scenes. I drew 100 Turkish scenes the detector had never been evaluated on sampled (seed-fixed) from the 180 scenes left over after the first study's 120 were removed. Zero overlap. This matters: the detector's emotion-rule was revised between studies, and you cannot honestly test a revision on the same data you used to design it.
A genuinely independent human rater. Not me this time. Someone who is not a doctrine expert, who saw only the six definitions and the masked scene texts not the detector output, not my expectations. That removes the conflict of interest that weakened the first study, where I was both the tool's author and its only rater.
Two LLMs as additional raters. I gave Gemini 2.5 Flash and Grok the exact same definitions the human got, with no hint of how the detector works, and had them label all 100 scenes. The point was not "which model is best." It was: when a feature requires reading for meaning, does a large model do better than a regex or does it hit the same wall?
The result
Against the independent human rater, here is the picture for the rules that require inference:
Materialized metaphor. Human marked 9 scenes positive. The detector marked 72 it fires on "did a physical-parameter word appear," not on the literary move itself, so it massively over-flags (κ ≈ 0.02, replicating the first study). Gemini caught 1 of the 9. Grok caught 0. Two capable language models, given a clean definition, did not recover the construct either. Atmosphere contradiction. Human marked 44 positive. The detector caught 0, Gemini 2, Grok 3. All three missed the large majority.
For surface features the story is different but, on this sample, uninformative. Temporal anchor showed high agreement everywhere but the human marked it present in 99 of 100 scenes, because the corpus scenes overwhelmingly open with an explicit clock reference ("eleven eighteen," "twenty-three thirty-two"). When a feature is near-universal and unmissable by design, high agreement reflects corpus construction, not detector skill. That rule was effectively not tested. The same skew affects four of the six rules; only atmosphere contradiction had a balanced class split.
How to read this and how not to
The tempting reading is "even the LLMs failed, so these features are beyond machines." I want to be careful here, for two reasons.
First, a disclosure: I am not a neutral party on the question of whether language models can judge writing. Neither, in a sense, are the two models I used as raters. The interpretation has to come from the numbers, not from any model's authority — including the models doing the rating.
Second, the data supports two readings at once, and this study cannot separate them. (a) These features genuinely require inference that current automatic raters regex or LLM cannot do at human level. (b) The definitions are not operational enough, so different raters apply them inconsistently. The evidence for (b) is real: the two LLMs disagreed not just with the human but with each other on micro-focus, Gemini marked 9 positive, Grok 82. If the construct were crisp, they would have converged. Telling (a) from (b) needs a second independent human rater; with only one, I can report that machines did not match a human here, but not yet why.
The honest takeaway
A rule-based proxy is adequate for surface lexical features and unreliable for inferential ones. On fresh data, two frontier LLMs did not close that gap — but whether that is about the limits of machines or the looseness of the definitions is not yet settled. That distinction is the next experiment, not a conclusion I can draw today.
There is a connection here to something I have called summarization bias the tendency of language models to collapse a narrative's physical, shown layer back into an abstract emotional label, and (the sharper, still-untested form) to do the same when evaluating text. The pattern in this study — models stumbling exactly at the boundary between the physical/shown and the inferred — is consistent with that hypothesis. Consistent is not the same as confirmed. It remains a hypothesis I have not run a clean experiment on, and the conflict of interest above applies doubly to it.
What Is Summarization Bias?
Within the framework of "Objective Projection" and Narrative Engineering developed by Levent Bulut, it refers to a structural flaw in AI-generated narratives.
The full numbers — per-rule κ, confusion counts, the corpus-skew analysis are in the findings report that travels with this work. If you want to be the second human rater, the data and definitions are open (HF DOI 10.57967/hf/8960, Zenodo 10.5281/zenodo.19511369). Disagreeing with the labels is still the most useful thing you could do to this study. The dataset lives at huggingface.co/datasets/leventbulut/objective-projection The full methodology archive is at leventbulut.com.