The problem
Automated fake-news detection looks, on paper, like a solved problem. Fine-tune a transformer on a standard benchmark and you will comfortably clear 95% accuracy; RoBERTa has been reported at 99.99% on ISOT. Those benchmarks share an assumption that has quietly stopped being true: that the disinformation being detected was written by a human being. ISOT, FakeNewsNet and LIAR were all assembled before large language models existed.
That matters because the linguistic cues these detectors rely on are specific to human writing. Prior work established that human-written fake news carries excess first-person and second-person pronouns, hedging, subjectivity and simpler, more repetitive prose. An LLM asked to write a plausible news article produces none of those tells. So the question is whether veracity detectors have learned generic signals of deception, or simply the stylistic fingerprint of the particular authors in their training set.
Building a dataset that could answer it
Answering it requires varying authorship while holding the task constant, and no existing dataset does that. So the study pairs ISOT, whose real and fake articles are both human-authored (2016 to 2017), with GNews-Gemini, a new dataset built for this purpose: real articles collected through the GNews API (2021 to 2024), and fake counterparts generated from them by prompting Gemini 2.0 Flash Lite.
The fakes were not produced with a single "write something false" prompt. Each real article was rewritten under one of five explicit manipulation strategies, so that failure modes stay legible instead of averaging together.
| Strategy | What the prompt asks for |
|---|---|
| Partisan Right | Reframes events for a conservative audience, adding fabricated detail to reinforce the narrative |
| Partisan Left | The mirror image, reframed for a progressive audience |
| Conspiracy | Introduces doubt about official explanations, implies hidden agendas, cites alleged insider information |
| Exaggeration | Amplifies scope with dramatic false detail and fabricated statistics |
| Narrative Shift | Keeps the real entities but alters their implications using fabricated quotes |
Splitting the strategies is what makes the eventual result interpretable. Without it, a transfer failure could be blamed on any number of things. With it, you can ask whether models fail to generalise across manipulation type or across authorship, and separate the two.
Experiments
RoBERTa was fine-tuned across five conditions: within-dataset on each corpus, cross-dataset in both directions, mixed training on the combined data with a stratified 80/20 split, an authorship baseline separating ISOT fakes from Gemini fakes, and leave-one-strategy-out, training on four Gemini strategies and testing on the held-out fifth. All runs used three epochs on a single A100, batch size 16, learning rate 5 × 10⁻⁵, 500 warmup steps and weight decay 0.01, with the best per-epoch checkpoint carried to test.
Results
| Condition | Accuracy | F1 |
|---|---|---|
| ISOT, within-dataset | 99.3% | 99.3% |
| GNews-Gemini, within-dataset | 96.9% | 96.9% |
| ISOT → GNews-Gemini | 48.3% | 20.0% |
| GNews-Gemini → ISOT | 51.5% | 25.8% |
| Mixed training | 97.4% | 97.4% |
| Authorship (human vs LLM fakes) | 99.7% | 99.7% |
Transfer collapses to chance in both directions. The F1 scores are the more damning number: 20.0% and 25.8% indicate the models are not merely uncertain, they are failing systematically on one class.
Authorship dominates veracity
To test whether the authorship boundary explains the collapse, the same architecture was trained to separate ISOT fakes from Gemini fakes. Both classes are fake, so veracity carries no signal at all. That classifier reached 99.7%.
The control that rules out the obvious objection
An obvious alternative explanation is that the model simply cannot generalise to unseen manipulation types. The leave-one-strategy-out experiments rule this out: training on four Gemini strategies and testing on the fifth yields 92.3% to 97.5% accuracy on every held-out strategy. Models transfer cleanly across manipulation strategy when authorship is held constant, and fail completely when it is not. The failure is specific to source.
Why the linguistic features cannot rescue it
Classic misinformation markers do not merely weaken across the boundary, they reverse. Relative to real news, ISOT human-authored fakes show a +779% increase in first-person pronouns and are 29% more readable. LLM fakes invert both: first-person pronouns drop 53% and readability falls 41%, meaning more complex text. Quotation use goes from −59% in ISOT to +147% in GNews-Gemini, and hedging from −1% to +88%.
A detector that has learned "oversimplified text with first-person pronouns implies misinformation" is therefore not just uninformed about LLM fakes, it is actively miscalibrated against them. Only two markers point the same way in both corpora: subjectivity rises and source attribution falls.
Where the signal lives
Silhouette scores on base RoBERTa embeddings, before any fine-tuning, show almost no intrinsic structure: 0.015 by dataset and 0.016 by veracity. Within ISOT, veracity separation reaches 0.046, but within GNews-Gemini only 0.014. Authorship separation among the fakes is 0.043, roughly as strong as the best veracity signal anywhere in the data and about three times the dataset-level drift.
That last comparison also addresses the temporal confound. ISOT is from 2016 to 2017 and GNews-Gemini from 2021 to 2024, so era rather than authorship could in principle drive the collapse. But restricted to real news only, source separation is 0.014, far below the 0.043 authorship separation among fakes. Authorship dominates temporal drift.
Implications
- Reported accuracies are optimistic in a specific direction. Any detector benchmarked on a corpus where authorship and veracity are confounded is reporting a number that will not survive deployment against LLM-generated content.
- The failure is silent. The model does not become uncertain when it crosses the boundary. It stays confident and becomes wrong, which is the worst available combination for a moderation pipeline.
- It is fixable with data, not architecture. Mixed training recovers 97.4%, so the problem is coverage rather than model capacity. The catch is that coverage requires representative samples from each new generative model, which is a moving target.
- Evaluation design was the intervention. No amount of further fine-tuning on a confounded benchmark would have surfaced this. Building the dataset that separates the two factors is what made it visible.
Limitations
Three worth stating plainly. The synthetic fakes come from a single model, so other model families may carry different linguistic signatures, though the leave-one-strategy-out consistency is mild evidence for stability within a generator. The texts are news snippets, title plus description at roughly 212 characters, which reflects how most people encounter news on social media but may hide patterns that only appear in full-length articles. And the datasets differ in era as well as authorship, which the embedding analysis argues against as an explanation but does not eliminate by design.
Presented as an oral at the International Conference on Big Data Analytics (ICBDA) 2026, where it received the Best Presentation Award. Work carried out at the Predictive Analytics and AI Research Lab, NYU Courant, advised by Prof. Anasse Bari.