LUCAS LIEBE[ 01 // SYSTEMS ][ 02 // KITCHEN ]
[ POTSDAM: --:-- CET ⇄ DAEJEON: --:-- KST ]
DATE: 2023-11⚡ 70.4% Accuracy / 85.65% Weighted F1 vs. 26% OpenAI Classifier

UNCOVER: Identifying AI Generated News Articles by Linguistic Analysis and Visualization (KDIR '23)

PythonStanford CoreNLPLogistic RegressionTopic ModelingStylometryWeb Visualization
[ ⬇ DOWNLOAD PDF ]

01 // Overview & Core Contribution

Published at KDIR 2023 — 15th International Conference on Knowledge Discovery and Information Retrieval (Hasso Plattner Institute & University of Potsdam).

Trust-sensitive domains such as journalism and plagiarism detection cannot rely on opaque black-box detectors that offer low accuracy and zero accountability. UNCOVER is an explainable, human-in-the-loop linguistic analysis and visualization framework that distinguishes human-written from LLM-generated news articles without requiring a pre-trained LLM detector.

02 // Explainable Linguistic Pipeline

  • Character & Syntactic Stylometry: Extracts top-100 character trigrams and dependency-tree syntactic trigrams (via Stanford CoreNLP) to train interpretable author-group logistic regression classifiers.
  • Topic Evolution Model (TEM): Adapts temporal topic-flow graphs to single-article granularity—treating paragraphs as temporal periods and sentences as documents—to track how themes split, merge, or stagnate across an article.
  • Graph Connectivity Classification (TEcm): Computes four topological graph metrics (theme-identifier ratios, incoming-edge coverage, and longest connected period chain) that reveal how AI articles tend to stay locked on a single static topic while human articles transition cleanly between sub-themes.
  • Entity Coreference Diagram & Decision-Tree Fusion: Visualizes rolling entity occurrences across sentences alongside a transparent decision tree combining stylometric and TEcm confidence scores.

03 // Key Evaluation Highlights

Evaluated on a newly constructed corpus of 5,237 human (The Guardian) and LLM-generated (GPT-2, GPT-3, GROVER) news articles, plus an in-person expert user study (n = 13):

  • Detection Performance: Achieves 70.4% accuracy and 85.65% weighted F1-score on unseen multi-generator test articles—substantially outperforming OpenAI’s baseline classifier (26%).
  • Human-in-the-Loop Usability: Expert participants reached 69% classification accuracy, rated the Topic Evolution Graph highest for visual clarity (4/6) and decision influence (5/6), and a majority expressed interest in adopting the tool for text verification.
  • Open-Source Artifacts: github.com/hpicgs/unCover