LUCAS LIEBE[ 01 // SYSTEMS ][ 02 // KITCHEN ]
[ POTSDAM: --:-- CET ⇄ DAEJEON: --:-- KST ]
DATE: 2026-07⚡ 77.56% Accuracy / 86.51% Weighted F1 vs. 56.68% Winston AI

Detecting and Comparing LLM Capabilities to Human Writers Through Linguistic Analysis (SNCS '26)

PythonStanford CoreNLPWord2VecfastTextRandom ForestStylometryTopic Modeling
[ ⬇ DOWNLOAD PDF ]

01 // Overview & Extension Over KDIR ’23

Published in SN Computer Science (Springer Nature, Vol. 7, July 2026) — Co-First Author with Jannis Baum, alongside Tim Barz-Cech, Willy Scheibel, and Jürgen Döllner (KAIST, Hasso Plattner Institute & University of Potsdam).

Building on our initial KDIR 2023 framework, this journal extension substantially upgrades unCover’s linguistic feature extraction, semantic graph modeling, and cross-lingual evaluation to detect modern instruction-tuned LLMs (GPT-4, Gemini) without sacrificing white-box explainability.

02 // Key Upgrades Over Original unCover

  • Word-Level Stylometry & Random Forest Meta-Classifier: Expands stylometric profiling from character and dependency-tree syntactic trigrams to include word trigrams, replacing hard threshold comparisons with a shallow Random Forest (max_depth = 5) over normalized author-group logistic regressions.
  • Semantic Topic Evolution Model (TEM-2): Replaces literal word-overlap and term-correlation heuristics with Word2Vec / fastText embedding distances weighted by a vector-magnitude meaningfulness function (mf(w)), using symmetric topic average distance (tad) and asymmetric topic mover’s distance (tmd) to link themes even when exact keywords do not repeat.
  • Expanded Graph Metrics (TEGM 1–7): Augments the four global graph-connectivity ratios with three local structural metrics—mean topics per period, mean words per topic, and mean word energy—classified via a depth-bounded Random Forest.
  • Cross-Lingual Generalization (German): Adapts the pipeline to German news (N-TV vs. GPT-4/Gemini) using German Word2Vec/fastText embeddings, spaCy, and NLTK without hyperparameter re-tuning.

03 // Key Evaluation Highlights

Evaluated on an expanded benchmark of 7,157 English (The Guardian vs. GPT-2, GPT-3, GPT-4, Gemini, GROVER) and 2,295 German news articles:

  • Higher Accuracy & Human-Author Protection: Boosts overall English detection accuracy from 70.4% (KDIR ’23) to 77.56% (86.51% weighted F1 under double false-positive penalty for human authors), outperforming commercial SOTA detector Winston AI (56.68%) and reaching 80% accuracy on verified human journalism.
  • Modern LLM & Cross-Lingual Coverage: Detects GPT-4 (92%) and Gemini (95%) with high precision, achieves 82.57% on the external ArguGPT essay benchmark, and reaches 66.45% zero-tuning accuracy on German news articles.
  • Code, Datasets & Container: github.com/hpicgs/unCover • Zenodo Code Archive • Docker: lucasliebe/uncover