Leakage Aware Evaluation of Arabic Text Classification: Quantifying Document Level Leakage and the Sufficiency of Linear TF IDF Baselines

Abstract

The volume of Arabic text online keeps growing, and with it the need for systems that sort it into meaningful categories. A quieter problem often passes unnoticed: when long documents are split into shorter segments and pieces of the same document fall on both sides of the boundary between training and testing, the reported scores can look far better than the model deserves. This study treats that document level leakage as its central subject. Using a purpose built multidomain Arabic corpus of 1,000 segments drawn from 183 source documents across seven categories, we first measure how much leakage distorts the numbers. Under a naive random split, an optimized Linear SVM appears to reach 0.9800 accuracy; under a grouped split keyed to the document identifier, the same model reaches 0.8528. The gap of about 12.7 accuracy points is attributable to leakage, and across models the inflation ranges from roughly 11.9 to 30.4 points. Leakage also flattens the ranking, so a weak Naive Bayes looks almost as strong as the best model. Under the honest protocol, an ablation shows that character level TF IDF alone matches the combined word and character representation and slightly exceeds it on Macro F1, 0.7801 against 0.7713, while word features mainly refine precision. McNemar tests and a document clustered bootstrap show that the strongest linear models are statistically indistinguishable. A fastText baseline reaches 0.6497 accuracy, and two pretrained Arabic transformers, AraBERT and MARBERT, perform comparably to the linear model rather than surpassing it. A class level analysis exposes the General category as a structural confound whose removal raises accuracy to 0.9349. Overall, leakage aware evaluation paired with subword rich TF IDF and a Linear SVM forms a strong, honest, and practical baseline for Arabic text classification.