Dynamic LLM Evaluation

ArxivRollBench Leaderboard

ArxivRollBench is a rolling benchmark for measuring how much LLM evaluation can be inflated by public-test contamination and uneven training. It builds fresh SCP tasks from recent arXiv papers, evaluates them once while private, then releases the expired benchmark for reproducibility.

Releases

32024B, 2025A, 2026A

Domains

8Scientific arXiv areas

Tasks

S/C/PSequencing, Cloze, Prediction

Metric

Valid AccMain ranking score

Leaderboard

Model results by release and domain

#	Model	Valid Acc	Coverage	Raw Acc	S	C	P

Diagnostics

Top-model SCP profile

Loading SCP data...

Protocol

Experimental settings

Primary score

Pooled valid accuracy, ranked by non-null valid responses with null-rate warnings.

Secondary score

Raw accuracy and coverage remain visible for models with empty or invalid outputs.

Tasks

SCP: Sequencing, Cloze, and Prediction over recent arXiv article contexts.

Domains

CS, Math, Physics, Statistics, EESS, Quantitative Biology, Quantitative Finance, and Economics.

Prompting

Models are asked to return only the final multiple-choice answer.

Splits

The compact -50 split is the recommended default; full-release data is retained for broad audits.

Evaluation

API and open-weight runs cache every sample response before parsing to avoid unnecessary reruns.

Warnings

Models with high null or empty-response rates stay ranked, but are marked explicitly.

Usage

Evaluate your model

lm-evaluation-harness

lm_eval --model hf --model_args pretrained="your-model",parallelize=True --tasks arxivrollbench2026a-50 --log_samples --output_path ./logs

Citation

@article{Liang_Yu_Shiyu_Ye_Hu_2026,
  title={How Much Do Large Language Model Cheat on Evaluation? Benchmarking Overestimation Under the One-Time-Pad-Based Framework},
  volume={40},
  url={https://ojs.aaai.org/index.php/AAAI/article/view/41098},
  DOI={10.1609/aaai.v40i44.41098},
  abstractNote={Overestimation in evaluating large language models (LLMs) has become an increasing concern. Due to the contamination of public benchmarks or imbalanced model training, LLMs may achieve unreal evaluation results on public benchmarks, either intentionally or unintentionally, which leads to unfair comparisons among LLMs and undermines their realistic capability assessments. Existing benchmarks attempt to address these issues by keeping test cases permanently secret, mitigating contamination through human evaluation, or repeatedly collecting and constructing new samples. However, these approaches fail to ensure reproducibility, transparency, and high efficiency simultaneously. Moreover, the extent of overestimation in current LLMs remains unquantified. To address these issues, we propose ArxivRoll, a dynamic evaluation framework inspired by one-time pad encryption in cryptography. ArxivRoll comprises two key components: i) SCP (Sequencing, Cloze, and Prediction), an automated generator for private test cases, and ii) Rugged Scores (RS), metrics that measure the proportion of public benchmark contamination and training bias. Leveraging SCP, ArxivRoll constructs a new benchmark every six months using recent articles from ArXiv and employs them for one-time evaluations of LLM performance. Extensive experiments demonstrate the high quality of our benchmark, and we provide a systematic evaluation of current LLMs.},
  number={44},
  journal={Proceedings of the AAAI Conference on Artificial Intelligence},
  author={Liang, Zi and Yu, Liantong and Shiyu, Zhang and Ye, Qingqing and Hu, Haibo},
  year={2026},
  month={Mar.},
  pages={37636–37644}
}