LLM4SE Reproducibility Crisis: 86.7% of Papers Fail Audit

86.7% of LLM4SE Papers Carry Reproducibility Failure Modes. Institutional Reputation Now Rests on Mathematical Proof, Not Badges.

Why Deans, ORIC Directors, and Tier-1 Researchers Must Treat Reproducibility as Integrity Infrastructure — and How ScholarMark Turns LLM4SE Studies Into Auditable, Production-Ready Evidence.

By DecentraSec Team

The first large-scale empirical study of its kind examined 640 papers in LLM-for-software-engineering research published between 2017 and 2025 across premier software engineering, machine learning, and natural language processing venues. The authors manually annotated each paper and associated artifact against a seven-category reproducibility-smell taxonomy. The result: 86.7% exhibited at least one reproducibility smell; only 13.3% came out clean [1]. That is not the same as saying all 86.7% failed a re-execution attempt. It is more damning in one respect: the failures are structural and observable before execution. Artifact-evaluation badges certify that files exist; they cannot certify that research can be trusted. The reproducibility crisis is not a culture problem; it is an infrastructure failure. Institutions that replace symbolic badges with cryptographic, tamper-evident custody and provenance will secure the next decade’s grants, rankings, and industry partnerships. Those that do not will inherit the liability.

The Evidence: 86.7% of 640 LLM4SE Papers Show Reproducibility Failure Modes

Reproducibility smells, by category, across the 640 papers [1]:

• Access and Legal: 35.9%
• Code and Execution: 35.5%
• Versioning: 32.2%
• Environment and Tooling: 21.1%
• Model specification: 9.2%
• Data: 8.0%
• Documentation: 3.6%

Categories are not mutually exclusive; a single paper can carry multiple smells.

Companion evidence is more damning. An analysis of 85 commercial LLM-centric papers published at ICSE 2024 and ASE 2024 found that, among the 18 papers that both used OpenAI models and provided artifacts, only five were complete enough to execute, and none fully reproduced the original results [2]. A 2026 Springer mapping study extends the diagnosis into model-driven engineering [3]. Where centralized repositories host these artifacts, integrity rests on operator-controlled storage. A file can be replaced, deleted, or silently corrupted without a verifier noticing unless content hashes are published and checked against a tamper-evident artifact custody layer.

For Deans and ORIC Directors, this evidence base underpins enterprise adoption of AI-generated code. The research your institution endorses feeds directly into production-grade engineering decisions elsewhere. Notice what is absent from this list: a “badge integrity” category. The badge system never measured what matters.

Why Badges Betray: Five Technical Failure Modes in Artifact Evaluation

The failure rate is not random noise; it is the predictable output of five technical failure modes.

1. Non-determinism without pins

LLMs produce different outputs absent pinned model snapshots, commit hashes, and decoding parameters. “The same prompt” is not “the same experiment” because inference also depends on model weights and quantization, sampling parameters, prompt template, and runtime. A paper that names only the model family has not specified the experimental object.

2. Dependency drift

Authors describe environments; they do not lock them. Floating dependency versions turn today’s clean reproduction into next quarter’s silent corruption. A requirements file without version pins or hashes is a description, not an attestation.

3. The badge gap

The 640-paper study is explicit: badges “often signal artifact presence but do not consistently guarantee execution fidelity or long-term reproducibility” [1]. A badge is a claim, not a proof. Content-addressed hashing plus deterministic environment attestation is the structural replacement: store every artifact by the digest of its bytes, and verify that the executed environment matches the declared lockfile.

4. Trust by assertion

Centralized repositories can mutate or lose artifacts. Their integrity rests on operator assertion, not on mathematics anyone can verify. Unless each read returns a digest that can be checked against an independently published commitment, no external party can distinguish an honest operator from a compromised one.

5. Access and legal lockouts

35.9% of papers [1] carry constraints that block re-execution outright. This is the quietest and most fatal failure because no execution framework can legally re-run an artifact a reviewer is not permitted to retrieve. What infrastructure can do is make the constraint explicit, auditable, and bound to the artifact record.

Every one of these is a supply-chain security problem: an artifact moves through storage, review, and reuse, and each step can introduce drift. Reproducibility is not a virtue signal; it is a custody and attestation problem. Algorithmic Integrity, as used here, means a machine-checkable invariant: for any claimed output O, there exists a verifiable record binding O to the declared computation f, inputs I, environment E, model snapshot M, and decoding configuration C, such that a third party can recompute or audit f(I,E,M,C) and test the binding without trusting the record’s author.

From Assessment to Enforcement: The Reproducibility Maturity Model Diagnoses but Cannot Cure

Credit where due: the Reproducibility Maturity Model (RMM) is a genuine advance over binary artifact certification. Its five axes — Accessibility, Environment Specification, Versioning Rigor, Execution Fidelity, Legal Openness — name the right problems.

The blind spot is structural. The RMM grades maturity; it does not provide custody, attestation, or enforcement. A maturity score is a snapshot of intent, not a guarantee of behavior. It cannot prevent a file from being replaced after assessment or prove to a third party that the assessed artifact is the one later retrieved.

The checklist trap follows. Journals and conferences adopt reproducibility checklists and artifact-evaluation tracks, yet a checklist cannot generate an executable, tamper-evident artifact. Compliance paperwork is not computational trust.

Consider the agentic-review irony: if institutions deploy AI agents to screen reproducibility at scale, then AI screening without immutable provenance automates an unverifiable process — garbage in, faster. For an AI reviewer’s judgment to be auditable, the reviewer must consume the same machine-readable lineage — commit hash, decoding parameters, prompt template, dataset version, environment container — not a PDF and a badge. ScholarMark’s AI lineage layer records exactly this custody chain.

ORIC Directors who adopt RMM-style assessment without enforcement infrastructure are buying a report card, not a guarantee.

The Institutional Imperative: Treat Reproducibility as Integrity Infrastructure

For Tier-1 Researchers, reproducible work is becoming a publication gate, not a courtesy. Where venues offer reproducibility tracks, authors who submit mathematically verifiable artifacts clear those gates faster and defend their work in review without ambiguity.

For Deans, research rankings and faculty brand hinge on reproducible outputs. A failed-reproduction scandal is a reputational event; a reproducible lab is a recruiting and rankings asset.

For ORIC Directors, funders and industry partners increasingly ask audit-grade questions: which exact model produced this result, under which license, with which data, and can an independent party re-execute it? Reproducibility-as-infrastructure converts compliance from burden to commercialization asset — research that can be re-executed can be licensed, audited, and shipped. The append-only lineage from dataset and model snapshot to executable output makes research compliant, auditable, and citable as a CI/CD-style gate. This is what decentralized provenance infrastructure delivers.

The strategic window is open. The primary literature now frames reproducibility as a systems-engineering requirement, not a documentation preference [1][2]. Early adopters set the standard; late adopters inherit the liability.

The ScholarMark Blueprint: Reproducibility as Tamper-Evident Infrastructure

ScholarMark treats reproducibility as an infrastructure property, not a cultural aspiration — Integrity Infrastructure deployed as Distributed Infrastructure, with Decentralized Provenance at its core.

Decentralized Provenance is not blockchain branding. It is a hash-linked provenance log: each artifact is content-addressed by a cryptographic digest of its bytes, and each entry references the digest of its parent and the digests of its declared inputs. Any change to an input changes its address and invalidates every descendant. Verification is a deterministic recomputation any party can perform against published digests — no operator’s database need be trusted.

Integritas Vault provides versioned, tamper-evident custody for every code file, dataset, environment lockfile, model snapshot, and prompt — content-addressed and hash-chained. It directly answers dependency drift and trust by assertion, and it makes access constraints explicit rather than invisible.

The AI Integrity Layer records full model and data lineage — commit hash, decoding parameters, prompt template, dataset version, environment container — making every automated review auditable and attributable. It answers non-determinism and the badge gap.

Mathematical Validation and Provenance verify that an artifact’s recorded state matches its claimed inputs and supply the append-only chain from dataset and model to executable output. The check is mechanical: every output is bound to a digest of its input manifest; a verifier recomputes under the declared environment and compares results using the protocol’s stated equivalence rule — exact for deterministic pipelines, statistical for stochastic LLM outputs. This replaces the badge: proof, not presence.

Why distributed infrastructure wins: trust derives from mathematical verification and tamper-evident custody, not from a repository operator’s assertion. Most failure modes become structurally detectable — and therefore operationally much harder to hide — rather than dependent on a badge committee’s judgment. Access and legal lockouts remain a policy constraint: ScholarMark records them and makes them visible, but it cannot grant a license to closed model APIs.

The enterprise-grade payoff: ScholarMark turns AI software-engineering research into a compliant, auditable, CI/CD-style gate — trustworthy enough to ship into production and defend before any reviewer, funder, or partner. Your institution’s certification rests on mathematical proof, not badge claims.

The Institutional Pilot Grant: Funded ScholarMark Deployment for Research Pipelines

The 640-paper study quantifies the gap. The RMM names it. Only infrastructure closes it. DecentraSec now accepts applications for the Institutional Pilot Grant — a funded collaboration for Deans, ORIC Directors, and Tier-1 research labs to deploy ScholarMark’s tamper-evident custody and lineage infrastructure across a real publication pipeline. Selected institutions receive implementation support, integration with existing artifact-evaluation workflows, and a formal reproducibility benchmark for their pilot cohort.

Pilot institutions retain full ownership of their artifacts and data. This is infrastructure for your integrity, not a lock-in.

Apply for the Institutional Pilot Grant.

---

References

1. DOI: 10.48550/arXiv.2512.00651 — Siddiq et al., “Large Language Models for Software Engineering: A Reproducibility Crisis.” First large-scale empirical study of reproducibility in 640 LLM4SE papers (2017–2025); seven-category reproducibility-smell taxonomy; Reproducibility Maturity Model.
2. DOI: 10.48550/arXiv.2510.25506 — Angermeir et al., “Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies.” Analysis of 85 commercial LLM-centric papers published at ICSE 2024 and ASE 2024.
3. DOI: 10.1007/s10664-026-10921-4 — “Large language models in model-driven engineering: a systematic mapping study,” Empirical Software Engineering.