AwesomeREA.org

2022 · Conference paper

ExeBench: an ML-scale dataset of executable C functions

Jordi Armengol-Estapé; Jackson Woodruff; Alexander Brauckmann; José Wesley de Souza Magalhães; Michael F. P. O'Boyle

ExeBench is a dataset, not a decompiler. It extracts C functions from GitHub and AnghaBench, makes them compilable by locating real headers and definitions or injecting synthetic ones, and generates input/output examples from function signatures with a C++ wrapper that runs each function. After exact deduplication the authors report 4,485,817 compilable and 687,843 executable functions, of which 124,612 have rich IO that a constant or identity implementation cannot pass; the paper also describes x86 and ARM assembly at O0, Os and Ofast for each row. Decompilation studies such as SLaDe and LLM4Decompile later used it to test whether decompiled functions re-execute correctly.

Scope and limitations

IO is generated from signatures without program context: 10 pairs per function (30 across three seeds of synthetic definitions), arrays default to 32 elements, and success falls below 5% for functions with pointer–integer conversions and stays low for goto, labels and complex control flow, so executable functions skew simpler than the compilable population. Synthetic IO is produced with injected stand-in definitions rather than the functions’ real dependencies; only 4,464 functions have rich IO with real definitions. Finite IO pairs test behavior on sampled inputs, not semantic equivalence, and deduplication removes exact matches after tokenization only. Running functions with real dependencies executes untrusted code; the authors used restricted containers. The paper’s exact v1.00 data is available only on request, while the public v1.01 release has revised splits (for example 2,134 instead of 2,232 real test functions). Each function keeps its source repository’s license; ExeBench additions are MIT.

Why it fits this collection

Adds the executable C-function benchmark behind several decompilation papers in this section, so readers can check its construction, IO limits and version differences directly.

Publication and review details

Venue
ACM SIGPLAN MAPS 2022
Status
Conference paper
Publication / posting date
June 13, 2022
Date source
The ACM article page and ACM’s Crossref deposit both give publication on 2022-06-13, which is also the day of the MAPS ’22 symposium. The date follows the publisher’s stated publication date, not the event schedule.
DOI
10.1145/3520312.3534867

ACM metadata in Crossref and on the article page gives the sentence-case title and pages 50–59 of the Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming; the 10-page PDF capitalises it as 'ExeBench: An ML-Scale Dataset of Executable C Functions'. No arXiv version was found. The live ACM pages served a browser check to plain HTTP clients and were read through a browser session; the article page marks the full text world-readable, while the PDF's 2022 notice licenses publication rights to ACM, so no open content license is asserted for the paper. The reviewed 2022-09-15 repository commit has no LICENSE file; its README assigns each C function its source repository's license and releases IO examples and wrappers under MIT. Its helper compiles a function's assembly with the C++ wrapper using g++ -O0 and runs it as a host subprocess without its own sandbox. Hugging Face revision 093085f (2023-03-09) hosts v1.01 as zstandard JSON-lines TAR archives with no license tag in the card metadata; the v1.00 dataset jordiae/exebench_legacy requires contacting the authors. No data was downloaded and no code was run.

Sources reviewed

Source review only; experiments were not rerun for this collection. Cite the original authors using the linked paper or BibTeX.

Primary sources

  1. https://dl.acm.org/doi/10.1145/3520312.3534867
  2. https://dl.acm.org/doi/pdf/10.1145/3520312.3534867
  3. https://api.crossref.org/works/10.1145/3520312.3534867
  4. README.md GitHub · jordiae/exebench · Revision 8b47d48
  5. exebench/__init__.py GitHub · jordiae/exebench · Revision 8b47d48
  6. examples/basic.py GitHub · jordiae/exebench · Revision 8b47d48
  7. https://huggingface.co/datasets/jordiae/exebench
  8. https://huggingface.co/datasets/jordiae/exebench/blob/093085f8558cfd53de8e2c8f4ccc7b9e73dc22ae/README.md