xlm.tasks.humaneval.eval

pass@k scoring for HumanEval / HumanEval+.

Candidate extraction, program assembly, sandboxed execution and the pass@k estimator follow FlexMDM's evals/humaneval_compare/passk.py; the extraction-robust (any-of-4 modes) score follows robust_passk.py.

HumanEvalPassKEval

Post-hoc evaluator: pass@k on HumanEval (base) and HumanEval+ (plus).

Expects prediction rows with task_id, sample_k and either generated_text or text. Reports, per extraction mode and variant, {humaneval,humaneval_plus}/{mode}/pass@k plus the extraction-robust {humaneval,humaneval_plus}/robust/pass@k (a sample passes if any mode passes). When output_dir is set, also writes per_sample.jsonl, summary.json and robust_summary.json in the reference layout.

Hydra::

post_hoc_evaluator:
  _target_: xlm.tasks.humaneval.HumanEvalPassKEval

build_candidate_code(*, task, completion, mode)

HumanEval candidate for mode; the prompt skeleton is the prefix.

run_one(program, *, timeout_s, python_bin=None)

Run program via python -I - (stdin avoids argv limits on HE+ tests).

pass_at_k(n, c, k)

Unbiased pass@k (Chen et al., 2021) from n samples with c correct.

prediction_completion(pred, task)

Completion text after the prompt.

Prefers generated_text. Otherwise strips the encoded prompt (prompt.strip() + "\n") from the front of the full decode text.