xlm.tasks.humaneval

HumanEval / HumanEval+ code generation eval (pass@k).

Data and scoring follow FlexMDM's evals/humaneval_compare: HumanEval prompts with base and HumanEval+ tests (:mod:data), four completion extraction modes, sandboxed execution and unbiased pass@k (:mod:eval).

HumanEvalDatasetManager

Bases: EvalDatasetManager

Eval manager that expands each HumanEval task into n_samples rows.

full_name is kept for naming only; rows come from :func:load_humaneval_tasks so the prompts match the reference loader exactly.

HumanEvalPassKEval

Post-hoc evaluator: pass@k on HumanEval (base) and HumanEval+ (plus).

Expects prediction rows with task_id, sample_k and either generated_text or text. Reports, per extraction mode and variant, {humaneval,humaneval_plus}/{mode}/pass@k plus the extraction-robust {humaneval,humaneval_plus}/robust/pass@k (a sample passes if any mode passes). When output_dir is set, also writes per_sample.jsonl, summary.json and robust_summary.json in the reference layout.

Hydra::

post_hoc_evaluator:
  _target_: xlm.tasks.humaneval.HumanEvalPassKEval

load_humaneval_tasks()

Load all 164 HumanEval problems with both base and plus tests.

pass_at_k(n, c, k)

Unbiased pass@k (Chen et al., 2021) from n samples with c correct.

humaneval_pred_preprocess_fn(example, tokenizer, *, sep='\n')

Prompt prefix (prompt.strip() + sep) with an empty target, as in dream_code_adaptation training.