xlm.tasks.humaneval.data

HumanEval / HumanEval+ task loading.

Ports the task loader of FlexMDM's evals/humaneval_compare/common.py: 164 problems from openai/openai_humaneval (test split) joined on task_id with evalplus/humanevalplus so every task carries both the base (test_base) and plus (test_plus) test code. HumanEval and HumanEval+ share prompts, so one set of generations is scored against both.

HumanEvalDatasetManager

Bases: EvalDatasetManager

Eval manager that expands each HumanEval task into n_samples rows.

full_name is kept for naming only; rows come from :func:load_humaneval_tasks so the prompts match the reference loader exactly.

load_humaneval_tasks()

Load all 164 HumanEval problems with both base and plus tests.

build_sample_rows(tasks, n_samples)

One row per (task, sample_k), task-major like the reference work list.