Skip to content

xlm.tasks.humaneval.score

Score HumanEval / HumanEval+ pass@k from prediction JSONL files.

Use this when generation was sharded across jobs (HumanEvalDatasetManager(shard_index=..., num_shards=...)) and the in-job post-hoc evaluator was disabled; it concatenates the per-shard predictions and runs :class:HumanEvalPassKEval once over the full sample set.

python -m xlm.tasks.humaneval.score \
    --predictions logs/<job>_shard*/runs/*/predictions/test/humaneval_prediction/*.jsonl \
    --n-samples 16 --output-dir /path/to/humaneval_passk