xlm.tasks.humaneval.score
Score HumanEval / HumanEval+ pass@k from prediction JSONL files.
Use this when generation was sharded across jobs
(HumanEvalDatasetManager(shard_index=..., num_shards=...)) and the in-job
post-hoc evaluator was disabled; it concatenates the per-shard predictions and
runs :class:HumanEvalPassKEval once over the full sample set.
python -m xlm.tasks.humaneval.score \
--predictions logs/<job>_shard*/runs/*/predictions/test/humaneval_prediction/*.jsonl \
--n-samples 16 --output-dir /path/to/humaneval_passk