Skip to content

xlm.tasks.iwslt14_de_en

IWSLT14 German→English (RDM text + joint CharBPE) preprocessing.

Iwslt14MtEval

Corpus MT metrics over logged IWSLT14 predictions.

Hydra::

post_hoc_evaluator:
  _target_: xlm.tasks.composite_eval.CompositePostHocEvaluator
  evaluators:
    prediction:
      mt:
        _target_: xlm.tasks.iwslt14_de_en.mt_eval.Iwslt14MtEval
        mode: sacrebleu_detok

joint_char_bpe_v1_dir()

Absolute path to the packaged joint CharBPE v1 directory.

Prefer IWSLT14_JOINT_CHAR_BPE_V1 when set; otherwise the files shipped under xlm.tasks.iwslt14_de_en.tokenizer_v1 (works for editable and wheel installs).

iwslt14_mt_preprocess_fn(example, tokenizer)

Encode DE/EN text to raw BPE id lists (no special tokens).

Collators own BOS/EOS, truncation, padding, and noise.