xlm.tasks.cfg

SyntacticMetricsEvaluator

Post-hoc evaluator for CFG syntactic metrics on logged predictions.

Computes parse_rate, rule_coverage, rule_entropy, template_diversity, etc. on decoded text from JSONL.

eval(predictions, tokenizer=None)

Evaluate predictions and return updated predictions + aggregated metrics.

compute_syntactic_metrics(sentences, grammar_str, *, tokenizer=str.split, max_parses_per_sent=50, parse_mode='first', include_lexical_rules=False, strip_terminals_for_rules=True, return_per_sample=False)

Compute syntactic metrics (rule coverage, entropy, template diversity).

parse_mode: "first" uses only first parse per sentence; "all" aggregates productions across all parses. include_lexical_rules=False focuses on syntax.