xlm.generative_perplexity
AutoModelForCausalLMGenerativePerplexityEvaluator
Bases: GenerativePerplexityEvaluator
__init__(name, batch_size=64, device=None)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
Name of the pretrained model. Currently supported models: gpt2-* |
required |
batch_size
|
int
|
Batch size for the evaluator. The default should work on A100. (default: 64) |
64
|
load(generator_tokenizer, device)
Load the pretrained model and tokenizer.
- EOS We need to make sure that if the generator generates EOS token. Then the evaluator can evaluate it correctly. That is, the eval tokenizer recognizes it as single EOS token and the eval model is trained to predict it.
- PAD We need PAD token in eval tokenizer because retokenization will produce sequences of varying lengths. Some models like GPT2 only have EOS token. In those cases, we will not be able to evaluate EOS generation because PAD will be set to EOS and the EOS token will be lost.