xlm.utils.text

Text utility functions for xlm.

remove_trailing_pads(text, tokenizer)

Remove trailing pad tokens from decoded text.

Parameters:

Name	Type	Description	Default
text	str	Decoded text string that may contain trailing pad tokens	required
tokenizer	Tokenizer	Tokenizer instance containing pad_token	required

Returns:

Type	Description
str	Text with trailing pad tokens removed