Skip to content

xlm.backbones.dream

Dream-family backbone (modeling + tokenizer + DreamConfigBase).

DreamConfigBase

Bases: PretrainedConfig

Architecture fields shared by Dream and DreamOn decoder LMs.

DreamBaseModel

Bases: DreamPreTrainedModel

Transformer decoder consisting of config.num_hidden_layers layers. Each layer is a [DreamDecoderLayer]

Parameters:

Name Type Description Default
config DreamConfigBase

DreamConfigBase

 required

DreamTokenizer

Bases: PreTrainedTokenizer

Construct a Dream tokenizer. Based on byte-level Byte-Pair-Encoding.

Same with GPT2Tokenizer, this tokenizer has been trained to treat spaces like parts of the tokens so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not:

>>> from transformers import AutoTokenizer

>>> tokenizer = AutoTokenizer.from_pretrained("Dream-org/Dream-v0-Base-7B", trust_remote_code=True)
>>> tokenizer("Hello world")["input_ids"]
[9707, 1879]

>>> tokenizer(" Hello world")["input_ids"]
[21927, 1879]
This is expected.

You should not use GPT2Tokenizer instead, because of the different pretokenization rules.

This tokenizer inherits from [PreTrainedTokenizer] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

Parameters:

Name Type Description Default
vocab_file `str`

Path to the vocabulary file.

 required
merges_file `str`

Path to the merges file.

 required
errors `str`, *optional*, defaults to `"replace"`

Paradigm to follow when decoding bytes to UTF-8. See bytes.decode for more information.

 'replace'
unk_token `str`, *optional*, defaults to `"<|endoftext|>"`

The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead.

 '<|endoftext|>'
bos_token `str`, *optional*

The beginning of sequence token. Not applicable for this tokenizer.

 None
eos_token `str`, *optional*, defaults to `"<|endoftext|>"`

The end of sequence token.

 '<|endoftext|>'
pad_token `str`, *optional*, defaults to `"<|endoftext|>"`

The token used for padding, for example when batching sequences of different lengths.

 '<|endoftext|>'
clean_up_tokenization_spaces `bool`, *optional*, defaults to `False`

Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. Not applicable to this tokenizer, since tokenization does not add spaces.

 False
split_special_tokens `bool`, *optional*, defaults to `False`

Whether or not the special tokens should be split during the tokenization process. The default behavior is to not split special tokens. This means that if <|endoftext|> is the eos_token, then tokenizer.tokenize("<|endoftext|>") = ['<|endoftext|>]. Otherwise, if split_special_tokens=True, then tokenizer.tokenize("<|endoftext|>") will be give ['<', '|', 'endo', 'ft', 'ext', '|', '>']. This argument is only supported for slow tokenizers for the moment.

 False

convert_tokens_to_string(tokens)

Converts a sequence of tokens (string) in a single string.