3;ji~' d Z ddlZddlZddlZddlmZ ddlmZmZm Z ddl
m Z ddl m
Z
mZ e
rdd lmZ ndZ ej e Zd
d d Z G d
de ZdS )z Tokenization class for model T5. N)copyfile)ListOptionalTuple )PreTrainedTokenizerFast)is_sentencepiece_availablelogging ) T5Tokenizerz spiece.modelztokenizer.json)
vocab_filetokenizer_filec \ e Zd ZU dZeZddgZeZg Z e
e e d< d fd
Z
ed efd Zed
Zddedee d ee fdZ dde
e dee
e d e
e fdZ dde
e dee
e d e
e fdZd Zd Z xZS )T5TokenizerFasta`
Construct a "fast" T5 tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models).
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as
"" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by
calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method
additional_special_tokens (`List[str]`, *optional*):
Additional special tokens used by the tokenizer.
add_prefix_space (`bool`, *optional*):
Whether or not the tokenizer should automatically add a prefix space
from_slow (`book`, *optional*, defaults to `False`):
Whether or not the tokenizer should be converted from a slow one. If `add_prefix_space` is set, this will be set to `True`.
input_idsattention_mask
prefix_tokensNd c
|ld |D }
t |
dk r |d t | D z
}nK|dk r)|t |
k rt d| d| d nd t | D }
|
}| t d d
| d < t j |f||||||d | || _ || _ d S )
Nc 4 g | ]}d t | v |S )
z,T5TokenizerFast.__init__..d s, [[[!LTWXYTZTZDZDZADZDZDZ r c g | ]}d | d S r > r is r r z,T5TokenizerFast.__init__..f s$ -Z-Z-ZA.?1.?.?.?-Z-Z-Zr r zBoth extra_ids (z!) and additional_special_tokens (zk) are provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensc g | ]}d | d S r" r$ r% s r r z,T5TokenizerFast.__init__..n s$ HHH! - - - -HHHr zXYou set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizersT from_slow)r eos_token unk_token pad_token extra_idsadditional_special_tokens) lenrange
ValueErrorlogger warning_oncesuper__init__r
_extra_ids) selfr
r r) r* r+ r, r- add_prefix_spacekwargs extra_tokens __class__s r r4 zT5TokenizerFast.__init__V sQ % 0[['@[[[L< 1$$)-Z-ZyIYIY-Z-Z-ZZ))Q 9L0A0A#A#A y Sl IHuY7G7GHHHL(4 % ' j
#'F;
) &?
%#r returnc \ | j r$t j | j ndS )NF)r
ospathisfiler6 s r can_save_slow_tokenizerz'T5TokenizerFast.can_save_slow_tokenizer s$ 26/Lrw~~do...uLr c | t j v rEt j | }|||k r|S |'t j d| d| d| d| d t |S )NzGThis tokenizer was incorrectly instantiated with a model max length of z which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on z( automatically truncating your input to zM when padding/encoding.
- If you want to encode/pad to sequences longer than z you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.)r max_model_input_sizeswarningswarn
FutureWarning) pretrained_model_name_or_pathmax_model_lengthinit_max_model_lengthdeprecated_max_model_lengths r !_eventually_correct_t5_max_lengthz1T5TokenizerFast._eventually_correct_t5_max_length s (O,Q Q Q*9*OPm*n '$05JN^5^5^,,&.
g3g g 6 g g
4 g g %@
g g g " r save_directoryfilename_prefixc
| j st d t j | s t
d| d d S t j ||r|dz ndt d z }t j | j
t j | k r2t | j
| t
d| |fS )NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory- r
zCopy vocab file to )
rA r0 r= r> isdirr1 errorjoinVOCAB_FILES_NAMESabspathr
r info)r6 rL rM out_vocab_files r save_vocabularyz T5TokenizerFast.save_vocabulary s +
w}}^,, LLT^TTT U U U F o M_s 2 22QbcoQp p
7??4? + +rw~/N/N N N T_n 5 5 5 KK>n>> ? ? ? r token_ids_0 token_ids_1c ` || j gz }|
| j |z S || j gz }| j |z |z S )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) eos_token_idr )r6 rY rZ s r build_inputs_with_special_tokensz0T5TokenizerFast.build_inputs_with_special_tokens sK & "T%6$77 % 3 3%):(;;K% 3kA Ar c z | j g}|t ||z dgz S t ||z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
Nr )r\ r. )r6 rY rZ eoss r $create_token_type_ids_from_sequencesz4T5TokenizerFast.create_token_type_ids_from_sequences sS ! {S())QC/ /;${2S899QC??r c b t t t d | j S )Nc J t t j d| d uS )Nz)boolresearch)r s r z5T5TokenizerFast.get_sentinel_tokens.. s bi0A1&E&E!F!Fd!R r )listsetfilterr- r@ s r get_sentinel_tokensz#T5TokenizerFast.get_sentinel_tokens s1 RRTXTrss t t
r c D fd D S )Nc : g | ]} | S r$ )convert_tokens_to_ids)r tokenr6 s r r z:T5TokenizerFast.get_sentinel_token_ids.. s' ZZZe**511ZZZr )rj r@ s `r get_sentinel_token_idsz&T5TokenizerFast.get_sentinel_token_ids s) ZZZZt?W?W?Y?YZZZZr )NNr r r r NN)N) __name__
__module__ __qualname____doc__rT vocab_files_namesmodel_input_namesr slow_tokenizer_classr r int__annotations__r4 propertyrc rA staticmethodrK r r r rX r] r` rj ro
__classcell__)r: s @r r r ) s $ $L *$&67& !M49!!! "& -$ -$ -$ -$ -$ -$^ M M M M XM \ *! !c !HSM !]bcf]g ! ! ! !* JNB B 9B3;DI3FB
cB B B B6 JN@ @ 9@3;DI3F@
c@ @ @ @,
[ [ [ [ [ [ [r r )rs r= rd rD shutilr typingr r r tokenization_utils_fastr utilsr r
tokenization_t5r
get_loggerrp r1 rT r r$ r r r s ' & ( ( ( ( ( ( ( ( ( ( > > > > > > 8 8 8 8 8 8 8 8 ,,,,,,,K
H % %#1EUVV @[ @[ @[ @[ @[- @[ @[ @[ @[ @[r