Hi, I would like to add tests for Longformer tokenizer

@SaulLu I would like to add tests for Flaubert

Hey I would like to contribute for Electra,Pointers please!

Thank you all for offering your help!

@Rajathbharadwaj ,sure! what do you need help with? Do you need more details on any of the steps listed in the main post?

Hi, first time contributor here-could I add tests for Splinter?

Is anyone else encountering this error with the cookiecutter command? my dev environment set up seemed to have went all fine...
Also I had run the command inside the tests/splinter directory

@faiazrahman , thank you so much for working on this! Regarding your issue, if you're in the tests/splinter folder, can you try to run cookiecutter ../../templates/adding_a_missing_tokenization_test/ ?

You should have a newly created folder cookiecutter-template-BrandNewBERT inside tests/splinter. 🙂

If that's the case, you'll need after to do something like:

mv cookiecutter-template-BrandNewBERT/test_tokenization_brand_new_bert.py .
rm -r cookiecutter-template-BrandNewBERT/

Keep me posted 😄

Thanks so much @SaulLu turns out it was due to not recognizing my installed cookiecutter so i sorted it out there. 👍

Hi @anmolsjoshi, @tgadeliya, @Rajathbharadwaj and @farahdian,

Just a quick message to see how things are going for you and if you have any problems. If you do, please share them! 🤗

Thanks @SaulLu ! I've been exploring the tokenization test files in the repo just trying to figure out which ones would be a good basis for writing a tokenization test for splinter... if you have any guidance on this it would be super helpful!

Hey @SaulLu my apologies, been a bit busy. I'll get started ASAP, however, I still didn't understand where exactly I should run the cookie cutter

Help on this would be helpful 😄

Hi @farahdian ,

Thank you very much for the update! To know where you stand, have you done step 3)? Is it for step 4) that you are looking for a similar tokenizer? 🙂

Hi @Rajathbharadwaj ,

Thank you for the update too!

I still didn't understand where exactly I should run the cookie cutter

You can run the cookie cutter command anywhere as long as the command is followed by the path to the folder adding_a_missing_tokenization_test in the transformers repo that you have cloned locally.

When you run the command, it will create a new folder at the location you are. In this folder you will find a base for the python test file that you need to move inside the tests/electra folder of the transformers local clone. Once this file is moved, you can delete the folder that was created by the cookie cutter command.

Below is an example of the sequence of bash commands I would personally use:

(base) username@hostname:~$ cd ~/repos
(base) username@hostname:~/repos$ git clone git@github.com:huggingface/transformers.git
[Install my development setup]
(transformers-dev) username@hostname:~/repos$ cookiecutter transformers/templates/adding_a_missing_tokenization_test/
[Answer the questionnaire]
(transformers-dev) username@hostname:~/repos$ mv cookiecutter-template-Electra/test_tokenization_electra.py transformers/tests/Electra
(transformers-dev) username@hostname:~/repos$ rm -r cookiecutter-template-Electra/

Hope that'll help you 😄

Appreciate your patience @SaulLu ! Yup I've done step 3 and generated a test tokenization file with cookiecutter. Now onto working on the setUp method 😄

@farahdian , this is indeed a very good question: finding the closest tokenizer to draw inspiration from and identifying the important difference with that tokenizer is the most interesting part.

For that there are several ways to start:

1. Identify the high level features of the tokenizer by looking at the contents of the model's "reference" checkpoint files (listed inside the PRETRAINED_VOCAB_FILES_MAP global variables in the tokenizer's files) on the hub. A similar model would most likely store the tokenizer vocabulary in the same way (with only a vocab file, with both a vocab and a merges files, with a sentencepiece binary file or with only a tokenizer.json file).
2. Read the high level explanation of the model in the transformers documentation (e.g. for Splinter)
3. Read the paper corresponding to the model
4. Look at the implementation in transformers lib
5. Look at the original implementation of the model (often mentioned in the paper)
6. Look at the discussions on the PR in which the model was added

For the model you're in charge @farahdian:

• Transformers's doc mention that:

  Use SplinterTokenizer (rather than BertTokenizer), as it already contains this special token. Also, its default behavior is to use this token when two sequences are given (for example, in the run_qa.py script).

• Splinter's paper mention that:

  Splinter-base shares the same architecture (transformer encoder (Vaswani et al., 2017)), vocabulary (cased wordpieces), and number of parameters (110M) with SpanBERT-base (Joshi et al., 2020).

• And SpanBERT's paper mention that:

  We reimplemented BERT’s model and pre-training method in fairseq (Ott et al., 2019). We used the model configuration of BERT large as in Devlin et al. (2019) and also pre-trained all our models on the same corpus: BooksCorpus and English Wikipedia using cased Wordpiece tokens.

• and the vocabulary files of bert-base-cased (vocab file) and of splinter-base (vocab file) look very similar

Given these mentions, it seems that Splinter's tokenizer is very similar to Bert's one. It would be interesting to confirm this impression and to understand all the differences between SplinterTokenizer and BertTokenizer so that it is well reflected in the test 🙂

Hi @Rajathbharadwaj ,

Thank you for the update too!

I still didn't understand where exactly I should run the cookie cutter

You can run the cookie cutter command anywhere as long as the command is followed by the path to the folder adding_a_missing_tokenization_test in the transformers repo that you have cloned locally.

When you run the command, it will create a new folder at the location you are. In this folder you will find a base for the python test file that you need to move inside the tests/electra folder of the transformers local clone. Once this file is moved, you can delete the folder that was created by the cookie cutter command.

Below is an example of the sequence of bash commands I would personally use:

(base) username@hostname:~$ cd ~/repos
(base) username@hostname:~/repos$ git clone git@github.com:huggingface/transformers.git
[Install my development setup]
(transformers-dev) username@hostname:~/repos$ cookiecutter transformers/templates/adding_a_missing_tokenization_test/
[Answer the questionnaire]
(transformers-dev) username@hostname:~/repos$ mv cookiecutter-template-Electra/test_tokenization_electra.py transformers/tests/Electra
(transformers-dev) username@hostname:~/repos$ rm -r cookiecutter-template-Electra/

Hope that'll help you smile

Thank you so much @SaulLu
I understood now, however, I am skeptical about slow_tokenizer_use_sentencepiece question, but I set it to True as it had the tokenization_electra.py file but I didn't understand

"Set true if the tokenizer defined in the tokenization_brand_new_bert.py file uses sentencepiece"

So did I select correctly? Or should I set it to False? Apologies for asking so many questions 😄

However now I've started adding tests for Electra will keep you posted if I run into something I don't understand.

Thanks for helping once again!

Hi @SaulLu,
I think my case the easiest one, because Longformer model uses actually the same tokenizer as RoBERTa with no differences. So, I adapted tests(small refactor and changes) from RoBERTa tokenizer and prepare branch with tests. Nevertheless, I really want to dive deeper and study code of TokenizerTesterMixin and if after that I will find some untested behaviour, I will add new tests.
But I think I have one doubt, that you can resolve. Are you anticipating from Longformer tests to have different toy tokenizer example than in RoBERTa tests? Or maybe I should write my own tests from scratch?

@Rajathbharadwaj , I'm happy to help! Especially as your questions will surely be useful for other people

however, I am skeptical about slow_tokenizer_use_sentencepiece question, but I set it to True as it had the tokenization_electra.py file but I didn't understand
"Set true if the tokenizer defined in the tokenization_brand_new_bert.py file uses sentencepiece"
So did I select correctly? Or should I set it to False? Apologies for asking so many questions smile

Some XxxTokenizer (without the Fast at the end, implemented in the tokenization_xxx.py file), use a backend based on the sentencepiece library. For example T5Tokenizer uses a backend based on sentencepiece: you can see this import at the beginning of the tokenization_t5.py file:

On the contrary, BertTokenizer for example does not use a sentencepiece backend.

I hope this helped you!

Hi @tgadeliya ,

Thanks for the update!

But I think I have one doubt, that you can resolve. Are you anticipating from Longformer tests to have different toy tokenizer example than in RoBERTa tests? Or maybe I should write my own tests from scratch?

In your case, I wouldn't be surprised if Longformer uses the same tokenizer as RoBERTa. In this case, it seems legitimate to use the same toy tokenizer. Maybe the only check you can do to confirm this hypothesis is comparing the vocabularies of the 'main" checkpoints of both models:

!wget https://huggingface.co/allenai/longformer-base-4096/raw/main/merges.txt
!wget https://huggingface.co/allenai/longformer-base-4096/raw/main/vocab.json
!wget https://huggingface.co/roberta-base/raw/main/merges.txt
!wget https://huggingface.co/roberta-base/raw/main/vocab.json

!diff merges.txt merges.txt.1
!diff vocab.json vocab.json.1

Turn out the result confirms it!

Hi, I'm happy to take MobileBert

I'd like to work on ConvBert.

Identify the high level features of the tokenizer by looking at the contents of the model's "reference" checkpoint files (listed inside the PRETRAINED_VOCAB_FILES_MAP global variables in the tokenizer's files) on the hub. A similar model would most likely store the tokenizer vocabulary in the same way (with only a vocab file, with both a vocab and a merges files, with a sentencepiece binary file or with only a tokenizer.json file).

@SaulLu I'm having trouble identifying ConvBert's 'reference' checkpoint files on the hub. Would you kindly provide more guidance on this?

Hi @elusenji ,

In the src/transformers/models/convbert/tokenization_convbert.py file you can find the global variable PRETRAINED_VOCAB_FILES_MAP:

In particular YituTech/conv-bert-base is a reference checkpoint for ConvBert.

Is this what you were having trouble with? ☺️

Yes, this helps!

Hi @SaulLu, I am happy to write tests for RemBert. Thanks.