A very good first issue IMO!

See huggingface/transformers#4829 (comment)

we should update the eval dataset to pick one start_position (or the most frequent one)

Optionally, use the huggingface/nlp library to get the eval dataset, and hook it into the Trainer.

Also referenced in huggingface/transformers#6997 (comment)

Well, Gumbel Distribution is magical. Basically, given a sequence of K logits, i.e., "\log a_1, \log a_2, ..., \log a_K" and K independent gumbel random variables, i.e., "g_1, g_2, ..., g_K". We have

\argmax_i \log a_i + g_i ~ Categorical({a_i / sum(a)})

This gives you a very simple way to sampl