Both appear in the tokenizer, but it's not clear to me which to use for padding examples in batches (or just when padding to consistent length for specific kernels).
Also, why the second pad token is same as token 32014? I assume it is on purpose. Could you please help to explain the reason?
Some results have been hidden because they may be inaccessible to you
Show inaccessible results
Feedback