The Wayback Machine - https://web.archive.org/web/20230320120338/https://github.com/huggingface/tokenizers/pulls
Skip to content

Pull requests: huggingface/tokenizers

New pull request New
20 Open 434 Closed
20 Open 434 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Adding ByteFallback support for tokenizers.
#1183 opened Mar 17, 2023 by Narsil Loading…
1
4
Use LTO for release and benchmark builds
#1157 opened Jan 31, 2023 by csko Loading…
Allow to build without onig or fancy-regex
#1144 opened Jan 13, 2023 by llogiq Loading…
5
Bump dirs from 3.0 to 4.0
#1142 opened Jan 6, 2023 by hvaara Loading…
5
TEST Please Ignore
#1134 opened Dec 23, 2022 by hvaara • Draft
1
Add spaces_between_special_tokens and cleanup tokenization spaces
#1095 opened Nov 3, 2022 by ArthurZucker Loading…
5
[WIP] Unigram trainer seems odd, ignoring some suffixes entirely
#1081 opened Oct 6, 2022 by Narsil Loading…
1
Add wasm32 emscripten target support for python binding
#1021 opened Jul 1, 2022 by messense Loading…
Use sentencepiece's protobuf module instead of the local protobuf file
#992 opened Apr 29, 2022 by tma15 Loading…
Parallelize unigram trainer
#976 opened Apr 6, 2022 by mishig25 Loading…
1
7
Update outdated dependencies and feature-gate CLI
#958 opened Mar 21, 2022 by MarcusGrass Loading…
3
Attempt to make unigram faster 2.
#921 opened Feb 24, 2022 by thomasw21 Loading…
7
Attempt to make Unigram trainer parallel.
#920 opened Feb 24, 2022 by Narsil Loading…
Merges cannot handle tokens containing spaces.
#909 opened Feb 16, 2022 by Narsil • Draft
13
JVM - Add bindings with Java API
#842 opened Dec 7, 2021 by nguyenvietyen Loading…
2
Make AddedVocabulary more versatile
#720 opened May 28, 2021 by Backfighter Loading…
1
add Zenodo DOI Badge
#716 opened May 25, 2021 by SaulLu Loading…
3
C++ bindings
#559 opened Dec 9, 2020 by alexeyr • Draft
2 of 7 tasks
5
Add normalization option to Chinese characters (using OpenCC) and separate symbols from merging
#473 opened Oct 19, 2020 by ecchochan Loading…
remove use of parallel iterators except in batch methods
#308 opened Jun 17, 2020 by epwalsh Loading…
3
ProTip! Adding no:label will show everything without a label.