This is not an official Google product.
- This fork utilizes multiple cores and smart buffering to divide-and-conquer large datasets. Takes a 10B token source from over an hour to under 10 minutes. Reduces memory overhead by ~50 percent.
sysreq/sentencepiece_parallel
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Repository files navigation
Releases
No releases published
Languages
- C++ 96.7%
- Python 1.0%
- SWIG 0.8%
- CMake 0.8%
- Jupyter Notebook 0.6%
- Perl 0.1%