README.md
April 23, 2023 ยท View on GitHub
This is a native port of the new RWKV tokenizer.
Usage
- Install Rust.
- Run
python tokenizer_fast.py.
License
Licensed under either of
- Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.