Tiktokenizer
March 3, 2023 ยท View on GitHub
Tiktokenizer
Online playground for openai/tiktoken, calculating the correct number of tokens for a given prompt.
Special thanks to Diagram for sponsorship and guidance.
https://user-images.githubusercontent.com/1443449/222598119-0a5a536e-6785-44ad-ba28-e26e04f15163.mp4