Tokenizer module ⛓️

April 19, 2025 · View on GitHub

Used by the parser to convert Oniguruma pattern and flag strings into a list of tokens, as the first step in builing an AST.

Note

This module is not exported by oniguruma-parser, since tokenization is considered an implementation detail of the parser.

About

Created by Steven Levithan and contributors.

If you want to support this project, I'd love your help by contributing improvements (guide), sharing it with others, or sponsoring ongoing development.

MIT License.