README.org

#+TITLE: Tree Sitter Noir

Noir grammar for tree-sitter. Noir is a ZK circuit programming language created by Aztec, you can find it [[https://noir-lang.org/docs][here]] (or [[https://github.com/noir-lang/noir][GitHub]]).

** Status

Grammar works, queries and refinements are the focus now.

See [[./PROJECT.org][tree_sitter_noir's project planning file]] for TODOs, and further resources etc.

** Build / Develop / Contribute

You will need a C compiler, and the [[https://github.com/tree-sitter/tree-sitter/releases][tree-sitter-cli]].

For convenience you can use [[https://bun.sh/][Bun]] to manage installing the tree-sitter-cli. Additionally a [[./justfile][justfile]] (see: [[https://github.com/casey/just][Just]]) provides simpler access to common project commands. Execute just anywhere within this project repo to see a list of available commands, common ones are:

To install dependencies for development: #+begin_src sh just bootstrap #+end_src

To build and test: #+begin_src sh just build #+end_src

To run all tests without rebuilding: #+begin_src sh just test #+end_src

The justfile is configured (by default) to echo the commands it executes; it's syntax is very readable.

*** Developer Environments / Tooling :PROPERTIES: :CUSTOM_ID: h:C72FA3F7-3C23-4334-9590-5FF173EDB81C :END:

The =.jam= /directory/ includes a =Containerfile= (aka. =Dockerfile=) for container-based workflows. It is primarily written for use with [[https://podman.io/][Podman]] but should also work with Docker. It is currently written for macOS hosts running containers; i.e. an implicit Linux-VM which itself is actually managing containers.

The =jam= /file/ (shell-script) at the repository root can be used to build/create/run development containers. These are not based on "Development Containers" from Micro$oft.

To build a container image with all dependencies: #+begin_src sh ./jam make #+end_src

To create and run the built container image, replacing any that may exist: #+begin_src sh ./jam run #+end_src

This repo's files are bind-mounted into the container at =/home/jammy/project= so no work is lost when deleting/re-building/re-creating (etc) the container so long as you stick within that bind-mounted directory.

You can ssh into the container via: #+begin_src sh ./jam ssh #+end_src

**** Debugging

There are two approaches to debugging the grammar, if the rule in question involves an external scanner it is advised to use a debugger like =gdb=, =lldb=, or other appropriate command-line (optionally DAP-capable) debuggers. This is because you will want to step through execution and set breakpoints to examine what is happening. You can still use this approach if the rule does not involve an external scanner.

***** Command Line Debuggers / DAP

The included [[#h:C72FA3F7-3C23-4334-9590-5FF173EDB81C][Jam developer environment]] pre-configures a debugging-ready environment using =gdb= (which is DAP capable).

The easiest way to create a debuggable parser is to not attempt to attach to the tree-sitter CLI but to use the tree-sitter API via your language of choice (pre-provided is C-based [[./debug_parser.c][debug_parser]]) and set a breakpoint where =ts_parser_parse_string= (or equivalent) is called. From there you can step through the parser and examine behaviour.

Useful watchpoints can be found in [[./.gdbinit][gdbinit]] but at the minimum would be =(char)lexer->lookahead=.

***** Debug Logs

If debugging rules wholly constructed through the tree-sitter DSL you can pass =-0= to =tree-sitter generate= to compile the parser in debug mode, and then =-d= to =tree-sitter test= to output textual logging information.

./jam ssh just clean-all just bootstrap just debug

*** Org Files

**** Grammar

Noir has no formal language specification yet so I took to exploring the /entire/ Noir compiler frontend (parser, and lexer) to determine the language grammar. It is superficially similar to Rust's (as Noir's syntax is heavily inspired by it) but it is /not/ the same. This grammar has been created from the ground up through arguably too much time exploring said frontend but I believe it to be the proper way to get this done.

Part of that necessitated taking notes which are in the [[./noir_grammar.org][noir_grammar]] org-file. This file documents AST nodes (as defined in the Noir compiler frontend, herein: Noirc), their EBNF grammar rules (should they differ from their literal in-source documentation, if any), the hierarchical relationship between these AST nodes, and most importantly the tree-sitter grammar rules alongside.

So, the [[./grammar.js][grammar]] DSL file is created by tangling (processing and exporting) the noir_grammar file.

If you wish to contribute you may edit either the noir_grammar file or the resulting grammar.js.

**** Test

Tree-sitter test files do not allow comments. You can get them if you consider the target language's comments as viable but this then requires augmenting the CST for the test you wish you add comments to, and the comments are not in the right place regardless.

The [[./test.org][test]] org-file contains tree-sitter tests and also comments so I can explain to myself (and others) why a test is the way it is, any TODOs etc. This file is tangled (processed and exported) to create the test files tree-sitter expects at [[./test/corpus][./test/corpus]].

If you wish to contribute you may edit either the test org-file or the resulting corpus text files.

** Bindings

No bindings as currently generated by the tree-sitter CLI will be supported in this repo. Files are littered everywhere and they are extremely frustrating to deal with for version control. Assuming that their generation is deterministic one can create the bindings from any commit or tagged release by running:

#+begin_src sh just binding #+end_src

You will need to answer =y= at the prompt (required to prevent accidental generation).

** Misc

Tree-sitter requires a JS runtime (Node, Deno, or Bun) to create =grammar.json= from =grammar.js=, the former of which is used to auto-generate the C-based parser. Given this requirement I've chosen to install the tree-sitter CLI using said JS runtime (Bun specifically) as a small win on things-to-install-burden.

If you'd prefer Node, or Deno, or the standalone tree-sitter CLI then modify the justfile at the root of this repository accordingly.