Development

Repository Structure

This repository is split into multiple projects at the crates folder. Each project has its own dependencies and tools used to build, test, and ship different parts of the repository. For example, a Rust environment for the compiler, a Python environment for documentation, and a NodeJS environment for linters. This allows us to implement different binaries, APIs, and internal tools/scripts, and package/version them together, while having minimal inter-dependencies.

All dependencies should have exact full versions, and we can rely on tooling to automatically upgrade it over time. It allows us to have perfectly reproducible builds for every commit, a critical aspect for compilers, and developer tools in general.

Currently, the repository has the following projects:

• .changeset/: pending user visible changes not released yet.
• .devcontainer/: self-contained Docker image to develop, build, test, and publish.
• crates/:
  • infra/: contains the CLI used for development, and utilities to build/test/run/debug all other projects.
  • codegen/: language analysis APIs that convert input crates to output crates below.
    • ebnf/: EBNF snippet serializer for language definitions
    • generator/: Generates Rust source code from language definitions
    • spec/: Language specification generator
  • language/: Building blocks for how languages can be defined
  • metaslang/:
    • cst/: A generic library for building, navigating and querying concrete syntax trees
    • bindings/: Computes semantic language bindings from parsed source code
    • graph_builder/: Construct binding graphs from parsed source code
  • solidity/:
    • inputs/: Solidity language definition.
    • outputs/: different packages and artifacts produced from it.
      • cargo/crate: Rust runtime and API.
      • cargo/wasm: WASM bindings and runtime.
      • npm/package: NodeJS bindings and runtime.
• documentation/: mkdocs site to render project documentation.

Dev Containers

To make the developer experience as seamless and consistent as possible, we recommend using the VS Code devcontainer included in this repository. It is a light image that has the minimum required tools to build this project. If you are not familiar with containerized development, I recommend taking a look at the official VS Code guide. Using a devcontainer allows us to quickly setup/teardown the environment, and install/setup different dependencies for different projects, without polluting the local environment. In the future, it will enable us to include Windows and Mac OS specific images for cross-platform testing.

If you would like to still develop outside a container, this should still be possible, as the CI will guarantee that your changes are safe. We intend to keep the images to a bare minimum, and install most dependencies through scripts you can run locally. However, using a common development container means sharing and standardizing useful settings and extensions for the editor (VS Code), the terminal (zsh), and any other tools.

In the future, if we decide to enable code spaces, we can have a 1-click button to create and warm up a powerful dev machine to use in seconds, and running completely remote in a browser tab. It will make it trivial to switch between different versions and branches, or even use and debug multiple ones at the same time from different tabs.

Hermit

To install language-specific binaries and packages, we use Hermit, which installs all tools only when it is first needed/invoked, so you can quickly setup and build different projects quickly. It also takes care of updating your $PATH as you cd in and out of different projects, to make sure you are using the right tools every time. Follow this guide to install it locally to your machine, or simply build any included project, and it will bootstrap itself if it is missing.

Infra CLI

To ensure consistency, and a good experience for first-time developers, all build/test/run/debug commands should be written, versioned, and documented inside the infra_cli crate. This means that any dev instructions are well documented, versioned, and verified/executed with every build. It also means that we can minimize any manual setup or teardown steps during development, and just rely on that cli.

You can access all such commands (from the hermit environment) by just running the infra script, which just refers to $REPO_ROOT/scripts/bin/infra. If this is your first time contributing, we recommend starting with infra --help to familiarize yourself with its capabilities.

Versioning and Publishing

We manage versioning through changesets. Each pull request can describe what user facing changes it is introducing, and include this information as a "changeset" markdown file along with source changes. These changeset files are analyzed and used to create another pull request to bump the repository version and update dependencies.

Once the version bump is merged, .github/workflows/publish.yml runs three jobs in sequence:

1. prepare — builds the npm tarball (pnpm pack) into target/publish-artifacts/ and uploads it as a workflow artifact (retained for 7 days). On the version-bump merge, this is what's about to ship; on every other PR/push, prepare writes a SKIPPED marker because the local version already matches what's on npm.
2. review — downloads the prepared artifact and runs pnpm publish --dry-run against the tarball plus a batched cargo publish --dry-run. The cargo dry-run packages every crate, verify-builds each one in isolation from the extracted .crate, and resolves internal path-deps against the bumped versions — the closest dress rehearsal short of the actual upload.
3. publish — gated by the slang-release environment (manual approval). Uploads the prepared npm tarball to npm and runs cargo publish --no-verify per crate against crates.io, both via OIDC trusted-publishing tokens.

For npm, the bytes a reviewer signs off on are exactly the bytes that get uploaded — publish ships the same tarball prepare built. For cargo there is no equivalent artifact handoff: cargo publish re-packages from source on upload. The review step exercises the same packaging and verify-build code paths the publish step uses, so the gap is "no byte-level handoff", not "untested code at publish time". --no-verify at publish keeps the workspace build.rs and proc-macros from running alongside the crates.io OIDC token.

Managing Dependencies

Automated dependency updates are handled by Renovate via $REPO_ROOT/renovate.json. The config runs on a weekly schedule (Mondays) and covers cargo, npm, pipenv, github-actions, hermit, and dockerfile dependencies.

Grouping strategy: minor and patch updates are grouped into a single PR per ecosystem to reduce review overhead. Major-version bumps arrive as individual PRs, since they often require code changes. GitHub Actions updates are grouped including majors.

Benchmark dependencies: tree-sitter, tree-sitter-solidity, and solar are grouped into a dedicated PR labeled ci:perf, since version changes can affect benchmark baselines. This keeps them isolated from other dependency updates so regressions are easy to attribute.

Security: Renovate opens vulnerability-fix PRs immediately (bypassing schedule and cooldown) using both GitHub's Security Advisory database and OSV.

The following still need to be updated manually:

1. Rust toolchains: $RUST_STABLE_VERSION and $RUST_NIGHTLY_VERSION defined in hermit.hcl and updated via rustup install.
2. Node.js major versions (only even LTS releases): bump the hermit package, then @types/node follows automatically.

Supply-Chain Firewall (Socket)

CI installs dependencies behind Socket Firewall (sfw) in firewall mode, which intercepts package-manager traffic (cargo, npm, pip) and blocks packages Socket flags as malicious. The setup-sfw composite action installs it and exports SFW_PREFIX, which prefixes the dependency-fetching steps (infra setup, infra check, and the cooldown and renovate checks). It runs only in CI — local infra commands are unaffected.

Required by default. If Socket Firewall can't be installed (e.g. a socket.dev outage), the Setup SFW step fails the job rather than let dependency installs run unprotected, so we never publish dependencies that were installed unprotected. Jobs where the firewall is best-effort — PR/benchmark CI, and release jobs that only consume artifacts already built behind SFW — opt out with optional: "true", which logs a ::warning:: and continues unprotected instead.

When a build fails because of Socket

The two failure modes look different:

• SFW unavailable (not a block). The Setup SFW step fails with ::error::sfw not found after install … (or, on optional: "true" jobs, logs a ::warning:: and continues unprotected). This is a Socket/network availability problem, not a flagged package — re-run once Socket is reachable.
• A package was blocked. A wrapped step (usually infra setup) fails and the sfw output names the offending package, version, and why Socket flagged it. This is the firewall doing its job. Note that firewall mode blocks transitively — a flagged package deep in the dependency tree fails the whole install, even if you didn't add it directly.