Plan for New Modules Implementation

February 9, 2020 · View on GitHub

This document has been archived.

This document summarized the work that went into the new ECMAScript Modules implementation that shipped in Node.js 12.0.0.

Plan for New Modules Implementation

This document outlines the plan for building a new implementation to support ECMAScript modules in Node.js. The effort is split up into phases:

  • Phase 0 branches off of current Node but removes much of the Node 8.5.0+ --experimental-modules implementation so that a new implementation could be built in its place.

  • Phase 1 adds the “minimal kernel,” features that the modules working group felt would likely appear in any potential new ES modules implementation.

  • Phase 2 fleshes out the implementation with enough functionality that it should be useful to average users as a minimum viable product.

    • At the completion of Phase 2, the old --experimental-modules implementation was replaced with this new one (still behind the --experimental-modules flag). It was released as part of Node 12 on 2019-04-23.

  • Phase 3 improves user experience and extends the MVP.

    • At the completion of Phase 3, the new implementation’s experimental flag was dropped. It was released as part of Node 13.2.0 on 2019-11-21.

  • Phase 4 are items that were under development in earlier phases but weren’t finished when the new implementation’s experimental flag was dropped; these items may continue development after unflagging and potentially ship in later versions of Node.js.

The effort is currently in Phase 4

At every phase, the following standards must be maintained:

  • Spec compliance (#132): We must always follow the ES spec.
  • Browser equivalence (#133): There’s room for debate in specific cases, but in general if Node is doing something that browsers also do, Node should do it in the same way. Alternatively, code that executes in both environments should produce identical results.
  • Don’t break CommonJS (#112): We cannot cause breaking changes with regards to CommonJS.

See also the features list in the README.

Phase 0: Start Fresh

From current shipping Node, the following changes were made to strip out most of the Node 8.5.0+ --experimental-modules implementation so that a new implementation could be built in its place:

  • Remove support in the import statement of formats other than ESM:

    • No CommonJS.
    • No JSON.
    • No native modules.

  • Remove dynamic path searching:

    • No extension adding.
    • No directory resolution, including no support for index.js or index.mjs.
    • No support for main field for ESM.

  • Remove current VM implementation

  • Remove current Loader implementation

These changes were implemented in https://github.com/nodejs/ecmascript-modules/pull/6.

Phase 1: The Minimal Kernel

The “minimal kernel” consists of features that the @nodejs/modules group have agreed will be necessary for all potential iterations of our ESM implementation. Phase 1 does not include features that preclude other potential features or implementation approaches; and Phase 1 also does not include some features that should naturally be built in a later phase of development, for example because those features depend on features planned for Phase 1.

  • module.createRequireFromPath (nodejs/node#19360) is the only way to import CommonJS into an ES module, for now.

  • import statements will only support files with an .mjs extension, and will import only ES modules, for now.

    • No JSON or native modules; createRequireFromPath can be used to get these.

  • import.meta.url.

    • Already in the existing implementation.

  • Dynamic import().

    • Already in the existing implementation.

  • Support for built-in modules with named exports

    • Already in the existing implementation.

Phase 2: Minimum Viable Product: Required to Upstream

Phase 2 fleshes out the implementation with enough functionality that it should be useful to average users as a minimum viable product. At the completion of Phase 2, the old --experimental-modules implementation was replaced with this new one (still behind the --experimental-modules flag).

The work through the end of Phase 2 landed in Node.js master as part of https://github.com/nodejs/node/pull/26745 and was released in Node 12.0.0.

Phase 3: Path to Stability: Removing --experimental-modules Flag

Phase 3 improves user experience and extends the MVP. Phase 3 is malleable based on how things proceed while working on this phase. At the end of this phase, the --experimental-modules flag is dropped.

  • Better mechanism for creating require function: createRequire.

  • "exports" field: for consumers of a package, map the paths of deep imports to provide encapsulation (an explicit public API); pretty specifiers (no file exensions or paths that include things like dist/) and flexibility for future package versions renaming or moving files without affecting the package’s public API. Applies to both ESM and CommonJS.

  • Define behavior for builtin globals between CommonJS and ESM. Does modifying a builtin in one module system carry over into the other? If it does, we may have major performance concerns.

  • Shortcut to resolve to package root.

  • Finalize behavior of import of CommonJS files and packages.

  • Dual CommonJS/ESM packages: Support packages with both CommonJS and ESM sources that can be used in either environment.

    • At time of unflagging: "main" (or "exports": { ".": "file.js" } overriding "main") points to exactly one file, and full filenames are required (by default), so there is no possibility of an import specifier pointing to different files in ESM versus CommonJS; unless --experimental-conditional-exports is used (see next bullet). Without that flag, dual packages must provide secondary entry point via a path, e.g. 'pkg/module' or 'pkg/commonjs'.
    • With --experimental-conditional-exports, paths within the package.json "exports" block can have separate entry points per environment.
    • Landed in https://github.com/nodejs/node/pull/29978 and https://github.com/nodejs/node/pull/30051 behind --experimental-conditional-exports flag.
    • Unflagged in https://github.com/nodejs/node/pull/31001 and shipped in Node 13.7.0.

Phase 4: Further Improvements After Unflagging

Now that the implementation has shipped, further efforts are listed on the README.