zsh-completion-sync

A zsh plugin that automatically loads completions added dynamically to FPATH or XDG_DATA_DIRS

What does it do?

Anytime after the start of your shell if new entries have been added to your FPATH, this plugin will reload your completion system to make the new completions functions immediately available. It also loads extra paths from $XDG_DATA_DIRS to comply with the XDG spec. This plugin is especially useful/important for users of dynamic development shells via tools like direnv, nix-direnv, nix develop, which dynamically put specific versions of executables on the front of $PATH and typically surface matching completion scripts via $XDG_DATA_DIRS

Motivation

Normally zsh only builds the pairing of executable name and completion functions once at shell startup, when compinit is called, typically while sourcing ~/.zshrc. For this, only directories on $FPATH are considered. Any additions to $FPATH after this point, will make the functions themselves available for autoloading, but the completion system will not consider them anymore. This means that any tools that are installed non-permanently, will never have their completions picked up, since they will generally not be on the startup $FPATH (or $PATH for that matter). Users of nix and direnv are hit especially hard by this, since they rely on non-permanent changes to the running shell or shells with dynamically generated $PATH. This plugin bridges that still unresolved need for zsh users. (Note, for fish users, there is a similar plugin here)

Installation

When sourced, this plugin only defines functions and registers hooks. The hooks are only executed the first time the prompt is rendered. Therefore it is not very sensitive to load order, however it should be loaded late to ensure that it only runs after all other hooks that could affect the relevant env vars have run.

Quickstart

Nix mkShell (via nix-direnv or nix develop)

Should work out of the box.

Most packages have shell completion scripts for bash,zsh, and fish available in their share directory. mkShell automatically adds their share directories to $XDG_DATA_DIRS, so if they ship a zsh/site-functions dir in them (i.e. they used installShellCompletion --zsh, which is most), this plugin will add them to the $FPATH and make the functions available for completion.

nix-shell -p

Currently, nix-shell does not currently set nativeBuildInputs (https://github.com/NixOS/nix/issues/4254) and therefore does not set/expand $XDG_DATA_DIR. This plugin offers a slightly hacky workaround which tries to discover fpaths from entries on $PATH. You can enable it by setting zstyle ':completion-sync:path' enabled true. Since this behaviour does not conform to the meaning of the $PATH variable (as opposed to $FPATH and $XDG_DATA_DIR) it is not enabled by default. Putting relevant packages into nativeBuildInputs (once supported) is the preferred way.

direnv (without nix)

Ensure that .envrc exports an extended $XDG_DATA_DIRS. each path should contain a folder zsh/site-functions or zsh/$ZSH_VERSION/functions, (often this path also contains completion scripts for other shells and is therefore shell agnostic and preferrable over setting fpath directly). Alternatively, expand $FPATH with the folders containing the completion scripts (Remember that direnv uses a bash subshell, so you need to use the scalar $FPATH and not the array $fpath)

How does it work?

This plugin registers a hook for both precmd (run each time before the prompt is displayed) and chpwd (run when changing directories). It registers itself at the end of the hook chain when it is loaded, in the hope of going last after all hooks that might change the env vars (especially after the dir env hook).

XDG_DATA_DIRS sync

The first time this feature loads, it will enumerate all subpaths of XDG_DATA_DIRS that contain zsh functions ($dir/zsh/site-functions, $dir/zsh/$ZSH_VERSION/functions, or $dir/zsh/vendor-completions which is a non-standard path used by debian packages) and then prepend all of these paths that are not on the FPATH yet to the FPATH. This avoids overriding custom overrides for internal functions again by putting the zsh install directory on the front of the path, because it is also reachable via $XDG_DATA_DIRS.

Then everytime $XDG_DATA_DIRS changes, the plugin then enumerates the zsh function subpaths again and then diffs that against the last state of the function dirs. It then adds/removes these from fpaths as indicated by the diff. Note, that unlike in the initialization, this will always prepend or remove the first occurence of a path from the fpath. If a directory is dynamically added during runtime, we assume that the user wants it to take priority.

PATH sync

The PATH sync feature is a workaround for nix-shell -p, which puts the bin/ folder of nix a set of nix derivations on the PATH, but doesn't export their share/ directory on $XDG_DATA_DIRS. (This is in tune with how nix works, since only nativeBuildInputs and not packages are meant to export their data dirs). In nix, these share/ dirs are siblings to the bin/ path on $PATH and if they have a corresponding zsh completions, those are installed in share/zsh/site-functions. So, to discover these fpaths, the plugin tests for the existence of a directory at $p/$relPath for every $p from $PATH and $rp from the configured array of relative paths. The default works for discovering any zsh FPATHs in nix derivations on $PATH. The resultant array of paths is then added to the front of $FPATH (preserving priority order from $PATH) if the relevant path if not on $FPATH yet. When there is a change in $PATH, the plugin builds the array from the current $PATH again and diffs it against the last state, adding or/removing to/from the front of $FPATH as needed.

zsh-autocomplete Compatibility

Since zsh-autocomplete includes very heavy customization of the zsh compsys, we cannot simply use the existing mechanism of calling compinit again. Instead we need to source (i.e. initialize) all of zsh-autocomplete again. To accomplish this, the plugin tries to detect if a named directory ~autocomplete exists. (ZAC defines this directory and points it at the directory containing it's plugin file). We then attempt to source a file named zsh-autocomplete.plugin.zsh in this directory. If such a file is found, sourcing it again replaces our calls to compinit with. This detection is pretty robust. Should it ever fail, it can either be turned off by setting zstyle ':completion-sync:compinit:compat:zsh-autocomplete' enabled false. Alternatively, you can directly set the path to the zsh-autocomplete plugin file by using a custom hook

In the future we hope that the integration can be made more lightweight with specific functions to just reload the matching of commands and completion functions, or even allow for adding/removing specific completion entries.

Options

You can use zstyle to control the behaviour of the plugin. This includes enabling/disabling certain feature-sets and fine-grained control over debug logging.

XDG_DATA_DIRS sync

To turn the XDG_DATA_DIRS feature on or off, set zstyle 'completion-sync:xdg' enabled (default enabled). Note that disabling this feature during runtime will not remove the dirs added to the fpath at startup, it will only pause the syncing. To avoid adding paths from $XDG_DATA_DIRS to $FPATH, set the zstyle in .zshrc (i.e. before the first prompt is rendered)

PATH to FPATH sync / nix-shell compat

To enable detecting zsh function paths from the binary search path, set zstyle :completion-sync:path enabled true (default false).

The plugin searches the relative paths indicated in the array set in zstyle :completion-sync:path rel_dirs (default: ("../share/zsh/$ZSH_VERSION/functions" '../share/zsh/site-functions')). The default is setup based on the typical nix derivation structure, which has its share//XDG_DATA_DIR as a sibling to the bin/ directory on the path.

Note that if you want nix-shell to put you into zsh by default and only need support for nix-shell compatibility, you should install this plugin for compatibility which does put nix fpaths onto path already and does not require this plugin's workaround.

Custom Pre- and Post- Hooks

This plugin allows running custom hooks just before and/or after reloading compinit. This is especially helpful to reintroduce changes which where overridden by compinit reinvocation. (For example zsh-autocomplete binds tab to its completion function, but the user might want to use tab for something else or use something like fzf as entrypoint to the completion functions)

zstyle ':completion-sync:compinit:custom:pre-hook' enabled true
zstyle ':completion-sync:compinit:custom:pre-hook' command 'this string will be eval'ed'

zstyle ':completion-sync:compinit:custom:post-hook' enabled true
zstyle ':completion-sync:compinit:custom:post-hook' command 'this string will be eval'ed'

Priority of custom compinit options

The following options all control how the zsh compsys is reloaded when a change in fpath is detected. These options take priority in the following order

1. Custom (default: disabled)
2. zsh-autocomplete (if enabled and zsh-autocomplete found, default: enabled)
3. Builtin compinit (default: disabled)
4. compinit (using the currently loaded compinit)

Custom Command

If the existing options are not sufficient or you want to fine tune how compsys is reloaded, you can set a custom command

zstyle ':completion-sync:compinit:custom' enabled true
zstyle ':completion-sync:compinit:custom' command 'this string will be eval'ed'

Compinit arguments

When using zsh-autocomplete, default or builtin compinit, you can use zstyle ':completion-sync:compinit' arguments to set an argument of arguments to use when invoking compinit. Note that -d /path/to/compdump is managed outside of this zstyle and should not be set inside arguments

Example: This will disable compdumping, and fpath testing for existing compdump files, both of which are expensive processes

zstyle ':completion-sync:compinit' arguments -D -C

zsh-autocomplete Compatibility

The plugin zsh-autocomplete (ZAC) customizes the behaviour of zsh's compsys to an extreme degree. it does this so much, that any calls to the builtin compinit will break the plugin in subtle to overt ways. As a protection, the plugin disables compinit completely after it has loaded (see Builtin Compinit for details). This leaves us without a good option on how to reload our completions system. While forcing use of the builtin compinit function allows reloading, this will lead to aforementioned breakage. Instead, we can make use of the fact, that zsh-autocomplete initialization is idempotent and source the complete plugin again to ensure that zsh-autocomplete re-initializes on the current FPATH

The automagic way

By default, this plugin will try to detect the installation path of zsh-autocomplete for you.

zstyle ':completion-sync:compinit:compat:zsh-autocomplete' enabled true # default: true

The hard way

In order to re-initialize ZAC , this plugin needs to know where ZAC is installed

zstyle ':completion-sync:compinit:custom' enabled true
zstyle ':completion-sync:compinit:custom' command 'source <path to zsh-autocomplete.plugin.zsh>'

Builtin Compinit

Caution: This is hacky and will probably break something.

zstyle ':completion-sync:compinit:builtin-compinit' enabled true #default: false

Some plugins patch or otherwise modify the compinit function. this can leave us unable to access the functionality necessary for reloading the completion system. As a workaround, this option, if enabled will first copy the existing compinit function out of the way and autoload compinit from the fpath again. The result of this is almost always the builtin compinit or a function meant to act as a drop-in replacement and therefore safe to call. Note that while this will bruteforce re-enable the ability to reload completions, there is often a very good reason why plugins patch compinit. Sometimes, it is only the goal of improving performance ensuring that multiple calls to compinit in .zshrc via different plugins do not slow down shell start. In this case, where compinit was no-op'ed for performance, it is safe to enable this option. On the other hand, plugins like zsh-autocomplete (which made me originally write this workaround) no-op the function, because it's default behaviour will break zsh-autocomplete in subtle ways. Using builtin-compinit with it gets completions for new commands, but at degraded autocomplete functionality across the board.

Debug Logging

Examples:

  # Enable debug logging by default
  zstyle ':completion-sync:*' debug true
  # Turn off the very verbose line diffs
  zstyle ':completion-sync:*:diff' debug false
  # Turn off debug logging about candidate paths
  zstyle ':completion-sync:**:candidate' debug false
  zstyle ':completion-sync:**:candidate:*' debug false

You can control the debug logging very precisely if you need to, but for most use cases the options above are sufficient. If you need more precision, search for :completion-sync: in the source.

Optimizations

Zsh's completion system is necessarily opiniated because it was designed and built in a very different time, including where installed packages/binaries didn't change quickly and cpu time was sparse compared to today. Design decisions (especially around caching) made sense then, but are standing in the way of performant dynamic updates to the completion system. To regain some performance this plugin uses a few optimizations. These are:

fast-add

When adding entries to the (front of) the FPATH, we can optimize the performance of the plugin by skipping a full reload of the compinit system and only manually adding in any compdef and autoload calls that a related to the added entries on the FPATH. This should still lead to the exact same state, since adding completion definitions to the front of the fpath will mean that they become the preferred function definition for said completion. When removing entries from the FPATH, a full reload is still required.

Overall, the optimization is crude because it can lead to a lot of independent compdef calls, therefore it is still considered experimental for now.

This optimization has graduated from expiremntal and is enabled by default, to disable this optimization, use

zstyle ':completion-sync:compinit:optimizations:fast-add' enabled false

no-caching

zsh's compinit caching is very limited, it implicitly assumes that the set of completions changes very infrequently and not during the lifetime of a shell (basically whenever a user installs packages or changes their .zshrc). Therefore caching only works if the compdump file is an exact match for the rarely changing fpath. Inside of the circumstances where this plugin is typically used, this will mostly lead to unnecessary churn due to almost constant cache-misses. This option sets the compdump location ($_comp_dumpfile and similar variables) to /dev/null, and adds -D -C to the compinit arguments, forcing instant cache misses without extraneous checks, and disables constructing a useless compdump file

This optimization has graduated from expiremntal and is enabled by default, to disable this optimization, use

zstyle ':completion-sync:compinit:optimizations:no-caching' enabled false

copy-compdump (experimental)

By default, this plugin uses a separate, initially empty zcompdump file per shell to avoid temporary shell environments polluting the user-global compdump file. With the experimental "copy-compdump" option, each shell will start out with a copy of the user-global compdump file and the plugin will attempt to maintain an up-to-date user-global compdump file by copying the cache dump back to the user-global spot when run inside a login shell (since a login shell should never start out in a temporary shell environment).

Given the limitations of zsh's caching system it is unclear if there are many usecases where this optimization will help much. Most use-cases will be sped up more by using no-caching.

Disabled if no-caching is enabled

To enable this optimization, use

zstyle ':completion-sync:compinit:experimental:copy-compdump' enabled true

frozen-first-cache (experimental)

(only relevant if no-caching is also enabled).

Modifies the logic of no-caching, so that the compdump file path is not set to /dev/null until just before this plugin's precmd hook runs for the first time. This allows for an optimization in which the first compinit call (which should always lead to the same state of the compsys unless .zshrc changed) is loaded from cache for a faster startup. Afterwards, behaves exactly like no-caching (i.e. forces full reloads to traverse the $FPATH without caching).

Note that when using this option, you need to handle cache-invalidation yourself, otherwise your completions might be incomplete or broken on shell start.

Most use cases will not need this option and can simply ensure that zsh-completion-sync is initialized after the synchronous call to compinit [-C -D] -d /path/to/compdump, however when using zsh-autocomplete, you should not call compinit directly, but let ZAC call it in its initialization hook. With no-caching, the compdump path will then already have been set to /dev/null, negating the benefit of having a properly maintained cache file for startup.

TL;DR: If you use a plugin which runs compinit late in a precmd hook (like zsh-autocomplete), use this option, and make sure this plugin is initialized after the compinit calling plugin

To enable this optimization, use

zstyle ':completion-sync:compinit:experimental:frozen-first-cache' enabled true