#+TITLE: TRAMP-RPC #+AUTHOR: Arthur Heymans arthur@aheymans.xyz #+OPTIONS: toc:2

A high-performance TRAMP backend for Emacs that uses a binary RPC server instead of parsing shell command output.

• Overview

Traditional TRAMP works by piping shell commands over SSH and parsing their output. This approach is robust but slow, especially for operations that require many round-trips (like directory listings or VC operations).

TRAMP-RPC replaces this with a lightweight Rust server that runs on the remote host. Emacs communicates with it using MessagePack-RPC over SSH, resulting in significantly faster file operations.

** Why TRAMP-RPC?

| Aspect | Original TRAMP | TRAMP-RPC | |------------------+--------------------------+--------------------------| | Communication | Shell commands + parsing | MessagePack-RPC protocol | | Latency | Multiple round-trips | Single round-trip | | Batching | Not supported | Multiple ops per request | | Shell dependency | Required on remote | Not needed | | Binary required | None | ~850KB Rust server |

• Features

• Fast file operations via binary RPC protocol (2-38x faster than shell-based TRAMP)
• Async process support (make-process, start-file-process)
• Full VC mode integration (git, etc.)
• Magit/Projectile optimizations with parallel git command prefetch
• Automatic binary deployment (download or build from source)
• Support for Linux and macOS (x86_64 and aarch64)
• Batch/pipelined requests for reduced round-trip latency
• Multi-hop support via SSH ProxyJump
• Filesystem watching with automatic cache invalidation
• PTY support for terminal emulators (vterm, eat)

• Requirements

• Emacs 30.1 or later
• Tramp 2.8.1.4 or later (install from GNU ELPA if your Emacs bundles an older version)
• =msgpack.el= 0.1.1 or later (installed automatically from MELPA)
• SSH access to remote hosts
• Remote host running Linux or macOS (x86_64 or aarch64)

• Installation

** From NonGNU ELPA (coming soon)

#+begin_src elisp (use-package tramp-rpc :ensure t) #+end_src

** From Git (Emacs 30+)

#+begin_src elisp (use-package tramp-rpc :after tramp :vc (:url "https://github.com/ArthurHeymans/emacs-tramp-rpc" :rev :newest :lisp-dir "lisp")) #+end_src

** Manual Installation

1. Clone this repository: #+begin_src bash git clone https://github.com/ArthurHeymans/emacs-tramp-rpc.git #+end_src

2. Add to your Emacs init file: #+begin_src elisp (add-to-list 'load-path "/path/to/emacs-tramp-rpc/lisp") (require 'tramp-rpc) #+end_src

** For Doom Emacs

1. In =packages.el=: #+begin_src emacs-lisp (package! msgpack) (package! tramp-rpc :recipe (:host github :repo "ArthurHeymans/emacs-tramp-rpc" :files ("lisp/*.el"))) #+end_src

2. In =config.el=: #+begin_src emacs-lisp (use-package! msgpack) (use-package! tramp-rpc) #+end_src

** From Nix flake

1. Include this repository as a flake input and add its overlay to nixpkgs: #+begin_src nix inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; emacs-tramp-rpc.url = "github:ArthurHeymans/emacs-tramp-rpc"; };

   outputs = inputs@{ nixpkgs, emacs-tramp-rpc, ... }: let system = "x86_64-linux"; inherit (nixpkgs) lib; pkgs = import nixpkgs { inherit system; overlays = [ emacs-tramp-rpc.overlays.default ]; }; in ...; #+end_src

2. Include tramp-rpc as you would any Elisp package from the emacsPackages scope, e.g. in Home Manager's programs.emacs.extraPackages: #+begin_src nix programs.emacs.extraPackages = epkgs: [ epkgs.tramp-rpc ]; #+end_src

3. Server binaries for =x86_64-linux= and =aarch64-linux= are automatically built and included in the Emacs package derivation; if you would like to change which systems are built, override the archs input: #+begin_src nix programs.emacs.extraPackages = epkgs: [ (epkgs.tramp-rpc.override { archs = [ pkgs.pkgsCross.riscv64-musl (import nixpkgs { inherit system; crossSystem = lib.systems.elaborate { config = "armv6l-unknown-linux-musleabihf"; }; }) ]; }) ]; #+end_src

• Usage

Access remote files using the rpc method:

#+begin_example /rpc:user@host:/path/to/file #+end_example

On first connection, the server binary is automatically obtained and deployed:

1. Release/package installs: download from GitHub Releases first (fastest, ~850KB download), then build from source if Rust is installed and download fails.
2. Git checkout installs: reuse a fresh source build or source-tree keyed cache. Automatic file operations do not prompt when a binary is missing; run M-x tramp-rpc-deploy-install-binary to choose whether to download a checksum-verified release binary, build the checkout with Cargo, or skip. A downloaded fallback remains keyed to that source tree, while strict build policy uses a separate identity.

The binary is cached locally in =/.emacs.d/tramp-rpc/= and deployed to =/.cache/emacs/tramp-rpc/= on the remote host.

** Deployment Commands

| Command | Description | |----------------------------------+--------------------------------| | M-x tramp-rpc-deploy-install-binary | Choose and deploy a missing git-checkout binary; use C-u to replace one | | M-x tramp-rpc-deploy-status | Show binary deployment status | | M-x tramp-rpc-deploy-clear-cache | Clear local binary cache | | M-x tramp-rpc-deploy-remove-binary | Remove binary from remote |

• Architecture

#+begin_example ┌─────────────┐ SSH/MessagePack-RPC ┌──────────────────┐ │ Emacs │ ◄────────────────────► │ tramp-rpc-server │ │ (tramp-rpc) │ │ (Rust) │ └─────────────┘ └──────────────────┘ #+end_example

** Module Organization

The Emacs Lisp client is organized into focused modules:

| Module | Lines | Purpose | |----------------------------+-------+--------------------------------------------| | tramp-rpc.el | ~2450 | Core RPC communication & file handlers | | tramp-rpc-process.el | ~1050 | Async process & PTY support | | tramp-rpc-magit.el | ~780 | Magit/Projectile optimizations & caching | | tramp-rpc-deploy.el | ~950 | Binary deployment & version management | | tramp-rpc-advice.el | ~330 | Advice functions for process/VC integration| | tramp-rpc-protocol.el | ~190 | MessagePack-RPC protocol implementation |

*** Core Module (tramp-rpc.el)

The core module provides:

• SSH ControlMaster connection management
• RPC call/batch/pipeline primitives
• Direnv environment caching for processes
• All TRAMP file handler operations (file-exists-p, file-attributes, insert-file-contents, etc.)
• File name handler registration and dispatch

*** Process Module (tramp-rpc-process.el)

Handles all process-related functionality:

• Async pipe processes (make-process, start-file-process)
• RPC-based PTY processes for terminal emulators
• Terminal resize handling for vterm/eat/shell-mode
• Process I/O queuing with async callback-based reading
• Adaptive polling for long-running processes

*** Magit Module (tramp-rpc-magit.el)

Optimizations for git/magit/projectile on remote hosts:

• Parallel git command prefetch via commands.run_parallel RPC (sends ~60+ git commands in a single round-trip)
• TTL-based caches for file-exists and file-truename with max-size eviction
• Filesystem watch management via server push notifications for cache invalidation
• Process-file cache for serving git commands from prefetched data
• Ancestor directory scanning for fast project/VC root detection
• Projectile integration (force git ls-files, alien indexing for remote)

*** Advice Module (tramp-rpc-advice.el)

Centralizes all advice functions:

• Process I/O: process-send-string, process-send-region, process-send-eof
• Process info: signal-process, process-status, process-exit-status
• Process metadata: process-command, process-tty-name
• VC integration: vc-call-backend for proper default-directory handling
• Eglot integration: Bypass shell wrapping for RPC connections

** RPC Server Methods

The dispatcher exposes these public methods:

| Category | Method | Description | |----------+--------+-------------| | Batch | batch | Execute up to 64 requests with bounded concurrency. | | File | file.stat | Return file metadata. | | File | file.truename | Resolve a file's canonical path. | | Directory | dir.list | List directory entries, optionally with attributes. | | Directory | dir.create | Create a directory. | | Directory | dir.remove | Remove a directory. | | File I/O | file.read | Read file bytes. | | File I/O | file.write | Write file bytes. | | File I/O | file.copy | Copy a file. | | File I/O | file.rename | Rename a file. | | File I/O | file.delete | Delete a file. | | File I/O | file.set_modes | Set file mode bits. | | File I/O | file.set_times | Set file timestamps. | | File I/O | file.make_symlink | Create a symbolic link. | | File I/O | file.make_hardlink | Create a hard link. | | File I/O | file.chown | Change file ownership. | | Process | process.run | Run a command synchronously. | | Process | process.start | Start a managed pipe process. | | Process | process.write | Write to managed process stdin. | | Process | process.read | Read managed process output. | | Process | process.status | Return managed process status. | | Process | process.close_stdin | Close managed process stdin. | | Process | process.kill | Signal a managed process. | | Process | process.list | List managed processes. | | PTY | process.start_pty | Start a pseudo-terminal process. | | PTY | process.read_pty | Read pseudo-terminal output. | | PTY | process.write_pty | Write pseudo-terminal input. | | PTY | process.resize_pty | Resize a pseudo-terminal. | | PTY | process.kill_pty | Signal a pseudo-terminal process. | | PTY | process.close_pty | Close a pseudo-terminal process. | | PTY | process.list_pty | List pseudo-terminal processes. | | System | system.info | Return server and host information. | | System | system.getenv | Read an environment variable. | | System | system.expand_path | Expand home-directory paths. | | System | system.statvfs | Return filesystem capacity information. | | System | system.groups | Return supplementary groups. | | Commands | commands.run_parallel | Run commands concurrently. | | Commands | ancestors.scan | Scan ancestor directories. | | High-level | highlevel.test_files_in_dir | Find named files in a directory. | | High-level | highlevel.locate_dominating_file_multi | Find an ancestor containing one of several names. | | High-level | highlevel.dir_locals_find_file_cache_update | Update directory-local file cache data. | | Watch | watch.add | Add a filesystem watch. | | Watch | watch.remove | Remove a filesystem watch. | | Watch | watch.list | List filesystem watches. |

• Binary Deployment

** How It Works

#+begin_example User connects via /rpc:host:/path │ ▼ Remote already has binary? ──yes──► Done │ no ▼ Check local cache (~/.emacs.d/tramp-rpc/VERSION/ARCH/) │ ├─ Found ──────────────────► Transfer to remote │ ▼ Download from GitHub Releases │ ├─ Success ────────────────► Cache locally, transfer to remote │ ▼ Build with cargo (if Rust installed) │ ├─ Success ────────────────► Cache locally, transfer to remote │ ▼ Error with helpful instructions #+end_example

** Supported Platforms

| Platform | Architecture | Status | |----------------+--------------+--------| | Linux | x86_64 | ✓ | | Linux | aarch64 | ✓ | | Linux | i686 | ✓ | | Linux | armv7 | ✓ | | Linux | armv5te | ✓ | | Linux | arm/ARMv6 | ✓ | | macOS | x86_64 | ✓ | | macOS (Apple Silicon) | aarch64 | ✓ |

** Manual Binary Installation

If automatic deployment fails, download manually from [[https://github.com/ArthurHeymans/emacs-tramp-rpc/releases][GitHub Releases]], extract the archive, and place the binary on the remote host at:

#+begin_example ~/.cache/emacs/tramp-rpc/tramp-rpc-server-VERSION #+end_example

For example: #+begin_example ~/.cache/emacs/tramp-rpc/tramp-rpc-server-0.9.0 #+end_example

• Building from Source

** Using Nix (recommended)

#+begin_src bash

Build for current platform

nix build

Cross-compile for specific target

nix build .#tramp-rpc-server-x86_64-linux nix build .#tramp-rpc-server-aarch64-linux

Development shell with all tools

nix develop #+end_src

** Using Cargo

#+begin_src bash cd server cargo build --release #+end_src

The binary will be at =target/release/tramp-rpc-server=.

** Cross-compilation with Cargo

#+begin_src bash

Install target

rustup target add aarch64-unknown-linux-gnu

Build (requires appropriate linker)

cargo build --release --target aarch64-unknown-linux-gnu #+end_src

• Configuration

#+begin_src elisp ;; Prefer building from source over downloading for release/package installs ;; (default: nil) (setq tramp-rpc-deploy-prefer-build t)

;; Git checkout policy (default: auto): ;; - auto: reuse source-keyed artifacts; use ;; M-x tramp-rpc-deploy-install-binary when one must be obtained; ;; use C-u M-x tramp-rpc-deploy-install-binary to replace an existing one ;; - build: strictly build from source, using a build-only binary identity ;; - release: use release-version binaries and paths (setq tramp-rpc-deploy-git-build-policy 'build)

;; Local cache directory (default: /.emacs.d/tramp-rpc/) (setq tramp-rpc-deploy-local-cache-directory "/.cache/emacs/tramp-rpc-binaries")

;; Remote installation directory (default: /.cache/emacs/tramp-rpc) (setq tramp-rpc-deploy-remote-directory "/.local/bin/tramp-rpc")

;; Disable automatic deployment (default: t) (setq tramp-rpc-deploy-auto-deploy nil)

;; Use different GitHub repo for downloads (setq tramp-rpc-deploy-github-repo "myuser/my-fork")

;; Download timeout in seconds (default: 120) (setq tramp-rpc-deploy-download-timeout 60) #+end_src

** Deployment fallback policy

When the expected remote binary already exists and is executable, TRAMP-RPC reuses it if no trusted local artifact can be obtained (for example because download, build, or local-cache access is unavailable). This fallback never covers a missing remote binary. Whenever a trusted local artifact is available, TRAMP-RPC compares SHA256 digests and either reuses the match or, with automatic deployment enabled, replaces a mismatch through the verified staging-and-activation operation. With automatic deployment disabled, a verified mismatch remains an explicit error.

** Process compatibility notes

• RPC PTYs disable kernel ECHO, ECHONL, and ONLCR. Emacs terminal consumers such as comint and eat perform local echo and line handling; code that relies directly on kernel echo or CRLF conversion will observe different PTY semantics.
• RPC PTY writes wait for the remote write acknowledgement and can therefore block for up to the RPC timeout when the remote program stops reading. Pipe writes remain queued by default; set tramp-rpc-synchronous-pipe-writes non-nil to make them synchronous too.

• Troubleshooting

** Check deployment status

Run M-x tramp-rpc-deploy-status to see:

• Current version
• Local architecture
• Whether Rust/cargo is available
• Cached binaries
• Download URLs

** diff-hl issues in dired

If you experience issues with diff-hl in dired buffers on remote hosts:

#+begin_src elisp (setq diff-hl-disable-on-remote t) #+end_src

** Connection issues

The server binary is deployed using standard SSH (scpx method by default). Ensure you can connect to the remote host with:

#+begin_src bash ssh -o BatchMode=yes user@host echo success #+end_src

** Download failures

If GitHub downloads fail (corporate firewall, etc.), you can:

1. Install Rust and let tramp-rpc build locally: #+begin_src bash curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh #+end_src

2. Download manually and place in the cache directory (see above)

3. Pre-deploy to remote hosts using your own method

• Protocol

TRAMP-RPC uses MessagePack-RPC over stdin/stdout with length-prefixed binary framing.

** Protocol History

TRAMP-RPC originally used JSON-RPC with newline-delimited messages. This was changed to MessagePack-RPC for several reasons:

| Aspect | JSON-RPC (old) | MessagePack-RPC (current) | |---------------------+-----------------------------+-------------------------------| | Binary data | Base64 encoded (~33% overhead) | Native binary type (no overhead) | | Message framing | Newline-delimited | Length-prefixed binary | | Non-UTF8 filenames | Required escaping/encoding | Native binary support | | Boolean false | JSON false | MessagePack false (0xc2) | | Message size | Larger (text format) | ~33% smaller (binary format) |

The switch eliminates encoding overhead for file transfers and fixes edge cases with non-UTF8 filenames that are valid on Unix filesystems.

** Why MessagePack?

MessagePack is a binary serialization format that provides:

• Native binary data support (no base64 encoding needed for file content)
• ~33% smaller messages compared to JSON
• Faster serialization/deserialization
• Proper distinction between null and false values

** Framing

Each message is prefixed with a 4-byte big-endian length:

#+begin_example <4-byte length> #+end_example

** Message Format

Request (conceptual structure): #+begin_src elisp ((version . "2.0") (id . 1) (method . "file.stat") (params . ((path . "/etc/passwd")))) #+end_src

Response (conceptual structure): #+begin_src elisp ((version . "2.0") (id . 1) (result . ((type . "file") (size . 2847) (mode . 420)))) #+end_src

** Binary Data

File content, paths, and process I/O are transmitted as raw binary (MessagePack bin type), eliminating encoding overhead and ensuring correct handling of:

• Non-UTF8 filenames
• Binary file content
• Arbitrary byte sequences in process output

• Performance

TRAMP-RPC significantly outperforms traditional TRAMP for most operations:

| Operation | RPC (median) | SSH (median) | Speedup | |------------------------+--------------+--------------+---------| | connection-setup | 31 ms | 1.17 s | 38.2x | | file-exists | 3.3 ms | 38.8 ms | 11.9x | | file-attributes | 3.4 ms | 23.3 ms | 6.8x | | dir-files-and-attrs | 3.5 ms | 95.9 ms | 27.1x | | file-read | 7.7 ms | 20.2 ms | 2.6x | | file-write | 74.2 ms | 219.9 ms | 3.0x | | directory-files | 12.1 ms | 37.2 ms | 3.1x | | copy-file | 43.1 ms | 192.0 ms | 4.5x | | 10x file-attributes | 37.6 ms | 304.7 ms | 8.1x |

Batch operations provide additional 2-4x speedup by combining multiple requests into a single round-trip (e.g., 10x file.stat drops from 37.6 ms sequential to 9.1 ms batched).

For detailed benchmarks and an in-depth technical comparison with original TRAMP, see [[file:doc/TECHNICAL_COMPARISON.org][Technical Comparison]].

• Testing

TRAMP-RPC includes a comprehensive test suite using Emacs ERT (Emacs Lisp Regression Testing).

** Test Categories

| Category | Tests | Requirements | |--------------------+-------+------------------------| | Protocol | 8 | None (pure Elisp) | | Conversion | 2 | None (pure Elisp) | | Server Integration | 4 | RPC server binary | | Multi-hop | 21 | None (pure Elisp) | | Autoload | 8 | None (pure Elisp) | | Remote File Ops | 53 | SSH + RPC server |

** Running Tests

*** Quick Protocol Tests (no dependencies)

#+begin_src bash ./test/run-tests.sh --protocol #+end_src

Or directly with Emacs:

#+begin_src bash emacs -Q --batch -l test/tramp-rpc-mock-tests.el
--eval "(ert-run-tests-batch-and-exit "^tramp-rpc-mock-test-protocol")" #+end_src

*** All Mock Tests (includes server integration)

#+begin_src bash ./test/run-tests.sh --mock #+end_src

This runs protocol tests plus server integration tests that communicate directly with the RPC server (no SSH needed).

*** Full Remote Tests (requires SSH)

#+begin_src bash TRAMP_RPC_TEST_HOST=your-remote-host ./test/run-tests.sh --remote #+end_src

Or:

#+begin_src bash emacs -Q --batch
-l test/tramp-rpc-tests.el
--eval "(setq tramp-rpc-test-host "your-remote-host")"
--eval "(ert-run-tests-batch-and-exit "^tramp-rpc-test")" #+end_src

** Test Files

• =test/tramp-rpc-tests.el= - Full ERT test suite for remote operations
• =test/tramp-rpc-mock-tests.el= - CI-compatible tests (no SSH required)
• =test/tramp-rpc-autoload-tests.el= - Autoload mechanism tests
• =test/run-tramp-tests.el= - Upstream tramp-tests.el against tramp-rpc backend
• =test/run-tests.sh= - Test runner script
• =test/run-autoload-tests.sh= - Autoload test runner

** CI Integration

The GitHub Actions workflow runs:

1. Rust Build - Builds for 4 targets (x86_64/aarch64 Linux/macOS) with format check
2. Elisp Byte-compile - Verifies all .el files compile without errors
3. Autoload Tests - Verifies method registration and handler setup
4. Protocol Tests - MessagePack-RPC encoding/decoding (no server)
5. Multi-hop Tests - ProxyJump conversion, connection keys, hop normalization
6. Server Integration Tests - Direct server communication (Rust binary)
7. Full Test Suite - Complete tests via SSH to localhost
8. Upstream TRAMP Tests - Runs tramp-tests.el against tramp-rpc backend

• License

This project is licensed under the GNU General Public License v3.0 or later - see the [[file:LICENSE][LICENSE]] file for details.

• Contributing

Contributions welcome! Please ensure code passes cargo clippy and cargo test before submitting.

For Emacs Lisp changes, also run the test suite:

#+begin_src bash ./test/run-tests.sh --mock #+end_src