README.md

July 27, 2026 · View on GitHub

vgi-rpc-java

Transport-agnostic RPC framework built on Apache Arrow IPC serialization — the Java port of vgi-rpc.
Built by 🚜 Query.Farm

Define RPC interfaces as ordinary Java interfaces. The framework derives Apache Arrow schemas from your method signatures and record component types, and hands you a typed client proxy with automatic serialization/deserialization. There are no .proto files or codegen steps — your Java types are the schema. Unlike JSON-over-HTTP, structured data stays in Arrow's columnar format for efficient transfer, which pays off for large or batch-oriented workloads.

This is a port of the Python reference implementation, vgi-rpc, and is wire-compatible with it: the same calls interoperate across the Python, Java, Go, and C++ peers (the conformance suite runs the Python driver against this Java worker over every transport).

Key features

Interface-based services — define a service as a typed Java interface; the client proxy preserves that interface for full IDE autocompletion.
Apache Arrow IPC wire format — columnar serialization for structured data.
Two method types — unary calls and streaming (producer and exchange patterns).
Transport-agnostic — stdio pipe, subprocess, Unix domain socket, raw TCP socket (trusted networks — no auth/TLS), shared memory, or HTTP.
Automatic schema inference — Java types and record components map to Arrow types; @ArrowField refines them.
Pluggable authentication — AuthContext + authenticators for HTTP (bearer, mTLS/XFCC; JWT/OAuth in the optional vgirpc-oauth module).
Runtime introspection — opt-in __describe__ RPC for dynamic service discovery, with a protocol hash that matches the Python reference byte-for-byte.
Shared-memory transport — zero-copy batch transfer between co-located processes (auto-negotiated on JDK 22+ via a multi-release overlay; transparent pipe fallback otherwise).
Large-batch externalization — oversized batches transparently spilled to S3 (vgirpc-s3) or GCS (vgirpc-gcs).

Requirements

Java 21+ at runtime. The shared-memory side-channel additionally requires JDK 22+ (where java.lang.foreign is GA); on 21 it transparently falls back to inline transfer.

Installation

Artifacts are published to Maven Central under the farm.query group.

Gradle (Kotlin DSL):

dependencies {
    implementation("farm.query:vgirpc:0.8.0")          // core: protocol, transports, HTTP, schema
    implementation("farm.query:vgirpc-oauth:0.8.0")    // optional: JWT / OAuth / PKCE auth
    implementation("farm.query:vgirpc-s3:0.8.0")       // optional: S3 external storage
    implementation("farm.query:vgirpc-gcs:0.8.0")      // optional: GCS external storage
}

Maven:

<dependency>
  <groupId>farm.query</groupId>
  <artifactId>vgirpc</artifactId>
  <version>0.8.0</version>
</dependency>

The core depends on Apache Arrow and SLF4J (API only — bring your own logging backend).

Quick start

1. Define a service as a Java interface (shared by client and server):

public interface Calculator {
    double add(double a, double b);
    String greet(String name);
}

2. Implement it and serve it. A worker typically serves over stdio so a parent process can drive it as a subprocess:

import farm.query.vgirpc.RpcServer;
import farm.query.vgirpc.transport.StdioTransport;

public final class CalculatorWorker {
    public static void main(String[] args) {
        Calculator impl = new Calculator() {
            public double add(double a, double b) { return a + b; }
            public String greet(String name)      { return "Hello, " + name + "!"; }
        };
        RpcServer server = new RpcServer(Calculator.class, impl);
        try (StdioTransport transport = new StdioTransport()) {
            server.serve(transport);
        }
    }
}

3. Call it through a typed proxy. The client launches the worker and gets back something that is a Calculator:

import farm.query.vgirpc.RpcConnection;
import farm.query.vgirpc.transport.SubprocessTransport;
import java.util.List;

var transport = new SubprocessTransport(List.of(
        "java", "--add-opens=java.base/java.nio=ALL-UNNAMED",
        "-cp", "worker.jar", "CalculatorWorker"));
try (RpcConnection conn = new RpcConnection(transport)) {
    Calculator calc = conn.proxy(Calculator.class);
    double sum    = calc.add(2.0, 3.0);   // 5.0
    String hello  = calc.greet("World");  // "Hello, World!"
}

Two things to get right:

Run with --add-opens=java.base/java.nio=ALL-UNNAMED on every JVM that touches the library (both the worker and the client above) — Apache Arrow accesses java.nio internals and throws on startup without it. Notice it's passed both to the client JVM and, in the SubprocessTransport command, to the spawned worker.

Compile services with -parameters — the framework binds call arguments by parameter name (matching the Python reference's keyword-argument wire semantics).

Modules

Module	Purpose
`vgirpc`	Core library — wire protocol, transports, HTTP server/client (Jetty 12), schema derivation, marshalling, external-location support, shared-memory primitive.
`vgirpc-oauth`	Optional OAuth/JWT support (JWKS validation, PKCE, signed cookies). Split out so core users don't pull `nimbus-jose-jwt`.
`vgirpc-s3`	Amazon S3 `ExternalStorage` backend for large-batch externalization.
`vgirpc-gcs`	Google Cloud Storage `ExternalStorage` backend.

Transports

Transport	Use case
stdio (`StdioTransport`)	Worker process driven over stdin/stdout by a parent.
subprocess (`SubprocessTransport`)	Client spawns and talks to a worker subprocess.
Unix socket (`UnixSocketTransport`)	Co-located processes over a domain socket.
shared memory	Zero-copy batch transfer for co-located processes; auto-negotiated on JDK 22+, transparent pipe fallback otherwise.
HTTP (`HttpServer` / Jetty 12)	Networked, stateless-server streaming; auth via authenticators.

Method types

Unary — request batch in, one result (or error) batch out.
Streaming — a RpcStream<S extends StreamState> whose state's process(input, out, ctx) runs once per tick, in two flavours: producer (server emits a sequence of output batches) and exchange (lockstep input batch → output batch).

Wire compatibility

When the Python and Java implementations disagree, Python is the reference. Wire format, metadata keys, error semantics, and stream-state token layout match byte-for-byte so the two interoperate. See the Python project's README for the higher-level protocol design.

Proxy proof

Proxy proof lets a worker refuse any request that did not arrive through a trusted proxy. The proxy mints a per-request HMAC-SHA256 over a timestamp, a fresh nonce and the worker's own identifier, keyed by a secret shared only with that worker. Unlike a forwarded assertion about what happened at a TLS terminator, a proof cannot be produced by someone who merely reaches the worker directly — without the secret there is nothing to replay.

var secrets = ProxyProof.parseSecrets("prod-use1:" + hexSecret);
var config = ProxyProof.Config.of(ProxyProof.Mode.REQUIRE, "worker-a", secrets);

HttpServer.Config.builder()
    .authenticator(ProxyProof.require(config, existingAuthenticator)) // inner may be null
    .proxyProofRequired(true)                                         // REQUIRE mode only
    .build();

It composes as an AND, not an alternative: do not pass the gate to Authenticator.chain, whose first-authenticated-wins semantics would let any later credential bypass it.

proxyProofRequired(true) advertises VGI-Proxy-Proof-Required: true on every response, GET /health and OPTIONS included, so an operator or proxy can confirm the worker really does reject unproofed requests — otherwise a misconfiguration turns the whole feature into a silent no-op. Set it in REQUIRE mode only: off and allow never deny, so they must not claim to. It is a separate knob because the gate arrives as an opaque Authenticator the server cannot introspect, and it advertises only — enforcement is entirely the gate's.

The key id doubles as the calling proxy's label, so AuthContext.claims().get("vgi_proxy_proof") records which proxy served each request — derived from the secret that verified, never from the transmitted field. OPTIONS, /.well-known/ and {prefix}/health stay reachable without a proof in every mode.

Needs no dependency beyond the JDK. The normative cross-language contract is docs/proxy-proof-spec.md in the vgi-rpc repository.