xlang-serialization.md

This page explains how to use Fory xlang serialization between C++ and other languages.

Overview

Apache Fory™ enables seamless data exchange between C++, Java, Python, Go, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin. Xlang mode ensures binary compatibility across all supported languages.

Create an Xlang Fory Instance

C++ defaults to xlang mode. Compatible schema evolution is also the xlang default. Set the mode explicitly in xlang examples:

#include "fory/serialization/fory.h"

using namespace fory::serialization;

auto fory = Fory::builder().xlang(true).build();

Xlang Example

C++ Producer

#include "fory/serialization/fory.h"
#include <fstream>

using namespace fory::serialization;

struct Message {
  std::string topic;
  int64_t timestamp;
  std::map<std::string, std::string> headers;
  std::vector<uint8_t> payload;

  bool operator==(const Message &other) const {
    return topic == other.topic && timestamp == other.timestamp &&
           headers == other.headers && payload == other.payload;
  }
};
FORY_STRUCT(Message, topic, timestamp, headers, payload);

int main() {
  auto fory = Fory::builder().xlang(true).build();
  fory.register_struct<Message>(100);

  Message msg{
      "events.user",
      1699999999000,
      {{"content-type", "application/json"}},
      {'h', 'e', 'l', 'l', 'o'}
  };

  auto result = fory.serialize(msg);
  if (result.ok()) {
    auto bytes = std::move(result).value();
    // write to file, send over network, etc.
    std::ofstream file("message.bin", std::ios::binary);
    file.write(reinterpret_cast<const char*>(bytes.data()), bytes.size());
  }
  return 0;
}

Java Consumer

import org.apache.fory.Fory;

public class Message {
    public String topic;
    public long timestamp;
    public Map<String, String> headers;
    public byte[] payload;
}

public class Consumer {
    public static void main(String[] args) throws Exception {
        Fory fory = Fory.builder()
            .withXlang(true)
            .build();
        fory.register(Message.class, 100);  // Same ID as C++

        byte[] bytes = Files.readAllBytes(Path.of("message.bin"));
        Message msg = (Message) fory.deserialize(bytes);

        System.out.println("Topic: " + msg.topic);
        System.out.println("Timestamp: " + msg.timestamp);
    }
}

Python Consumer

import pyfory

class Message:
    topic: str
    timestamp: int
    headers: dict[str, str]
    payload: bytes

fory = pyfory.Fory(xlang=True)
fory.register(Message, type_id=100)  # Same ID as C++

with open("message.bin", "rb") as f:
    data = f.read()

msg = fory.deserialize(data)
print(f"Topic: {msg.topic}")
print(f"Timestamp: {msg.timestamp}")

Type Mapping

Primitive Types

C++ Type	Java Type	Python Type	Go Type	Rust Type
bool	boolean	bool	bool	bool
int8_t	byte	int	int8	i8
int16_t	short	int	int16	i16
int32_t	int	int	int32	i32
int64_t	long	int	int64	i64
float	float	float	float32	f32
double	double	float	float64	f64
fory::float16_t	Float16	pyfory.Float16	float16.Float16	Float16
fory::bfloat16_t	BFloat16	pyfory.BFloat16	bfloat16.BFloat16	BFloat16

String Types

C++ Type	Java Type	Python Type	Go Type	Rust Type
std::string	String	str	string	String

Collection Types

C++ Type	Java Type	Python Type	Go Type	Rust Type
std::vector<T>	List<T>	list	[]T	Vec<T>
std::vector<fory::float16_t>	Float16List	Float16Array	[]float16.Float16	Vec<Float16>
std::vector<fory::bfloat16_t>	BFloat16List	BFloat16Array	[]bfloat16.BFloat16	Vec<BFloat16>
std::set<T>	Set<T>	set	map[T]struct{}	HashSet<T>
std::map<K,V> / std::unordered_map<K,V>	Map<K,V>	dict	map[K]V	HashMap<K,V>

Lists and Dense Arrays

std::vector<T> maps to Fory list<T> by default in handwritten C++ structs. Use the field metadata DSL's array node when the schema is dense array<T>.

Fory schema	C++ metadata sketch
list<int32>	fory::F(id).list(fory::T::int32())
array<bool>	fory::F(id).array(fory::T::bool_())
array<int8>	fory::F(id).array(fory::T::int8())
array<int16>	fory::F(id).array(fory::T::int16())
array<int32>	fory::F(id).array(fory::T::int32())
array<int64>	fory::F(id).array(fory::T::int64())
array<uint8>	fory::F(id).array(fory::T::uint8())
array<uint16>	fory::F(id).array(fory::T::uint16())
array<uint32>	fory::F(id).array(fory::T::uint32())
array<uint64>	fory::F(id).array(fory::T::uint64())
array<float16>	fory::F(id).array(fory::T::float16())
array<bfloat16>	fory::F(id).array(fory::T::bfloat16())
array<float32>	fory::F(id).array(fory::T::float32())
array<float64>	fory::F(id).array(fory::T::float64())

Temporal Types

C++ Type	Java Type	Python Type	Go Type
fory::Timestamp	Instant	datetime	time.Time
fory::Duration	Duration	timedelta	time.Duration
fory::Date	LocalDate	datetime.date	time.Time

Field Order Requirements

Critical: Fields are sorted by snake_case field name. The converted names must match across languages.

C++

struct Person {
  std::string name;   // Field 0
  int32_t age;        // Field 1
  std::string email;  // Field 2
};
FORY_STRUCT(Person, name, age, email);  // Order matters!

Java

public class Person {
    public String name;   // Field 0
    public int age;       // Field 1
    public String email;  // Field 2
}

Python

class Person:
    name: str    # Field 0
    age: int     # Field 1
    email: str   # Field 2

Type ID Consistency

All languages must use the same type IDs:

// C++
fory.register_struct<Person>(100);
fory.register_struct<Address>(101);
fory.register_struct<Order>(102);

// Java
fory.register(Person.class, 100);
fory.register(Address.class, 101);
fory.register(Order.class, 102);

# Python
fory.register(Person, type_id=100)
fory.register(Address, type_id=101)
fory.register(Order, type_id=102)

Compatible Mode

Xlang mode already uses compatible schema evolution by default. Keep that default for schemas that may evolve independently:

auto fory = Fory::builder().xlang(true).build();

Compatible mode allows:

• Adding new fields (with defaults)
• Removing unused fields
• Reordering fields

Troubleshooting

Type Mismatch Errors

Error: Type mismatch: expected 100, got 101

Solution: Ensure type IDs match across all languages.

Encoding Errors

Error: Invalid UTF-8 sequence

Solution: Ensure strings are valid UTF-8 in all languages.

Related Topics

• Configuration - Builder options
• Type Registration - Registering types
• Supported Types - Type compatibility