Wire CoreCrypto

This repository is part of the source code of Wire. You can find more information at wire.com or by contacting opensource@wire.com.

You can find the published source code at github.com/wireapp/wire.

For licensing information, see the attached LICENSE file and the list of third-party licenses at wire.com/legal/licenses/.

No license is granted to the Wire trademark and its associated logos, all of which will continue to be owned exclusively by Wire Swiss GmbH. Any use of the Wire trademark and/or its associated logos is expressly prohibited without the express prior written consent of Wire Swiss GmbH.

• Wire CoreCrypto
  • Documentation
  • Building
    • General Requirements
      • Pre-commit
    • JVM
    • Android
    • iOS
    • MacOS
    • Linux
    • WASM
    • Bindings
  • Testing
    • Rust Unit/Integration Tests
      • All Ciphersuites
      • Keystore Wasm tests
    • Platform-specific tests for WASM/Web
    • Platform-specific tests for Kotlin/JVM
    • Platform-specific tests for Android
    • Swift/iOS
    • Interop
    • End-to-end-identity (E2EI) testing
      • Preparing the container runtime environment
        • On Linux with Docker
        • On Linux with Podman
        • On macOS with Docker
        • On macOS with Podman
      • Choosing the OIDC identity provider
      • Running all tests at once
      • Running specific tests
      • Manually invoking tests
  • Formatting and Linting
    • Requirements
      • Swift
      • Kotlin
      • Markdown
      • Toml
  • Benchmarks
    • Nightly run
    • FFI Bindings
      • JVM
      • Browser
      • TS-Native
  • Profiling
  • Git workflow
  • Publishing
    • Versioning
    • Making a new release
      • Consider when making a release from a release branch

Documentation

• Docs home
• Architecture
• Publish Manually

Building

General Requirements

• rust: https://rustup.rs/
• GNU make: https://www.gnu.org/software/make/ (min version: 4.3)
• nextest: https://nexte.st/: cargo install --locked cargo-nextest
• Ensure your git is configured to sign commits
• fd: https://github.com/sharkdp/fd

Pre-commit

• Install the pre-commit framework
• Run pre-commit install to initialize the pre-commit hooks

JVM

• Install SDKMAN!: curl -s "https://get.sdkman.io" | bash
• Install Java 17: sdk install java 17.0.17-tem
• Install Kotlin: sdk install kotlin

make jvm       # make the JVM target
make jvm-test  # make and test the JVM target

Android

Install Android SDK and Build-Tools for API level 30+

export ANDROID_SDK_ROOT=~/Android/Sdk

Install the Android NDK. Make sure to set the ANDROID_NDK_HOME variable to point to the NDK installation.

Install android rust targets:

rustup target add x86_64-linux-android aarch64-linux-android armv7-linux-androideabi

Build:

make android

iOS

Install Xcode & its command-line tools: https://developer.apple.com/xcode/.

Install iOS rust targets:

rustup target add aarch64-apple-ios aarch64-apple-ios-sim

Build:

make ios
# Additionally, if you want to export a .XCFramework:
make ios-create-xcframework

MacOS

Install macOS rust targets:

rustup target add aarch64-apple-darwin

Linux

Install Linux targets:

rustup target add x86_64-unknown-linux-gnu

WASM

Make sure you have all prerequisites:

• Install the wasm32-unknown-unknown toolchain: rustup target add wasm32-unknown-unknown

• Install node.js (recommended way is via Volta)

• Install Bun (follow the instructions on Bun's website)

• Install wasm-bindgen-cli:

  wasm_bindgen_version="$(
    cargo metadata --format-version 1 |
    jq -r '.packages[] | select (.name == "wasm-bindgen") | .version'
  )"
  cargo install wasm-bindgen-cli --version $wasm_bindgen_version
  

  It is important to ensure that the wasm-bindgen-cli version always precisely matches the wasm-bindgen version in Cargo.lock (as shown by cargo info wasm-bindgen), because otherwise the wasm tests will not run.

• Install chromedriver

  bunx @puppeteer/browsers install --path ~/bin chrome-headless-shell
  bunx @puppeteer/browsers install --path ~/bin chromedriver
  

Build:

make ts       # make the typescript target
make ts-test  # make the typescript target and run tests

Bindings

Build bindings for Android, JVM, iOS and WASM

# builds bindings and targets for the JVM (macOS / Linux)
make jvm

# builds bindings and targets for Android
make android

# builds iOS framework
make ios-create-xcframework

# builds wasm binary & TS bindings
make ts

Testing

Rust Unit/Integration Tests

cargo nextest run

All Ciphersuites

cargo nextest run --features test-all-cipher

Keystore Wasm tests

Sometimes for the Keystore it is valuable to run Rust unit/integration tests on the WASM target.

• Ensure you are set up to build wasm
• Install wasm-pack: cargo install wasm-pack.

Then, just use wasm-pack to run the tests:

wasm-pack test --headless --chrome -- ./keystore --locked

Run tests containing the word proteus, additionally enabling the feature proteus-keystore:

wasm-pack test --headless --chrome -- ./keystore --locked --features proteus-keystore -- proteus

Unfortunately it appears that nextest doesn't work well with the wasm runner, so we're stuck with the basic test runner.

Platform-specific tests for WASM/Web

make ts-test

Note the CC_TEST_LOG_LEVEL environment variable. At 1 it emits browser console logs; at 2 it also emits CoreCrypto logs.

Platform-specific tests for Kotlin/JVM

make jvm-test

Platform-specific tests for Android

make android-test

Swift/iOS

Currently works on macOS only.

make ios-test

Interop

1. Make sure you've successfully run each platform-specific test suite that is supported on your machine, since the setup required for this test is a superset of the requirements for each platform.
2. Install Chrome and chromedriver
3. make interop-test

End-to-end-identity (E2EI) testing

Preparing the container runtime environment

Some tests require a working container runtime, so make sure to prepare one before running all tests. Platform-specific instructions follow below.

On Linux with Docker

Make sure to start the Docker service if it is not already running:

systemctl start docker.service

On Linux with Podman

# start socket activation, which will cause Podman to start once
# anything connects to the socket:
systemctl --user start podman.socket

# check that socket activation works
podman version

# if the above didn't work, depending on the distribution and installed packages,
# it may be necessary to configure the DOCKER_HOST variable to point to Podman's socket
export DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock

On macOS with Docker

Note: Docker under macOS requires Docker Desktop, which must run as a GUI application.

# install docker and docker-desktop
brew install docker docker-desktop

# start the Docker daemon by launching docker-desktop as a GUI application

# check if everything went fine
docker version

On macOS with Podman

# install Podman
brew install podman

# install podman-mac-helper
sudo podman-mac-helper install

# create new VM based on machine-os:5.5; note that we're explicitly specifying
# an older image version because the newest one seems to be broken
podman machine init --image docker://quay.io/podman/machine-os:5.5

# start the machine
podman machine start

# if everything went well, this should print server version `5.5.x`
podman version

# symlink docker to podman (test scripts and code assume existence
# of the `docker` command)
ln -s /opt/homebrew/bin/podman /opt/homebrew/bin/docker

Choosing the OIDC identity provider

Choose the OIDC identity provider to use in tests by setting the TEST_IDP variable:

# use Keycloak
export TEST_IDP=keycloak

# or Authelia
export TEST_IDP=authelia

Running all tests at once

Simply execute the run-e2ei-tests.sh script:

bash scripts/run-e2ei-tests.sh

The script will take care of cleaning up processes and containers that are started during tests.

Running specific tests

run-e2ei-tests.sh forwards its arguments to cargo nextest, which can be used to run a specific test, or any subset of tests, e.g.

bash scripts/run-e2ei-tests.sh alg::p256

Manually invoking tests

First, you need to start test-wire-server:

$ cargo run --locked --bin test-wire-server
[...]
127.0.0.1:20530

Note the IP and port printed by test-wire-server and export that as TEST_WIRE_SERVER_ADDR:

export TEST_WIRE_SERVER_ADDR=127.0.0.1:20530

Now that the environment is ready, you can run a specific test, or any subset of tests, e.g.

cargo nextest run --locked --ignore-default-filter -p wire-e2e-identity alg::p256

Once you are done with testing, terminate the IdP container that has been started:

# if you're using Keycloak
docker kill keycloak && docker rm keycloak

# if you're using Authelia
docker kill authelia.local && docker rm authelia.local

as well as the test-wire-server instance.

Formatting and Linting

For all languages we provide make rules for formatting and checking.

• make fmt
  • make rust-fmt
  • make swift-fmt
  • make kotlin-fmt
  • make ts-fmt
• make check
  • make rust-check
  • make swift-check
  • make kotlin-check
  • make ts-check

Requirements

Swift

We're using swift-format to format swift files and use swiftlint for linting.

Kotlin

We're using ktlint to format and lint kotlin files.

Markdown

We're using mdformat for consistent formatting of our markdown files. Install it with the following extensions

• mdformat-gfm
• mdformat-frontmatter
• mdformat-footnote
• mdformat-gfm-alerts
• mdformat-toc

Toml

We're using taplo to format .toml files.

Benchmarks

Nightly run

Every night we run benchmarks on the main branch and publish the results on the CoreCrypto Benchmarks page.

FFI Bindings

There are benchmarks for jvm, browser, and ts-native bindings run via the corresponding make rule. You can filter Benchmarks by setting BENCH.

JVM

make jvm-bench

Browser

make ts-browser-bench

TS-Native

make ts-native-bench

Profiling

We can attach a profiler to the JVM benchmarks. We provide configuration for async-profiler via make.

Install async-profiler

Set ASYNC_PROFILER_LIB to the .so or .dylib file of the installation, e.g. : export ASYNC_PROFILER_LIB=<install-path>/async-profiler/4.3/lib/libasyncProfiler.dylib

You can then create flamegraphs for any jvm benchmarks using:

make jvm-bench PROFILE=1

Results are saved to core-crypto/crypto-ffi/bindings/jvm/build/reports/async.

Git workflow

• The main branch is used as the everyday development branch.
• No merge commits. Always rebase on top of main.
• Release branches are named release/<series>, e.g. release/1.x, release/2.x.
• Release branches contain fixes relevant to their specific release series and are never merged to main.
• Release branches always branch off their first major release tag. For example, the output of git merge-base main release/2.x must be a commit pointed to by tag v2.0.0.
• Release branches are created lazily, that is, only when the first fix needs to be applied and released for a specific release series.
• Use conventional commits -- those are picked up by the changelog generator.
• If there is a JIRA ticket related to the change, you should mention it in either the PR title or the commit(s), with the following format: [TICKET_ID].
• Sign your commits and tags.
• Remove branches from the remote once you don't need them anymore.

Publishing

Versioning

The versioning scheme used is SemVer AKA Semantic Versioning.

Making a new release

1. Make a branch based on main to prepare for release (git checkout -b prepare-release/X.Y.Z)
2. Generate the relevant changelog section:

   git cliff --bump --unreleased
   

   and add it to the top of CHANGELOG.md. Make sure the version number generated by git cliff matches the release version.
3. If there are any release highlights, add them as the first subsection below release title:

   ## v1.0.2 - 2024-08-16
   
   ### Highlights
   
   - foo
   - bar
   - baz
   

4. In index.md, copy the commented-out table row from the bottom of the file to the appropriate place in the table, ordering by version number, descending. Search and replace the first 5 occurrences of x.x.x with X.Y.Z.
5. Run sh scripts/update-versions.sh X.Y.Z to update the versions of
   • all workspace member crates
   • package.json
   • crypto-ffi/bindings/gradle.properties Make sure the result of the script run is correct.
6. Make sure the changes look reasonable and complete; you can use the previous release as a reference
7. Push your prepare-release/X.Y.Z branch and create a PR for it
8. Get it reviewed, then merge it into main and remove the prepare-release/X.Y.Z branch from the remote
9. Now, pull your local main: git checkout main && git pull
10. Create the release tag: git tag -s vX.Y.Z
11. Push the new tag: git push origin tag vX.Y.Z
12. Create a new release on github, copying the relevant section from CHANGELOG.md
13. Voilà!

Consider when making a release from a release branch

1. Isolate the changes to index.md and CHANGELOG.md from the release commit itself
2. After the release is finished, cherry-pick the changes to index.md and CHANGELOG.md and get them into main
3. For release series 4.x and newer, docs upload happens automatically. If you released from the series 3.x or older, you need to trigger docs upload manually:
   1. On GitHub, go to the docs workflow
   2. Click the Run workflow button
   3. In the Use workflow from dropdown, choose release/5.x, in Tag to checkout provide your release tag