macOS

July 5, 2021 · View on GitHub

################################################ Utilities for building on Travis CI and AppVeyor ################################################

Update: Uploads, Rackspace, Anaconda

The original Multibuild default was to upload wheels to a Rackspace container, where Rackspace kindly donated the hosting to the Scikit-learn team. We had a URL pointing to the Rackspace container: http://wheels.scipy.org.

Rackspace finally stopped subsidizing this container, and the Rackspace of http://wheels.scipy.org is no more. Some projects using Multibuild have moved to using https://anaconda.org/scipy-wheels-nightly/ for weekly uploads and https://anaconda.org/multibuild-wheels-staging for staging wheels to PyPI.

Uploads to Anaconda

If you want to upload to Anaconda, and you don't need the extra storage space for nightly builds that Anaconda kindly donates to Numpy, Scipy etc, then you can do this with your own Anaconda organization.

See https://github.com/MacPython/nipy-wheels for a simple example.

  • Make an account at Anaconda.org.

  • Make an organization - for example we have used nipy.

  • Navigate to https://anaconda.org/nipy (but use your organization).

  • Go to your account menu towards the top left. This should be labeled with your organization name.

  • Select "Settings", then "Access".

  • Create an access token. Give it permission: "Allow write access to the API site", and (not sure if this is necessary) "Allow uploads to PyPI repositories".

  • "View" your token. It may be of form ni-1234abcd-12ab-34dc-1234-d1e1f3a4b5c6.

  • Encrypt this to your repository, maybe using the travis command line tool (gem install travis). Your command will be something like::

    travis encrypt -r MacPython/nipy-wheels ANACONDA_SECRET=ni-1234abcd-12ab-34dc-1234-d1e1f3a4b5c6

    Note that MacPython/nipy-wheels is your Github organization/repository. The encryption only applies to Travis-CI running against this repository.

  • Go to your .travis.yml file and add the output secure key. This will look something like::

    env: global: # Following generated with # travis encrypt -r MacPython/nipy-wheels ANACONDA_SECRET= # where has API write access to the anaconda.org # organization named in $ANACONDA_ORG - secure: "IqN7LjXWVBaijggUoB+ohjzFzH6nU0OyxznXEMgWoNxQJRiYXXKAt/Z5c4ldp9LUynefJO306M8foN4Gm8M8PNDlhjElzdOtIkGYtDKUXx7aXtrg8rPk1mzuM1F27er4Dbi7WFtpPClr8z8JKNNV50yeM1o2cXu4HgrPrRKgKk/2D8EQaPQlcOqul0O63D9AjVoW3EIG0aWEnZQQGfuGAPgyr0OS92LX2h1pcD2lNZHhqYmXmm5U0IwZmWL3Y0N7PO3VXcOCeIbiHAlJzhk4C4+86TT7DN+VhmfGyY/s61fOz47K+lEZLVqqeQki+HV75fti0XwYG7rjcSvDanNx+w2J/ogSLQpiNxZ0FZ+W8psXEaFUgFf7oXzRkW9gQ4KAsItEWHifq061ngr5AWLPLh+01LGP1Xg8wT5WEVUzBfD2uJPsy20DLcP9WGYa6cBNwtpqmUkdVgM3ZCPWlro7+v1kqxsKp91uh8SRKVlkD4mwbf0FnWxbNZ9v4Z9gs0pZoRclzL+/YcIcSTYAwiQRqaX7T0tpxaUZ0VYTMwCgpsufUX1idV1HV5+WKr9FUocoq+1RRW/JeXkisX9FRvem8cSGmnxB/hynlxoqzttCVMwtrKWPwxH4dHD+lavouho68Q7iBql1ZBZEhQy0O9NC1wr4Rg2CeDPZuzqVjmSPuXQ="

    When Travis-CI runs, this causes the ANACONDA_SECRET environment variable to contain the API key above.

    Also add this to your global environment variables::

    - ANACONDA_ORG="nipy"

  • To upload from Travis-CI, add a clause like this to the end of your .travis.yml::

    after_success: # Upload wheels to Anaconda.org - pip install git+https://github.com/Anaconda-Platform/anaconda-project - pip install git+https://github.com/Anaconda-Platform/anaconda-client - anaconda -t ANACONDASECRETupload−−force−u{ANACONDA_SECRET} upload --force -u {ANACONDA_ORG} ${TRAVIS_BUILD_DIR}/wheelhouse/*.whl

    wheelhouse/*.whl defines the files you want to upload.

  • You might also want to build and upload from Appveyor. To encrypt the API key above, go to https://ci.appveyor.com/account/matthew-brett/settings (where matthew-brett is the account from which your Appveyor job runs. Click on "Encrypt YaML" on the left. Type in your API key value (e.g. ni-1234abcd-12ab-34dc-1234-d1e1f3a4b5c6) as the value to encrypt. Click "Encrypt" and note the text it suggests. Now, at the top of your appveyor.yml file, add something like::

    environment:
  global:
    ANACONDA_ORG: "nipy"
    ANACONDA_SECRET:
      secure: Ds0PkQD0b/QOfoNoiPuFJb01zg0Mq0dkAxIG2jRXocCAereSXdWw6XYaDrutHWme

    where secure: is the text suggested from "Encrypt" above, and nipy is your Anaconda organization.

    Finally, add a clause like the following to the end of your appveyor.yml file::

    on_success: # Upload the generated wheel package to Anaconda.org - pip install git+https://github.com/Anaconda-Platform/anaconda-project - pip install git+https://github.com/Anaconda-Platform/anaconda-client - anaconda -t %ANACONDA_SECRET% upload --force -u %ANACONDA_ORG% nipy\dist*.whl

    where nipy\dist\*.whl finds the files you want to upload.

There's a simple example of these steps applied at https://github.com/MacPython/nipy-wheels.

Here is the NumPy code (for running on Travis CI) to upload to Anaconda: https://github.com/MacPython/numpy-wheels/blob/master/.travis.yml#L99

For projects housed under the MacPython GitHub organization, you have access to Anaconda upload tokens via the "Organization Secrets" https://github.com/MacPython/numexpr-wheels/settings/secrets . You can use these to move to GitHub Actions (they provide x86 machines for Windows, Linux and Mac). Otherwise we (please raise an issue here) will need to negotiate getting you tokens, or you can make your own, as above.

Use Github for uploads

Another option is to use GitHub for staging --- as do Cython for Travis CI <https://github.com/MacPython/cython-wheels/blob/master/.travis.yml#L144>_ and for AppVeyor <https://github.com/MacPython/cython-wheels/blob/master/appveyor.yml#L118>_.

Introduction

A set of scripts to automate builds of macOS and Manylinux1 wheels on the Travis CI <https://travis-ci.org/>_ infrastructure, and also Windows wheels on the AppVeyor <https://ci.appveyor.com/>_ infrastructure.

The Travis CI scripts are designed to build and test:

  • 64-bit macOS wheels built for macOS 10.9+
  • 64/32-bit macOS wheels built for macOS 10.6+
  • 64-bit manylinuxX_x86_64 wheels, both narrow and wide Unicode builds, where X is any valid Manylinux version: 1, 2010, 2014 or _2_24.
  • 32-bit manylinuxX_i686 wheels, both narrow and wide Unicode builds

You can currently build and test against Pythons 2.7, 3.5, 3.6, 3.7, 3.8 and 3.9

The small innovation here is that you can test against Linux 32-bit builds, both wide and narrow Unicode Python 2 builds, which was not easy on the default Travis CI configurations.

The AppVeyor setup is designed to build and test:

  • 64-bit Windows win_amd64 wheels
  • 32-bit Windows win32 wheels

You can currently build and test against Pythons 2.7, 3.5, 3.6, 3.7, 3.8, 3.9

How does it work?

Multibuild is a series of bash scripts that define bash functions to build and test wheels.

Configuration is by overriding the default build function, and defining a test function.

The bash scripts are layered, in the sense that they are composed of a number of scripts which are sourced in sequence, each one potentially overriding previous ones.

macOS

The following bash scripts are sourced in this order::

multibuild/common_utils.sh
multibuild/osx_utils.sh
env_vars.sh
multibuild/configure_build.sh
multibuild/library_builders.sh
config.sh

See multibuild/travis_osx_steps.sh

The macOS build / test phases run on the macOS VM started by Travis CI. Therefore any environment variable defined in .travis.yml or the bash shell scripts listed above are available for your build and test.

Build options are controlled mainly by the following environment variables:

  • MB_PYTHON_VER sets the Python version targetted: major.minor.patch for CPython, or pypy-major.minor for PyPy.
  • MB_PYTHON_OSX_VER sets the minimum macOS SDK version for any C extensions. For CPython targets it may be set to 10.6 or 10.9, provided a corresponding Python build is available at python.org <https://www.python.org/downloads/mac-osx/>_. It defaults to the highest version available. It's ignored for PyPy targets.
  • PLAT sets the architectures built for any C extensions: x86_64 or intel for 64-bit or 64/32-bit respectively. It defaults to the same arches as the target Python version: 64-bit for CPython macOS 10.9 or PyPy, and 64/32-bit for CPython 10.6.

In most cases it's best to rely on the defaults for MB_PYTHON_OSX_VER and PLAT, rather than setting them explicitly. Examples of exceptions to this guideline include:

  • setting MB_PYTHON_OSX_VER=10.6 to build a 10.6 64/32-bit CPython wheel for Python 2.7 (default for 2.7 is 10.9 64-bit)
  • setting MB_PYTHON_OSX_VER=10.6 and PLAT=x86_64 to build a 10.6 64-bit only wheel (10.6 would normally be 64/32-bit). Such a wheel would still have a platform tag of macosx_10_6_intel , advertising support for both 64 and 32-bit, but wouldn't work in 32-bit mode. This may be OK given how unlikely it is that there is still anyone actually running Python on macOS in 32-bit mode.

The build_wheel function builds the wheel, and install_run function installs and tests it. Look in multibuild/common_utils.sh for default definitions of these functions. See below for more details, many of which are common to macOS and Linux.

Manylinux

The build phase is in a Manylinux Docker container, but the test phase is in a clean container.

Build phase

Specify the Manylinux version to build for with the MB_ML_VER environment variable. The default version is 1. Versions that are currently valid are:

  • 1 corresponding to manylinux1 (see PEP 513 <https://www.python.org/dev/peps/pep-0513>_).
  • 2010 corresponding to manylinux2010 (see PEP 571 <https://www.python.org/dev/peps/pep-0571>_).
  • 2014 corresponding to manylinux2014 and adds more architectures to PLAT (see PEP 599 <https://www.python.org/dev/peps/pep-0599>_).
  • _2_24 corresponding to manylinux_2_24 (see PEP 600 <https://www.python.org/dev/peps/pep-0600>_).

The environment variable specified which Manylinux docker container you are building in.

The PLAT environment variable can be one of

  • x86_64, for 64-bit x86
  • i686, for 32-bit x86
  • s390x, for 64-bit s390x
  • ppc64le, for PowerPC
  • aarch64, for ARM
  • arm64, for Apple silicon
  • universal2, for both Apple silicon and 64-bit x86

The default is x86_64. Only x86_64 and i686 are valid on manylinux1 and manylinux2010.

multibuild/travis_linux_steps.sh defines the build_wheel function, which starts up the Manylinux1 Docker container to run a wrapper script multibuild/docker_build_wrap.sh, that (within the container) sources the following bash scripts::

multibuild/common_utils.sh
multibuild/manylinux_utils.sh
env_vars.sh
multibuild/configure_build.sh
multibuild/library_builders.sh
config.sh

See docker_build_wrap.sh to review the order of script sourcing.

See the definition of build_multilinux in multibuild/travis_linux_steps.sh for the environment variables passed from Travis CI to the Manylinux1 container.

Once in the container, after sourcing the scripts above, the wrapper runs the real build_wheel function, which now comes (by default) from multibuild/common_utils.sh.

Test phase

Specify the version to test with the DOCKER_TEST_IMAGE environment variable. The default version is dependent on PLAT:

  • matthewbrett/trusty:64, for x86_64
  • matthewbrett/trusty:32 for i686
  • multibuild/xenial_arm64v8 for aarch64
  • multibuild/xenial_ppc64le for ppc64le
  • mutlibuild/xenial_s390x for s390x

Other valid values are any in https://hub.docker.com/orgs/multibuild/repositories, using the correct platform code. Alternatively, you can use the substitution pattern multibuild/xenial_{PLAT} in the .travis.yml file.

See multibuild/docker_test_wrap.sh.

multibuild/travis_linux_steps.sh defines the install_run function, which starts up the testing Docker container with the wrapper script multibuild/docker_test_wrap.sh. The wrapper script sources the following bash scripts::

multibuild/common_utils.sh
config.sh

See docker_test_wrap.sh for script source order.

See install_run in multibuild/travis_linux_steps.sh for the environment variables passed into the container.

It then (in the container) runs the real install_run command, which comes (by default) from multibuild/common_utils.sh.

Standard build and test functions

The standard build command is build_wheel. This is a bash function. By default the function that is run on macOS, and in the Manylinux container for the build phase, is defined in multibuild/common_utils.sh. You can override the default function in the project config.sh file (see below).

If you are building a wheel from PyPI, rather than from a source repository, you can use the build_index_wheel command, again defined in multibuild/common_utils.sh.

Typically, you can get away with leaving the default build_wheel / build_index_wheel functions to do their thing, but you may need to define a pre_build function in config.sh. The default build_wheel and build_index_wheel functions will call the pre_build function, if defined, before building the wheel, so pre_build is a good place to build any required libraries.

The standard test command is the bash function install_run. The version run on macOS and in the Linux testing container is also defined in multibuild/common_utils.sh. Typically, you do not override this function, but you in that case you will need to define a run_tests function, to run your tests, returning a non-zero error code for failure. The default install_run implementation calls the run_tests function, which you will likely define in config.sh. See the examples below for examples of less and more complicated builds, where the complicated builds override more of the default implementations.

To use these scripts

If your project depends on NumPy, you will want to build against the earliest NumPy that your project supports - see forward, backward NumPy compatibility <https://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling/18369312#18369312>. See the astropy-wheels Travis file <https://github.com/MacPython/astropy-wheels/blob/master/.travis.yml> for an example specifying NumPy build and test dependencies.

Here are some simple example projects:

Less simple projects where there are some serious build dependencies, and / or macOS / Linux differences:

Multibuild development

The main multibuild repository is always at https://github.com/matthew-brett/multibuild

We try to keep the master branch stable and do testing and development in the devel branch. From time to time we merge devel into master.

In practice, you can check out the newest commit from devel that works for you, then stay at it until you need newer features.