Getting Started Resources

October 14, 2016 · View on GitHub

Most likely you know of C++. We use a particular style of C++ referred to as modern C++, based on patterns enabled by new features in C++11. The Style Guide and FAQ will cover many of these patterns and hopefully provide simple, safe building blocks for programming in C++. For those who learned C++ before C++11, there are a number of patterns that we encourage versus what you’re used to, as they tend to produce fewer errors and result in safer code.

We use all of C++11 except for:

  • std::regex, as GCC 4.8 doesn’t support it
  • std::thread, as GCC 4.8 on AIX doesn't support it, and the mingw support is experimental

Once we update our tooling to use a newer version of GCC, we’ll likely bump our language support to C++14.

We also use CMake for our project build system - usually to generate Makefiles that work for your target platform - so having some understanding of how it works is helpful.

The C++ Language

Many of the books below are available in our library.

Study Guide

Learn some basic CMake with CMake Tutorial

Add some more complete instruction from one of

Work through some exercises from Exercism C++ Exercises with the C++ Primer or The C++ Programming Language at hand.

Since the Puppet RAL is pretty familiar to many of us, it makes a useful example. Create a set of C++ classes representing the RAL, with the ability to print out a representation of them. Then create data structures to represent a catalog, including a relationship DAG. A basic example is included in ral-example.

Language References

Community

Practitioner Knowledge

Books

Blogs and Videos

Other material by Sean Parent

Tooling

A development environment on most platforms is pretty straight-forward, get Clang 3.5+ or GCC 4.8+. On Mac OS X, the system Clang from Xcode Command-Line Tools (required by Homebrew) is sufficient. On Windows, I recommend installing CMake and MinGW-w64 via Chocolatey.

Visual Studio compatibility isn’t guaranteed in our projects due to bugs in Visual C++ and their incomplete C++11 support.

Our current release pipeline uses

  • CMake 3.2.2
  • Apple LLVM 6.1.0 (Clang 3.6) on Mac OS X
  • MinGW GCC 4.8.3 with win32 threads on Windows, using Structured Exception Handling (SEH) for 64-bit exception handling and Set-jump-long-jump (SJLJ) for 32-bit (libpthread dropped because the library was buggy)
  • GCC 4.8.2 on all other platforms

JetBrains provides an IDE, CLion, that some of us find useful. For vim/emacs users, you may want to look at ycmd for C++ code completion based on Clang. On Mac, Xcode can also be used for building and debugging by preparing a project with cmake -G Xcode ….

CMake Issues

Some issues you might run into with CMake.

Include file order

If system include files are included before other dependencies - particularly Leatherman - you can end up using unexpected versions of libraries. For example, a version of rapidjson installed to /usr/local could be included before the version vendored in Leatherman, causing compilation errors. This can be fixed by using -DCMAKE_INCLUDE_DIRECTORIES_BEFORE=ON when running CMake.

CLion Config

Some suggested configuration to make your life easier.

Use ccache
brew install ccache

In CLion > Preferences, in Build, Execution, Deployment > CMake, add to Generation > CMake options

-DCMAKE_C_COMPILER=/usr/local/opt/ccache/libexec/cc -DCMAKE_CXX_COMPILER=/usr/local/opt/ccache/libexec/c++
Configure Editor
  • Enable Line Numbers Editor > General > Appearance > Show line numbers
  • CMake Indentation Editor > Code Style > CMake > Tabs and Indents > Continuation Indent = 4
  • C++ Indentation Editor > Code Style > C++ > Tabs and Indents > Continuation Indent = 4

TODO: Export a CLion Settings that matches our cpplint configuration.

Starting a Project

Libraries

Libraries approved by RelEng that are currently in use. Most libraries support UTF-8, although Boost on Windows needs to be rebuilt with ICU for full support.

A list of other libraries that may be useful is maintained at http://en.cppreference.com/w/cpp/links/libs. New libraries must be approved by RelEng.

Libraries we use are organized by a few rules:

  • Header-only libraries are vendored into a project. If they seem generally useful, add them as components of Leatherman. rapidjson is a useful example of how to do that.
  • Statically-linked libraries will be built and packaged, to be hosted on platform-specific packaging feeds (Apt, Rpm, Nuget). The pipelines for building these are managed by RelEng. Project dependencies are then made build-time dependencies in Vanagon projects.
  • Dynamically-linked libraries are built and packaged with the Vanagon project they will be shipped with.

Boost ∞

Boost comes with many sub-components. Currently our C++ toolchain only builds out a few of them. We’re evaluating building several more so that in the future more options are available to developers.

Boost support for packaging is at 1.58 and has the following compiled libraries: chrono, date_time, filesystem, locale, log, program_options, random, regex, system, thread

Boost also contains a number of header-only libraries that are also available. See the Boost documentation for all the details or The Boost C++ Libraries for a more structured introduction to all the libraries).

Documentation for Boost isn’t the greatest, but it has a lot of useful utilities. In the FAQ I’ll call common patterns that use them. We generally use chrono, date_time, filesystem, program_options, random, and regex directly. Particularly useful header-only libraries are algorithm, any, format, lexical_cast, range, uuid, and variant. Log and locale are more easily used through Leatherman. System and thread are required by other libraries we use.

On Mac OS X and up-to-date Linux platforms you can get a recent build of Boost using your package manager. On Windows build it from source in PowerShell with .\bootstrap mingw && .\b2 toolset=gcc --build-type=minimal install --prefix=$install boost.locale.iconv=off, or pull a pre-built package built with MinGW-w64 4.8.3. C++ compilers generally require you build with an ABI-compatible compiler, so if you use the pre-built package you should also use that compiler.

Leatherman ∞

We maintain a collection of libraries we’ve written that are useful to share across projects, or header-only libraries we’ve vendored to make available to those projects. Please add to this if you find yourself developing tools that would be useful to others. It covers a number of things:

  • catch - a C++ testing framework supporting TDD and BDD-style test structuring
  • curl - a C++ wrapper for libcurl; requires libcurl is installed
  • dynamic_library - cross-platform library for loading a dynamic library at runtime
  • execution - cross-platform command execution with options for handling input/output
  • file_util - useful filesystem utilities not covered by Boost.Filesystem
  • json_container - a container-oriented json abstraction built on rapidjson
  • locale - a cross-platform way to get a locale string to use with std::locale, because Windows; in the future this will also support i18n via Boost.Locale (or gettext directly)
  • logging - a simple logging library built on Boost.Log that takes care of Windows Unicode output; in the future it will also support lookup in message catalogs for i18n
  • nowide - a cross-platform wrapper to use UTF-8 for standard library IO on Windows
  • ruby - initializes and provides access to the Ruby runtime from a library loaded by Ruby or by loading the Ruby dynamic library
  • util - cross-platform environment manipulation, a simple regex helper, scoped system resources via RAII, and string and time utilities
  • windows - Windows-specific tools for accessing process and user info, the registry, WMI, and system errors

horsewhisperer ∞

An option parser supporting chained execution parsing, where parsing depends on what’s come before it. If Boost.Program_options doesn’t work for you, this might.

yaml-cpp ∞

INI ∞

OpenSSL ∞

libssh ∞

websocketpp ∞

Valijson ∞

cpp-project-template ∞

We’ve started a C++ project template you can derive new projects from, based around having a command-line executable and an associated dynamic library that can be linked into other programs. It includes things like

  • Enable make cpplint, a C++ linter for common style and programing errors
  • Enable make cppcheck, for fast static analysis that avoids false-positives
  • Enables -Wall -Werror to enforce most compiler static checking
  • AppVeyor CI configuration for Windows CI on Pull Requests
    • Sign in to AppVeyor using your Github account
    • If there are multiple AppVeyor accounts associated with your Github account, select 'puppetlabs'
  • Travis CI configuration for Linux CI on Pull Requests
  • Code Coverage metrics via coveralls.io
  • Dummy Rake tasks for acceptance, to fit a standard Jenkins CI template
  • Documentation via Doxygen

It exemplifies a common organizational structure we use across C++ projects, where executable and libraries inhabit separate sub-directories (exe and lib).

We use ‘.cc’ for C++ source, ‘.hpp’ for C++ header files, and ‘.c’ for pure C headers.

Vanagon ∞

Vanagon is a packaging tool to build a single package out of a project, which can itself contain one or more components. It’s used by Release Engineering for building native code projects, including Puppet Agent and a number of the libraries and tools mentioned above for pl-build-tools (both are expected to be open-sourced in the future).

Vanagon contains the build command for generating a package. It also contains the devkit command for populating a build environment and building a project on a target platform, without completing packaging and tearing the machine down, allowing easier testing or reproduction of build problems on specific platforms.

TODO: We plan to add an example for the cpp-project-template. With a Jenkins pipeline?

Cross-platform development

Cross-platform development can bring several challenges. This section provides instruction for how to approach each of them.

Development Environment

The first hurdle is often getting a development environment setup. One route is to use CLion everywhere, which provides a common GCC or Clang workflow with CMake on all major platforms.

Linux

Using an OS that updates packages regularly will make setup easier; look at ArchLinux or a recent Fedora release. The following commands will setup a dev environment with Boost and libcurl.

# ArchLinux
pacman -S make cmake gcc boost yaml-cpp curl git ruby
# Fedora
dnf install -y make cmake wget tar gcc-c++ boost-devel openssl-devel libcurl-devel git ruby

To setup cpp-project-template

git clone https://github.com/puppetlabs/leatherman
mkdir -p leatherman/build
pushd leatherman/build
cmake .. && make install -j
popd

git clone https://github.com/puppetlabs/cpp-project-template
mkdir -p cpp-project-template/build
cd cpp-project-template/build
cmake .. && make -j
Mac OS X

Using Homebrew

brew install cmake boost

Follow the steps above to setup cpp-project-template.

Windows

Using Chocolatey and Powershell

choco install -source https://www.myget.org/F/puppetlabs -y 7zip.commandline cmake git.install mingw-w64 pl-boost-x64 pl-toolchain-x64 pl-zlib-x64 pl-openssl-x64 pl-curl-x64

# Update PATH to include GCC and DLLs (so executables can find them)
$env:PATH = "C:\tools\mingw64\bin;C:\tools\pl-build-tools\bin;$env:PATH"
$N = 4

git clone https://github.com/puppetlabs/leatherman
mkdir -Path leatherman/build
pushd leatherman/build
cmake -G "MinGW Makefiles" -DCMAKE_INSTALL_PREFIX="C:\tools\pl-build-tools" -DCMAKE_TOOLCHAIN_FILE="C:\tools\pl-build-tools\pl-build-toolchain.cmake" -DBOOST_STATIC=ON ..
mingw32-make install -j ${N}
popd

git clone https://github.com/puppetlabs/cpp-project-template
mkdir -Path cpp-project-template/build
cd cpp-project-template/build
cmake -G "MinGW Makefiles" -DCMAKE_TOOLCHAIN_FILE="C:\tools\pl-build-tools\pl-build-toolchain.cmake" -DBOOST_STATIC=ON ..
mingw32-make -j ${N}

On Windows you want to limit the number of parallel processes, the example defaults to 4. These commands can be tweaked to work under Cygwin. In that case you want to use PATH=/cygdrive/c/ProgramData/chocolatey/bin:/cygdrive/c/tools/mingw64/bin:/cygdrive/c/tools/pl-build-tools/bin cmake ... to isolate the environment.

Testing

To run tests with verbose output, run make test ARGS=-V (or mingw32-make test ARGS=-V on Windows).

File Paths and System Calls

Obviously file paths on platforms differ. Refer to the file_paths specification.

Assume POSIX functions aren't available on Windows, and prefer Boost or C++ standard library functions. If specific POSIX functionality is needed, an equivalent will be needed for Windows; such helpers are good candidates for a library in Leatherman.

TODO: Add ways to get common paths on Windows.

Internationalization (i18n)

In light of today's global marketplace for software, internationalization support (abbreviated "i18n") is critical, and Puppet is making strides towards full i18n-compliance across all products. At a minimum, compliant projects must address localization of user-visible strings and provide correct handling of Unicode (UTF-8).

Localization

Leatherman provides support for marking localizable strings and generating and using message catalogs (.pot files) based on the gettext library. Please refer to the Leatherman documentation for details.

Unicode and UTF-8

Windows chose UTF-16 as the default Unicode representation, and all system calls are based on that; C++ standard library functions only support ASCII or other fixed 8-bit encodings (such as Latin-1). UNIX-based platforms generally support UTF-8 in system calls and the C++ standard library. std::string is just a byte container, so it safely holds UTF-8 characters; however when manipulating UTF-8 strings you should use a UTF-8 aware library such as UTF8-CPP or Boost.Regex. Our practice is to represent strings as UTF-8 internally, and deal with conversion when text is read/written.

TODO: Evaluate UTF8-CPP and put it on Library list.

Handling Input/Output

The driver in cpp-project-template contains useful examples of handling input/output safely across platforms.

// Use Boost.Nowide args to ensure command-line arguments are UTF-8.
// On Windows this accesses console arguments as UTF-16 and converts them to UTF-8.
// This uses the RAII pattern to modify argc/argv in-place, and revert them when exiting scope.
boost::nowide::args arg_utf8(argc, argv);

// Use Boost.Nowide cout/cerr to correctly print UTF-8 characters.
// On Windows this converts to UTF-16 before printing to the console.
setup_logging(boost::nowide::cerr);
boost::nowide::cout << "Hello!" << std::endl;

// Use Boost.Nowide cin to correctly read UTF-8 characters.
// On Windows this reads using a UTF-16 system call and converts to UTF-8.
std::string input;
boost::nowide::cin >> input;

Boost.Nowide's cout/cerr implementations come with the caveat that they aren't header-only, so if Boost.Nowide is statically linked into multiple shared libraries, input and output must be manually synchronized between those libraries.

Accessing Files

Leatherman is designed around using UTF-8 everywhere. That principally involves the use of Boost.FileSystem and Boost.Nowide.

Leatherman's file_utils make use of Boost.Nowide's iostream wrappers for reading and writing files, and Boost.FileSystem for accessing and manipulating files and directories.

// Initialize boost filesystem's locale to a UTF-8 default.
// Logging gets setup the same way via the default 2nd argument.
#if (!defined(__sun) && !defined(_AIX)) || !defined(__GNUC__)
    // Locale support in GCC on Solaris and AIX are busted, so skip it.
    boost::filesystem::path::imbue(leatherman::locale::get_locale());
#endif

// Open a UTF-8 path for reading or writing.
boost::nowide::fstream in("unicodeᐁfile");

// Rename a UTF-8 file.
boost::filesystem::rename("unicodeᐁfile", "unicode❄file");
Environment Variables

Environment variables can also contain Unicode characters, and should use Boost.Nowide wrappers for getenv and setenv to ensure they are UTF-8 encoded.