Metrics tracking

February 21, 2020 · View on GitHub

Best practices for measuring engagement on open source projects detail a myriad of data points to potentially be measured depending on how you define success and why you’re investing in open source (e.g., avoid duplicative custom software purchases, promote innovation and collaboration, popularity, being transparent about code or just having fun).

Regular tracking can also serve as an early warning system that allows for course correction, if necessary, and can help maintainers make better decisions. Per The Linux Foundation’s Tools for Managing Open Source Programs, many organizations are now creating a dashboard of metrics to track all of the data in one place and provide snapshots at a glance. With more information, you can:

  • Understand how users respond to a new feature
  • Figure out where new users come from
  • Identify, and decide whether to support, an outlier use case or functionality
  • Quantify your project’s popularity
  • Understand how your project is used
  • Raise money through sponsorships and grants

Example Metrics (to be measured monthly, quarterly, or annually):

  • Number of contributors (and the ratio of external to internal contributors): projects that consistently attract new external contributors are likely doing a good job maintaining the project, welcoming contributors, and incorporating feedback from the community
  • Number of pull requests submitted, open, and accepted (and length of time they remain open): tracking the number of PRs, and what happens with them, demonstrates how much code is being submitted by external contributors and is an indicator of outside interest; length of time PRs remain open also indicates how responsive and welcoming project maintainers are
  • Number of issues submitted (and length of time they remain open): users who don’t have the time, permission, or ability to submit a PR, but encounter problems with code can submit their bugs and feature requests as an issue
  • Number of commits per contributor (external vs. internal): number of external commits is an indication of how effective the project is at innovating in the open
  • Diversity of contributions (commits, fixing typos or bugs, commenting on an issue): the more varied the types of contribution, the better. Per GitHub’s Open Source Guide - “Open source is more than just code. Successful open source projects include code and documentation contributions together with conversations about these changes.”
  • Number of external adopters: each OS project should have a way to track organizations that opt to adopt the software in a production environment (in an ADOPTERS.md or listed in the README.md)
  • Number of projects created or contributed to (program-wide): it’s important to measure your organization’s open source success not only by the health of your own open source projects, but by its open source activity as a whole. This includes the health of the projects you rely on for product development and business operations
  • Maintainer Activity: are maintainers able to handle the volume of contributions received? Are they responding to the community? Track how long it takes to respond to contributions (not address), average time an issue remains open, whether issues get closed by PRs, whether stale issues get closed, and average time to merge a PR
  • Popularity/awareness indicators include: visitors to the project website; total number of followers or stars on GitHub; number of followers on social media accounts (e.g., Twitter, Medium, LinkedIn, etc.); news clips and media mentions; and number of meetups organized and hosted
  • Adoption metrics include: number of downloads; number of forks created; number of contributing external companies; stages of adoption (# of deployments in proof of concept and production); and number and quality of commercial dependencies

Resources