Testing

July 12, 2026 · View on GitHub

Ruby Tests
Previewing emails
Custom domains
Purchases
- Stripe
- PayPal
- Braintree
Debugging the overlay and embed widgets
Visual testing
Interacting with local environment

Ruby Tests

The whole test suite:

$ bin/rspec

A single file:

$ bin/rspec spec/requests/dashboard_spec.rb

A specific test:

$ bin/rspec spec/requests/dashboard_spec.rb:75

Dependencies

Run make local in a separate shell to spin up required services (e.g. db) through Docker.
Run RAILS_ENV=test bin/rails db:setup to set up the test database.
Run RAILS_ENV=test bin/rails js:export to update autogenerated JavaScript files for the test environment.

Fast local iteration

Run only the specs relevant to your local changes:

$ bin/rspec-changed             # specs mapped from files changed vs origin/main (+ staged/unstaged/untracked)
$ bin/rspec-changed --dry-run   # print the spec list without running it
$ bin/rspec-changed --base main # diff against a different base ref
$ bin/rspec-changed --fail-fast # unknown flags are passed through to rspec

It maps app/<dir>/**/foo.rb → spec/<dir>/**/foo_spec.rb, lib/**/foo.rb → spec/lib/**/foo_spec.rb, controllers additionally to their spec/requests counterparts, and includes changed *_spec.rb files directly. Files without a matching spec (views, frontend, config) are skipped, so it's a pre-push sanity pass, not a substitute for CI.

bin/rspec is Spring-enabled: after the first (slow, full-boot) run, subsequent runs skip Rails boot entirely by forking from a preloaded application. Some Spring tips:

$ bin/spring status         # is the preloader running?
$ bin/spring stop           # restart the preloader (after gem/env/initializer changes)
$ DISABLE_SPRING=1 bin/rspec spec/...   # bypass Spring for a pristine boot (e.g. when recording VCR cassettes)

Spring watches config/, .env* files, and Gemfile.lock and restarts itself when they change — but if test behavior looks stale or env vars don't take effect, bin/spring stop is the first thing to try.

On macOS, Spring's fork-based workers can crash inside Apple's Objective-C runtime (+[NSCharacterSet initialize] may have been in progress in another thread when fork() was called). If you hit that, set this in your shell profile:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

Integration testing

We use capybara with selenium (webdriver) for our integration specs.

We add e2e (also called request) specs for new features.
- They should cover at least the "happy path".
- They should also cover errors that are to be expected (invalid inputs, etc).
- Ideally, they would also cover some edge cases.
We also add e2e specs when making a change to an existing feature and we discover it doesn't have adequate specs.
When fixing a bug, it's better to create a failing e2e spec to reproduce it, then work on the fix. The fix should make the spec green.
We prefer to avoid unnecessary class name selectors in favor of locating elements by:
- Text content
- Input label
- Input placeholder
- ARIA attributes (might need to be added) for things that do not have a textual representation of their own
- In case none of the above works, a class name selector might be used. Ideally, not a js-* one.
For clicks, click_on "Text" or, if that doesn't work, find_and_click "selector", text: "Text", are preferred.
We prefer fill_in "locator", with: "value" or find("locator").fill_in(with: "value") over x.native.send_keys("...").
If there's an existing test that can be updated, that's preferred over a new test (which takes more lines of code and test suite running time)

Google Chrome

For integration specs, we use Chrome and chromedriver.

For Linux:

$ wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
$ sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
$ sudo apt-get update
$ sudo apt-get install google-chrome-stable

XQuartz

For macOS, you need to install XQuartz to run the request specs.

$ brew install xquartz

Reproduce Buildkite test failures locally

https://github.com/gumroad/web/issues/8410#issuecomment-318496067

Preventing flaky specs in Capybara

There are some basic tips you can use to prevent flaky specs in Capybara

Rely on expect(page).to have_selector("selector") as much as possible. This is a smart method in Capybara that handles several situations. Do not use find if you can help it. have_selector is your friend!
Do not use sleep, it is hacky and will lead to flakiness.
Use wait_for_ajax if you need to make sure all ajax requests have finished.
Test new specs multiple times, below is an example of how to run a spec many times locally:

for i in {1..10}; do
  echo "Run number $i\n-"
  bin/rspec spec/requests/product_creation_spec.rb:28
done

Troubleshooting

Failing specs?

If the spec is new or modified by your PR: It likely broke because of your changes. Try to fix it. Ask for help from the reviewer when you assign it for review if you can't fix it.

If you are sure the failing specs are unrelated to your PR: It could be a flaky spec. Confirm the spec passes multiple times locally but not on CI. Let the reviewer know when you assign it for review (example).

502 error navigating to checkout?

Try running this again:

RAILS_ENV=test bin/rails js:export

The bin/dev script automatically generates JS constants for the development domain, so without this command the tests will try to navigate to that instead of the test domain.

Cannot get chromedriver connection?

If you're running an integration spec and it times out with this error:

Selenium::WebDriver::Error::WebDriverError:
       unable to obtain stable chromedriver connection in 60 seconds (127.0.0.1:7055)

Rack times out

If you're running an integration spec and it times out with this error:

<Rack::Timeout::RequestTimeoutError: Request ran for longer than 60000ms >

You can disable Rack timeout locally. Create a file .env.test.local with the following contents:

DISABLE_RACK_TIMEOUT="1"

For the new env variables to take effect, you might need to run bin/spring stop before running the tests again.

VCR Cassettes

We use VCR to record HTTP interactions (Stripe, PayPal, etc.) so specs replay recorded responses instead of hitting real APIs in CI. Cassettes are YAML files under spec/support/fixtures/vcr_cassettes/.

When to regenerate cassettes

If your code change causes a spec to follow a new HTTP code path (e.g., removing a guard clause that previously short-circuited before an API call, changing request parameters, or adding a new external call), the existing cassette won't cover the new interaction. You need to regenerate it.

How to regenerate

Make your code change

Run the affected spec locally:

DISABLE_SPRING=1 bin/rspec spec/path/to_spec.rb

VCR automatically records new HTTP interactions into .yml cassette files
git add the new or updated cassettes under spec/support/fixtures/vcr_cassettes/
Commit them with your PR

Rules

Do not stub external API calls to work around missing cassettes. Run the spec locally to record the real interaction.
Do not share cassettes across test files. Scope them to specific tests to avoid collisions.
Delete stale cassettes when the test or endpoint they cover no longer exists.
If the spec needs real API credentials to record, see the deployment repo for credential setup.

Common scenarios

Scenario	What to do
Removed a guard clause (e.g., admin login block)	Test now reaches an API call it didn't before. Run locally to record new cassette.
Changed API parameters	Old cassette won't match. Delete it, run locally to re-record.
New test in an existing `:vcr` describe block	Just run locally. VCR auto-records.
Test passes locally but fails in CI with VCR error	You forgot to commit the cassette file.

Previewing Emails

You can preview emails locally at /rails/mailers

Custom domains

Setting up a custom domain on staging

Use domains-staging.gumroad.com instead of domains.gumroad.com in the DNS record.

Setting up a custom domain on local

Add a host entry in /etc/hosts

127.0.0.1 sample-custom-domain.example.com

Configure sample-custom-domain.example.com as a custom domain in localhost:3000/settings/advanced
Remove Rails.env.development? || part from this line
Restart the server and navigate to http://sample-custom-domain.example.com:3000

That will load the profile page of the creator through the custom domain.

Purchases

Stripe

When testing purchases, only use the test credit card numbers listed in Stripe's documentation.

Examples:

4242 4242 4242 4242
4000 0000 0000 0077

PayPal

We use Braintree as a gateway for PayPal as of writing this wiki.

Purchase flow (Buyer) - staging testing

Email: test@test.com (any email)
Password: password (any string will work)

Sandbox Gumroad account

Used for Payouts. No sandbox sales/transactions will show up here as Braintree does not integrate with the PayPal sandbox.

Credentials

Email: paypal-gr-sandbox@gumroad.com
Password: Saved in 1Password

IPN messages

We currently rely on IPN messages for:

Chargebacks and reversed chargebacks
Payouts completion notification, which also contains the payouts fee information.

Viewing IPN messages

Editing IPN endpoint

PayPal setting and other comments

No operation apart from payouts currently goes through the Gumroad PayPal app itself. We use user credentials w/ their classic API offerings to perform payouts.
Create and manage new sandbox accounts in the https://developer.paypal.com site. Use shared PayPal credentials or get an account created from someone who has payments admin access.

PayPal portal and IPN message encoding settings

At https://www.paypal.com/cgi-bin/customerprofileweb?cmd=_profile-language-encoding

Braintree

Used as a gateway for PayPal transactions. Only for transaction creation, user vaulting and refunds

Ask someone who has payments credentials for an account for Braintree in either sandbox or production.

Purchase flow (Buyer) - staging testing

Transactions made locally / staging will show up on Braintree only. Not in PayPal.

Debugging the overlay and embed widgets

The overlay and embed widgets both rely on iFrames and are best tested by running a host page on a separate web server. Sinatra is a great tool to get up and running quickly with a web page running on its own server.

Both widgets now detect the Rails application environment automatically -- simply include the script from whichever environment you wish to investigate, e.g. <script src="http://localhost:3000/js/gumroad.js"/>. The code snippets on the /widgets page also reflect the current environment.

Testing locally

There is a simple web app in our repository called web-overlay-test. Once you check it out, follow these instructions to test the overlay and/or embed.

Edit index.html to include the permalinks of your products
Run ruby web.rb
If you run into an error about missing sinatra run sudo gem install sinatra
Visit http://localhost:4567

Visual testing

We use QA Wolf for visual testing. QA Wolf is enabled by default on main branch and is not enabled by default on feature branches.

Interacting with local environment

Making a new subscription payment

# Run this in a rails console
sub = Subscription.last.charge!
sub.charge!

Note: If creating multiple subscription purchases in a short time, this method may run afoul of purchase model validations (e.g., Purchase#not_double_charged). In these cases you may comment out such validations for testing purposes.

You need to set up Apple Pay with a real card on an iPhone or touch bar Macbook. This card will not be charged on local and staging environments.
Add the domain to Stripe's list of Apple Pay Domains if testing on a deploy branch. discover.staging.gumroad.com is already added for staging. For local development see Apple Pay docs — the app must be exposed over HTTPS (e.g. via ngrok) since Stripe requires HTTPS for Apple Pay domain registration.

To overcome this:

# Run this in a rails console
purchase = Purchase.last
BlockedObject.browser_guid.active.find_by(guid: purchase.browser_guid).destroy
purchase.destroy

Making another purchase with a different buyer email should work now.