PlanningAlerts

August 28, 2026 · View on GitHub

example workflow View performance data on Skylight

Find out and have your say about development applications in your area.

This is the code for the web application side of things written using Ruby on Rails.

If you're interested in contributing a scraper read our step-by-step guide to writing scrapers on our scraping platform, morph.io.

PlanningAlerts is brought to you by the OpenAustralia Foundation. It was created by Matthew Landauer and Katherine Szuminska.

Development

Install Dependencies

Checkout The Project

git clone https://github.com/openaustralia/planningalerts.git
cd planningalerts

Resetting Your Environment

To force a rebuild from scratch, stopping and removing containers and volumes (including the databases):

docker compose down -v --rmi all
# Check if this volume is "still in use"
docker volume rm planningalerts_gem_cache

If you get a volume is in use error, remove the container using the given id and then retry, for example:

> Error response from daemon: remove planningalerts_gem_cache: volume is in use - [98ba717d944d01976ac9074e6a1119e70d3aebd53f5d5449957324926f0bbb4b]

docker rm 98ba717d944d01976ac9074e6a1119e70d3aebd53f5d5449957324926f0bbb4b
> 98ba717d944d01976ac9074e6a1119e70d3aebd53f5d5449957324926f0bbb4b

docker volume rm planningalerts_gem_cache

Overriding DB host and port for non docker dev

To use docker for the database, but allow you to run the application locally, for example to simplify single step debugging in IDE's, you can override the following ENV vars in .envrc (for direnv) or manually:

export DB_HOST=localhost
export DB_PORT=15432

Alternative web port

You can set up a second web server port so all your user and passwords from various projects are not all mixed together. For example, add the following to docker-compose.override.yml to use port 30PA (3072):

services:
  web:
    ports:
      - "3072:3000"

This adds an extra port mapping, local port 3000 will still be mapped to port 3000 on the container as well.

Setup The Database

Set up the databases - docker compose run web bin/rake db:setup

This will trigger a build if needed. You can manually trigger a build by first running - docker compose build

Start the application

Append --build to rebuild any changes to Dockerfile.

Run The Tests

  • In a separate window - docker compose run web bin/guard
  • Press enter to run all the tests

Setup Development Data

Initially db/seeds.rb (called by db:setup will have set up:

  • user admin@example.com
    • confirm via the email at http://localhost:1080/
    • Add the admin role using the rails console command User.first.add_role(:admin)
  • Authority: Marrickville Council, with
    • Application to demolish a swimming pool at 28 Grey Street, Emu Plains NSW 2750

Note: external links to morph.io and council site for this example are broken as the external site details have changed in the last 5+ years.

Using production data locally

If you want to use production data locally, follow the instructions in the doc/using_production_data_locally.md file.

Emails in development

In development all emails are sent locally to mailcatcher. The emails can be viewed at http://localhost:1080.

Email templates are developed using Maizzle. The development workflow goes as follows:

cd maizzle
npm run dev production

Then point your browser at http://localhost:3000/rails/mailers/. As you edit the templates in maizzle/src/templates, the templates will get automatically compiled to the erb template in the main rails app tree and will also refresh the browser.

Type checking

We're using Sorbet to add type checking to Ruby which otherwise is a dynamic language. To run the type checker:

docker compose run web bin/srb

To update the rbi files, use these commands:

docker compose run web bin/tapioca gem
docker compose run web bin/tapioca dsl
docker compose run web bin/tapioca dsl --environment=test

We use Shopify's tapioca gem to manage all our rbi files. We don't use bundle exec srb rbi ....

Other useful docker commands

  • docker compose ps - View running containers and their status
  • docker compose logs [-f] [service] - View logs for all or a specific service, add -f to continue watching (the up command also does this)
  • docker compose restart web - Restart the web container (eg when you make a change to the code)
  • docker compose down - Stop and remove containers (keeps volumes and images intact for quick restart)
  • docker compose exec web bash - Run a shell in the running container
  • docker compose run --rm web bin/rails console - Run a once-off command in new container and then clean it up afterwards
  • docker system prune --all - Remove all stopped containers, orphaned images / networks, build cache etc. (Remove the -v if you want to keep your databases and gem cache)

If things go wrong with gem updates

If things go wrong, force a rebuild from scratch (preserving DB volumes):

docker compose down
docker system prune --all
docker volume rm planningalerts_gem_cache
docker compose up --build

Then rebuild sorbet again in another terminal window:

docker compose run web bin/srb
docker compose run web bin/tapioca gem
docker compose run web bin/tapioca dsl
docker compose run web bin/tapioca dsl --environment=test

Deployment

The code is deployed using Capistrano. To deploy to production run:

bundle exec cap production deploy

This command is defined in config/deploy.rb.

Sometimes you want to deploy an alternate branch, for instance when deploying to the test stage. In this case you'll need to set the branch variable after recipes are loaded by using the --set argument instead of --set-before, e.g.

bundle exec cap staging --set branch=a-branch-i-want-to-test deploy

View more available Capistrano commands with:

bundle exec cap --tasks

Sentry release tracking

Deploys automatically record a release and deploy in Sentry (org oaf-org-au, project planning-alerts) so issues can be tied to the deploy that introduced them. This runs via a Capistrano hook using the Sentry CLI on your machine (part of #2049).

The hook supports both CLI v4 (binary sentry) and the legacy v3 (binary sentry-cli), preferring v4 when both are installed. See the v4 migration guide.

One-time setup per deployer machine (v4, recommended):

  1. Install the CLI: brew install getsentry/tools/sentry on macOS, or curl https://cli.sentry.dev/install -fsS | bash on Linux
  2. Authenticate: run sentry auth login — it opens Sentry in a browser to authenticate (the token needs the project:releases and org:read scopes)
  3. Verify: sentry auth status should report access to the OpenAustralia Foundation (oaf-org-au) org

If you still have v3 (sentry-cli), it keeps working: authenticate with sentry-cli login and verify with sentry-cli info.

Don't verify v4 with sentry info. It exits non-zero whenever no default org/project is set, even when your token is fine, so it isn't a usable check for whether the hook will run (#2183).

The repo's committed .sentryclirc holds the org and project. v3 reads that file directly. v4 ignores it, so the deploy hook reads the values out of it and passes them on the command line, which keeps the one committed file authoritative for both. If you want v4 to pick the file up for your own ad hoc commands, run sentry cli import.

Credentials are separate from that file and must never be committed. v3 reads your token from ~/.sentryclirc; v4 stores its own credentials when you run sentry auth login.

If the Sentry CLI is missing or unauthenticated the deploy still succeeds — the hook prints a warning and skips recording the release, so set it up before your next production deploy.

whatismyip trust-boundary check

GET /whatismyip reports the IP Rails sees for the request, to aid checking Cloudflare/ALB proxying with a one-line curl rather than a real sign-in or a log dig. It returns the IP, or the IP plus FAIL if that IP falls within Cloudflare's own published ranges, a sign the trust boundary isn't rewriting it to the real visitor IP. If Cloudflare's published ranges can't be fetched or look implausibly short, it returns the IP plus UNABLE TO CHECK rather than risk reporting a false pass.

Off by default; turn it on for a server with the provide_whatismyip Flipper feature flag.

curl https://www.planningalerts.org.au/whatismyip

Upgrading Ruby in production

Upgrading Ruby in production is an unbelievably painful process right now. I'm sorry. Let's make it simpler but in the meantime:

TODO: #2167 This section is now out of date as we now have a blue/green deploy setup that allows us to build a new image with a new ruby version and deploy it side by side with the old version and switch over between them

Note for new deploy on new servers: Make sure you run "bundle exec cap production puma:install" first. Otherwise the "puma:reload" capistrano task will fail

Update the application code

  • Change .ruby-version. Run tests to make sure nothing has broken.

Upgrade ruby in staging

  • In the oaf/infrastructure repo update roles/internal/planningalerts/meta/main.yml to add the new ruby version before the current one. The last listed one is the default. We don't yet want to change the default
  • Install the new ruby on the server by running ansible-playbook site.yml -l planningalerts. Remember to set your python virtual environment if you're using that.
  • Deploy new version of the application with upgraded .ruby-version to staging by running bundle exec cap staging deploy
  • Login to each webserver in turn (as root user). Then, cd /srv/www/staging/current; gem install bundler:1.17.3
  • Login to each webserver in turn (as deploy user). Then cd /srv/www/staging/current; bundle install --gemfile Gemfile --path /srv/www/staging/shared/bundle --deployment --without development test. This step is necessary if you're upgrading a ruby major version. You might be able to skip it if not.
  • Edit roles/internal/planningalerts/templates/default to change the ruby version used by passenger for staging in staging to the new version
  • Run ansible again with ansible-playbook site.yml -l planningalerts
  • Check deployed staging is still working by going https://www.test.planningalerts.org.au

Upgrade ruby in production

  • Deploy new version of the application with upgraded .ruby-version to production by running bundle exec cap production deploy
  • Check deployed production is still working by going https://www.planningalerts.org.au
  • Login to each webserver in turn (as deploy user). Then cd /srv/www/production/current; bundle install --gemfile Gemfile --path /srv/www/production/shared/bundle --deployment --without development test. This step is necessary if you're upgrading a ruby major version. You might be able to skip it if not.
  • Edit roles/internal/planningalerts/templates/default to change the ruby version used by passenger for production in staging to the new version
  • Run ansible again with ansible-playbook site.yml -l planningalerts
  • Check deployed production is still working by going https://www.planningalerts.org.au

During this keep a close eye on disk space on the root partition as this might get close to full

Tidy up

  • Remove old ruby version from roles/internal/planningalerts/meta/main.yml in the oaf/infrastructure repo
  • Rerun ansible with ansible-playbook site.yml -l planningalerts
  • If you did a ruby major version upgrade (e.g. ruby 2.6.6 to 2.7.4) then you should also clean up an old unused bundler directory that is taking up a lot of space. Login to each webserver in turn (as deploy user). Then cd /srv/www/staging/shared/bundle/ruby and remove the unused directory (e.g. 2.6.0). Do the same for production cd /srv/www/production/shared/bundle/ruby.

Adding a new authority

Someone has just written a new scraper for PlanningAlerts, woohoo! :tada: Now we need to add it to the site.

The first step is to fork their repository into the @planningalerts-scrapers organisation. This gives us control over the repository. If we didn't do that then the person could potentially inject bad data without us noticing. A more likely problem is that they go off and do something else and we have no control of the repository to fix things.

Once you've done that add it to morph and do an initial scrape to get some data. It's always a good idea to check that the scraper is getting the data we expect. Just like you'd do if someone had fixed a scraper and opened a pull request.

Speaking of pull requests, because we've forked the scraper GitHub turns off issues on forked repositories. It's a good idea to switch it back on for ours so that other people can open issues and pull requests against the @planningalerts-scrapers repository.

Now that we have a working scraper and some data we can add the new authority to PlanningAlerts. First, log into the admin backend and browse to the authorities section:

https://www.planningalerts.org.au/admin/authorities

Click New Authority in the top-right of the page. Now fill out all the details, here's what needs to go in each field:

  • Full name: The full name of the authority that's displayed throughout the site, e.g. Bellingen Shire Council
  • Short name: Used in the URL (must be unique), e.g. bellingen
  • Email: email address that comments get sent to. Try and find a specific DA comment address or failing that use the main contact email address for the council, e.g. da_comments@bellingen.nsw.gov.au
  • Scraping, morph name: The morph name of the scraper you just forked, e.g. planningalerts-scrapers/bellingen

Click Create Authority. Now scrape some applications so you can see them on the new authority page - click Scrape.

Visit the new authority page, e.g. https://www.planningalerts.org.au/authorities/bellingen/

You should see that some applications have already been collected. If not wait a few seconds and refresh the page. Once you've got some do a quick check on a few by clicking Browse all recent applications and selecting a few. Make sure the comment form is visible (that means you set an email address).

If all looks good then thank the contributor for helping tens of thousands of people get PlanningAlerts by tweeting about it from @PlanningAlerts. It's always fun to @mention the council too, sometimes we get a RT :grinning:

We've just added @BellingenShire thanks to @LoveMyData. Another 12,886 people can now get PlanningAlerts! e.g. https://www.planningalerts.org.au/applications/898071

Periodic jobs

We use sidekiq-cron gem to run system-wide periodic jobs that are not tied to a specific server. As noted in Gemfile Use "regular" cron for jobs that need to run on every machine (though none are currently defined for the deploy user).

See config/cron.yml for the list of periodic jobs.

Utilities

You can run rake tasks on the server using capistrano, for example:

bundle exec cap production rake:invoke TASK=my:rake_task
# Or with maintenance on before and off afterwards
bundle exec cap production rake:invoke:with_maintenance TASK=my:rake_task

Useful rake tasks:

  • db:version - Retrieve the current schema version number
  • db:stats - Show table count and size

Or you can run them locally, eg bundle exec rake db:stats

Contributing

  • Fork the project on GitHub.
  • Make a topic branch from the main branch.
  • Make your changes and write tests.
  • Commit the changes without making changes to any files that aren't related to your enhancement or fix.
  • Send a pull request against the main branch.

To run style and coding checks

bundle exec rubocop

To check for security updates

Either check Dependabot alerts for the main branch, taking note of when the last check was run, or run manually on current branch:

bundle exec ruby-audit
bundle exec bundle-audit

Credits

Our awesome contributors can be found on the PlanningAlerts site.

This project is tested with BrowserStack.

License

GPLv2, see the LICENSE file for full details.