Heroku’s Preboot feature is great, it enables seamless deployments on Heroku with a single command:
heroku features: enable preboot
Without Preboot downtime would be required on every release.
Preboot works by creating a new set of web dynos in the background for each release, after ~3 minutes the router then switches traffic to the new dynos. That 3 minute window gives your app servers enough time to boot and be ready to service requests, resulting in 0 downtime deployments!
However, preboot is pretty dumb
It does nothing to check the health of the new dynos. Once the 3 minute timer is up requests will be routed to your new dynos regardless of the state they’re in, even if they’ve crashed.
Ideally in a Blue-Green deployment system you have some kind of smoke test on the new release before switching traffic to it.
Solution: use the Release Phase to perform a smoke test
The release phase is used to perform commands before each release is deployed, traditionally things like database migrations. Most importantly if the release command fails, the deployment is halted. We can take advantage of that fact to perform a health check on every release.
Here’s a snippet taken from our release command:
export PORT=3210 export HOST=127.0.0.1 # Boot a web server in the running locally in the release dyno # This should be the same as the 'web' command in your Procfile (bin/start-nginx bundle exec puma -C config/puma.rb) & SERVER_PID=$! # Wait for the server to boot and for it to be listening for requests on the port while ! nc -z $HOST $PORT -w 1; do # Check to see if the server process is still running or if it has crashed if ! kill -0 $SERVER_PID; then exit 1 fi sleep 1; done # Once the server is up attempt to make a request to the `health_check` endpoint response=$(curl http://$HOST:$PORT/health_check) # Check to see if the response body is 'OKAY' if [[ "$response" != "OKAY" ]]; then echo '[Health Check]: FAILED' echo $response # Cleanup the localhost server kill $SERVER_PID exit 1 fi echo '[Health Check]: PASSED' kill $SERVER_PID exit 0
You can see from the snippet we are able to boot a server inside the release command and attempt to make a request to a ‘health_check’ endpoint. If the server doesn’t boot or the request returns an error the release command fails simultaneously preventing the deployment and saving our bacon.
This post is written by Nick Maher, an engineer in the Childcare Development squad at Koru Kids.