Migrating from Google App Engine to Google Cloud Run

I want to share how we migrated from Google App Engine to Google Cloud Run. Besides reducing our Google Cloud bill by 50%, we now enjoy better observability.

Background

The migrated app was an Angular SPA with a NodeJS backend connected to a Google SQL database and a couple of cron jobs. We started with Google App Engine because it was easy to set up and required no additional CI/CD setup. After a bit more than a year, our monthly bill climbed to over 1000€, and with over 20 requests/s at peak hours, we occasionally saw "Bad Gateway" errors, which were hard to debug. Additionally, a few things about App Engine annoyed us:

• Instances could only be resized in fixed steps (at least for the standard environment)
• You need to have a default service
• The version history was limited, so you needed to clean up old revisions
• Limited and hard-to-understand monitoring (e.g., I did never fully understand how many instances were running at a given time)
• The build process was a bit magic, and we needed to rebuild when deploying from staging to production, which violates 12-factor app principles.

After some research, we decided to migrate to Google Cloud Run, which is a more up-to-date product.

Preparation

• We reduced DNS TTLs for both the front end and the back end to the minimum possible value for our provider (in our case, 5 minutes).
• We created a migration document with detailed instructions for the migration and a checklist for tests after the migration
• We evaluated the best time for a maintenance window and communicated it to our customers
• We tested the migration with our staging environment after having set up all the instructions
• Changed cron job authentication to something based on headers, as App Engine cron jobs were protected by IP addresses.
• Set up CI/CD to build our application using buildpacks and push to Google Artifact Registry.
• Using this tool, we replaced our App Engine YAML files with Cloud Run YAML files.

Migration

Because our application is stateless, we could run the application both on App Engine and Cloud Run in parallel. So, migration was more or less only DNS changes. The hard part was setting up custom domains for Cloud Run. We decided to go for an External Application Load Balancer as custom domain mapping was still in beta and not available for our region. I hope this is soon available as this would make things a lot easier – similar to how it was using App Engine. Additionally, cron jobs had to be migrated to Cloud Scheduler. This was done after everything was running smoothly.

The migration itself was really smooth. With a total downtime of no more than 10 minutes due to DNS changes, our impact on business was minimal.

Result

We reduced our monthly bill by 50% while enjoying a better developer experience with better metrics.