Release with confidence: How testing and CI/CD can keep bugs out of production

If you look closely, you’ll notice that this test is exactly the same as the integration test. In fact, the only difference is that the BASE_URL variable is set so that the test points at a deployed Cloud Function instead of a locally-hosted one.

Though this trick is often specific to HTTP-triggered functions, reusing integration test code in system tests (and vice versa) can help reduce the maintenance burden created by your tests.

Other testing options

Let’s take a quick look at some other common types of testing, and how you can best utilize them with Cloud Functions.

Static tests

Static tests verify that your code follows language and style conventions and dependency best practices. While they are relatively simple to run, one major limitation you have to account for is their narrow focus.

Many static test options are free to install and easy to use. Linters (such as prettier for Node.js and pylint for Python) enforce style conventions, while dependency tools (such as Snyk for Node.js) check for dependency issues.

Load tests

Load tests involve creating vast amounts of traffic and directing it at your app to make sure your app can handle real-world traffic spikes. They verify that the entire, end-to-end system–including non-autoscaled components–are capable of handling a specified request load, which is usually a multiple of the peak number of simultaneous users you expect.

Load tests can be expensive, since they require lots of billed resources to run, and slow due to the external dependencies they rely on. On the plus side, many of the actual testing tools are free, including Apache Bench (“ab” on most Mac and Linux systems),Apache JMeter, and Nordstrom’s serverless-artillery project.

Security tests

Security tests verify that code and dependencies can handle potentially malicious input, and can be part of your unit, integration, system, or static testing. Beware: security tests have the potential to damage their target app environment. For example, a testing tool may attempt to drop a database or otherwise compromise the resources in its environment. The lesson here is: make sure to use a test or canary environment unless you are 100% sure the tool in question won’t hurt your production environment.

There are many free security testing options out there, including Zed Attack Proxy, Snyk.io, the Big List of Naughty Strings, and oss-fuzz, just to name a few. However, no automated security testing tool is perfect. If you are serious about security, hire a security consultant.

CI/CD FTW

At the beginning of this post, we mentioned two ways to minimize the frequency of bugs: testing and CI/CD. Now that we’ve covered testing, let’s take a look at how continuous integration and continuous deployment can provide an additional layer of defense against bugs in production.

The motivation for CI/CD is fairly straightforward. If you’re a developer, version control–whether it’s git branches or another system–is your source of truth. At the same time, code for Cloud Functions has to be tested and then redeployed manually. This presents no shortage of potential issues.

CI/CD systems automate this process, letting you automatically mirror any changes in version control to GCF deployments. CI/CD systems detect code changes using hooks in version control systems that are triggered whenever new code versions are received. These systems can also invoke language-specific command-line functions to run your tests, followed by a call to gcloud to automatically deploy any code changes to Cloud Functions.

There are many different CI/CD options available, including Google’s own Cloud Build, which natively integrates with GCP and source repositories. A basic CI/CD for Cloud Functions is fairly simple to set up and deploy with Cloud Build–see this page for more details.

In conclusion

Writing a thorough and comprehensive test suite, running it in a realistic “canary” environment, and automating your deployment process using CI/CD tools are techniques that can help you reduce your production bug rate. When used together, they can significantly increase the reliability and availability of your services while decreasing the frequency of buggy code and its resulting negative business impacts.

However, as we cautioned at the beginning, testing simply can’t catch every bug before it hits production. In our next post, we’ll discuss how to minimize the business impact of bugs that do make their way into your Cloud Functions based applications using monitoring and in-production debugging techniques.