Deploying the Cloud Spanner Emulator remotely

Welcome to the third part of our series on the Cloud Spanner Emulator. In the first part, we got an overview of Cloud Spanner and the emulator, as well as the various ways that it can be provisioned. In the second part, we explored the various options available for running the emulator locally, as well as how to build the emulator as one of the components in an application container.

The emulator can also be deployed on a remote GCE instance or Cloud Run. Today, we will deploy the emulator services on a GCE host manually and via Terraform. Finally, we will also run the emulator on Cloud Run.

Cloud Spanner emulator on GCE

In the previous post, we have deployed the application and the emulator on separate containers by attaching both containers to a Docker network. In environments with VPC / project level separation for dev, stage and production it might be useful to run an emulator on a dedicated remote host. Apart from the ability to point your applications to the public/private IP of the emulator instance, this also allows for collaborative troubleshooting of failed test cases, etc.

Manual deployment

This section covers the steps to provision a GCE instance and start the emulator services. For the sake of completeness, it has instructions starting from creating a VPC; however, you can skip this and make changes according to your environment.

If you have been working through this series so far, your gcloud config is likely set to the emulator. Before you proceed with the commands below, please switch to a different configuration (e.g., your default one)

Next, you need to ensure the default gcloud configuration is set correctly. Below we are enabling authentication, unsetting any API endpoint URL set previously, and setting the GCP project we intend to use in the default gcloud configuration.

Create a VPC, subnet and firewall rules (you might want to edit the firewall rule source range to be more restrictive):

Create an emulator VM. We will run the emulator service with the instance creation itself by passing –metadata startup-script. Replace the placeholder [Your-Project-ID] with your GCP project ID.

Once the instance comes up and the emulator services are running, you can follow the instructions from the earlier blog posts to deploy the sample application. The only difference is that we change localhost to the public IP address (or private IP address if you are working from the same VPC or connected via VPN).

NOTE – If you are using a public IP address here, all of the data exchanged between your client and the remote emulator will be transmitted in plain text over the internet. Please ensure that you’re not sending privacy-sensitive data.

Example configuration below:

Provisioning an emulator GCE instance via Terraform

In an environment that follows the GitOps style of deployments, having a Terraform template for provisioning the emulator instance can be useful as well. You can follow the instructions below to spin up an instance with the emulator running.

Clone this repo which contains the Terraform code modules that we will be using:

First, we will have to initialize Terraform so it would download and install the provider plugin and configure the child module.

Open the terraform.tfvars file and edit the name of the VM, Project ID, Region and Zone based on your environment.

Initialize Terraform:

And apply:

You can now connect to the VM verify if the emulator services are up and running.

Once the VM is up and running with the emulator services started, you can use the VM’s public or private IP address to configure SPANNER_EMULATOR_HOST and connect, similar to what we described in the Manual Deployment section above.

Cloud Spanner emulator on Cloud Run

Since the emulator is available as a pre-built Docker image (or you can manually build from the source), deploying the emulator services on Cloud Run is straightforward. Cloud Run supports gRPC (after enabling HTTP2). However, it is important to remember that when using Cloud Run, you will be able to route requests to only one port at any given point in time.

NOTE – While it is possible to run multiple processes inside the same container in Cloud Run (in this case, gRPC server and REST server), the requests can only be sent to one port.

If your application uses only client libraries or RPC API (gRPC), then you can configure Cloud Run to send requests to the 9010 port. If you use only the REST API, then you can configure port 9020. Also, remember to set minimum instances to ‘1’ to prevent the container from being removed and thereby losing data when there is no activity.

You can choose from any of the following options:

Option 1: Deploy emulator gRPC on Cloud Run

Option 2: Deploy REST server on Cloud Run

NOTE – Avoid using both options simultaneously, as that will create two different instances of the emulator, and two different copies of your database that would be out of sync, i.e. depending on which emulator instance serves your request, you may get completely different results.

Once done, you can configure your emulator profile pointing to Cloud Run’s URL like below:

If you are using REST server:

If you are using gRPC:

NOTE – We have not specified a port number above since the requests to the Cloud Run URL already route to the port used (9010 or 9020) directly. Therefore, you just need to add the Cloud Run URL, without the port.

Conclusion

Through this 3-part series of blogs, we introduced the Cloud Spanner Emulator and detailed various options available to start and use the emulator both locally and on a remote host. We also demonstrated how the emulator can be used in a development workflow as a no-cost experience for Cloud Spanner using a sample application.

Hope you found this useful!

To learn more about Cloud Spanner, visit the product page here and to learn more about the Cloud Spanner emulator, please see the documentation here.