Getting Compute Engine resources for batch processing just got easier

If you need to run an embarrassingly parallel batch processing workload, it can be tricky to decide how many instances to create in each zone while accounting for available resources, quota limits and your reservations. We are excited to announce a new method of obtaining Compute Engine instances for batch processing that accounts for availability of resources in zones of a region. Now available in preview for regional managed instance groups, you can do this simply by specifying the ANY value in the API.

The capacity-aware deployment method is particularly useful if you need to easily create many instances with a special configuration such as virtual machines (VM) with a specific CPU platform or GPU model, preemptible VMs, or instances with a large number of cores or memory size.

Now, when deploying instances to run embarrassingly parallel batch processing, such as financial modeling or rendering, you no longer have to figure out which zones support the required hardware and how many instances to create in each zone in a region to accommodate the requested capacity.

Assuming that any distribution of instances across zones works for your batch processing job, and that the workload doesn’t require resilience against zone-level failure, you can now delegate the job of obtaining the requested capacity to a regional managed instance group. A regional MIG with the new distribution shape ANY automatically deploys instances to zones where resources are available to fulfill your request, accounting for your quota limits. This works both when you create a group or when you increase it in size.

If you use reservations to ensure that resources are available for your computation, you should specify reservation affinity in a group’s instance template. A regional MIG with distribution shape ANY utilizes the specified reservations efficiently by prioritizing consumption of unused reserved capacity before provisioning additional resources.