4 steps to stop data exfiltration with Google Cloud
October 3, 2019
Deep Insight into "FIN7" Malware Chain: From Office Macro Malware to Lightweight JS Loader
October 3, 2019
If you’re part of a large enterprise using BigQuery, you’ll likely find yourself using BigQuery’s flat-rate pricing model, in which slots are purchased in monthly or yearly commitments as opposed to the default on-demand pricing. Enterprises favor flat-rate pricing because it gives your business predictable costs, and you’re not charged for the amount of data processed by each query.
In the flat-rate model, you pay for your own dedicated query processing resources, measured in slots, so you’ll likely want to manage how your business consumes these slots. You have the option to manage your BigQuery footprint by partitioning your purchased slots into reservations, and then assigning your Google Cloud Platform (GCP) projects to these reservations. Projects inside a reservation will have priority to use the reservation’s slots over other projects outside the reservation.
This flat-rate model presents a question we often hear from users: Can I allocate BigQuery slots at a more granular level than the GCP project level? These users generally have multiple applications inside the same GCP project, each with unique BigQuery resourcing needs, or just one application with varying resourcing needs (e.g. Apache Airflow running BigQuery jobs of varying priorities).
You should ideally separate your applications into their own projects, but what if you have multiple applications running on the same infrastructure (Hello, containers!)? We’ll describe here how to configure applications within the same project so their queries execute within separate projects with their own parent slot reservations.
Configuring applications for granular slot allocation
The diagram below describes a simple environment that accomplishes this. It uses three GCP projects. Two applications reside in Project_A: Application 1 to read from BigQuery and Application 2 to write to it. Project_B and Project_C exist solely to run queries executed by those applications.
