Microsoft Azure Training Day: Fundamentals
October 18, 2019
The Good, the Bad and the Ugly in Cybersecurity – Week 42
October 18, 2019
Editor’s note: We’re hearing today from DoiT International, a Google Cloud Premier and MSP partner, and two-time International Partner of the Year. They recently built a Grafana plugin for BigQuery, making it easier to visualize your data. Read on for details.
At DoiT International, we see data problems of all shapes and sizes. From complexity analysis to large-scale system design, there are a variety of tools that can help solve our clients’ technology and analytical needs. But sometimes there’s a tool that seems so necessary that we create and share it ourselves.
Which is why we built the Grafana plugin for BigQuery.
We love BigQuery for its unparalleled capability to execute queries very fast over very large datasets, and often encourage our customers to use it. We also see how much our customers love using Grafana to visualize their time-series data for monitoring, alerts, analysis, or some combination thereof. The two seem like a natural match, yet until recently, there wasn’t a way to bring them together.
Fortunately, Aviv Laufer, senior cloud engineer at DoiT International, found a way. Already familiar with the BigQuery API, he dug into the Grafana documentation and had a working prototype within a few weeks, and released a beta version shortly thereafter. After about a month, we’d solved the major bugs, become production-ready, and have been fielding feature requests from the community ever since.
Monitoring big data operations
Hundreds of companies are already taking advantage of the plugin so they can use both tools to their fullest extent. King, for instance, is using it to monitor the company’s big data operations. The mobile game developer, which famously brought the world Candy Crush Saga back in 2012, runs their data warehouse entirely in Google Cloud and uses BigQuery’s flat-rate subscription model. As King’s usage grew to support hundreds of projects, they were having trouble measuring slot utilization at the reservation or project level. They needed a better way to assess their usage patterns and query efficiency than scraping metrics from the Stackdriver API and consolidating those into yet another project to analyze with Grafana.
Since King was already piloting an alpha of the flat-rate usage export into BigQuery, and was familiar with using Grafana with Stackdriver, the plugin let them tap into the best of both worlds. For example, the following short standard SQL query obtains slot usage by project:
