In this blogpost you will learn how to build an open-source cloud asset inventory with CloudQuery and Grafana.

General architecture:

• CloudQuery will take care of extracting, transforming and and loading all your assets, across cloud and SaaS apps to PostgreSQL.
• Grafana will be used to query, visualizeVisualize & Monitor, and alert.

This is what you will get:

• All your assets configuration across cloud providers and SaaS apps in one database
• Vanilla PostgreSQL
• Reuse your current (assuming you use grafana) visualization, monitoring and alerting workflows
• 23 open-source filterable asset inventory dashboards for AWS and GCP including security & compliance dashboards.

Running

To try out the below dashboards (or build a custom one) you will need to connect the same PostgreSQL used by CloudQuery to Grafana as a datasource.

You can run out CloudQuery locally or in your cloud environment in a cron to update your PostgreSQL/RDS/Cloud SQL. See Quick Start here

Importing Dashboards

You can try out some of our pre-made dashboards by importing them straight from our GitHub

Dashboards Examples and Use Cases

Asset (Resource) Search

One of the most common but a lot of times non-trivial tasks is to find a specific when the only identifier you have is one piece of information across accounts and regions. Just a few examples (as number of real-world scenarios is really indefinite ):

• Finding an asset across accounts/regions by name/arn: In AWS specifically it might involve either clicking through 30+ regions, if you know in which account it is located or even more if not.
• Finding an ec2 instance by it’s public/private IP: This will also either involve click-ops, or bash sorcery.

Some of those can be also solved by AWS Config but has the following limitations:

• AWS Only - Works only on AWS resources (can’t ingest data from other services/cloud-providers).
• As this is using a proprietary subset of SQL, it can’t be integrated to your current Visualization, Monitoring Alerts workflows, such as Grafana.

This is why we created open-source “basic inventory” grafana dashboards for 23 services that you are free to use, customize or build completely new ones (feel free to share back or suggest other):

Filterable AWS dashboards by accounts and regions

Here is snippet from our EC2 dashboard:

Filterable GCP dashboards across projects

Similar challenge exists in GCP though the situation is a bit better in some sense as you have a single view for each resource/asset type per project. Though, If you want to have a single view of all types of assets across multiple projects (which is very common), this would be clickops or bash magic again.

Here is an example of GCP Compute Asset inventory dashboard:

Some of that can be solved with the in-house GCP Cloud Asset Inventory but has similar limitations:

• GCP Only - Works only on GCP resources (can’t ingest data from other services/cloud-providers).
• Custom Query Language (not sql) - You will need to learn a new query engine that might be also more limited then SQL.
• Can’t integrate with best-in-class visualization, monitoring alerting systems such as Grafana.
• Cannot integrate across different organization accounts

Security

You can create your own security views and dashboards that you can then monitor and alert. Each company has its own security and compliance policies but we will share a basic one in this blog (more is coming…) we found useful.

AWS Public/Private EC2 Instances

Filterable dashboards by vpc, subnet, region including public, private ec2 instances.

Summary

Hopefully by now you are as excited as us for the future of open-source cloud asset inventory and we are looking to hear your feedback, either on GitHub or Discord. Also, feel free to contribute back or request additional Grafana dashboards here.