Tutorials

Building an Infrastructure Data Lake with MongoDB

Introduction

The cloud infrastructure landscape has grown exponentially in recent years, bringing with it increased complexity and an overwhelming amount of data. To effectively manage this data and ensure that cloud resources are being optimized, developers need powerful tools that are both flexible and scalable. In this blog post, we will explore how CloudQuery, when combined with MongoDB, can help developers build a robust cloud infrastructure data lake for improved management and monitoring of cloud resources.

What is CloudQuery?

CloudQuery is an open-source ELT (Extract, Load, Transform) tool designed to transfer data from supported sources to supported destinations, such as your cloud infrastructure data to MongoDB. By pulling data from various cloud providers and platforms, CloudQuery enables developers to create a holistic view of their infrastructure, simplifying the management and monitoring process.

Why MongoDB?

MongoDB, a widely-used open-source NoSQL database, boasts exceptional performance, scalability, and versatility.
Its document-based data model is tailored to handle significant amounts of unstructured or semi-structured data, a common feature in cloud infrastructure data.
As MongoDB is particularly adept at storing and querying JSON data, it is well-suited for managing cloud infrastructure data that frequently consists of JSON objects.
This proficiency allows for streamlined management and organization of the various data structures found in cloud infrastructure management.
By utilizing MongoDB as the destination for CloudQuery, developers can efficiently store and manage their cloud infrastructure data in an organized fashion.

Setting up CloudQuery

To start building your cloud infrastructure data lake with CloudQuery and MongoDB, you'll first need to set up CloudQuery. Follow the quickstart guide, which offers precompiled binaries for various platforms, including Windows, macOS, and Linux. Follow the links to the guide for your respective platform.

Next, create a configuration file that specifies the cloud providers and resources you want to include in your infrastructure data lake. For example, you might include AWS, Google Cloud, and Azure, along with their respective resource types such as EC2 instances, S3 buckets, and virtual machines. You can find all supported cloud providers and resources in the CloudQuery Hub.

Setting up sources

Example with AWS, Azure and MongoDB

  • Go to the CloudQuery integrations page for AWS to MongoDB and follow it to create aws.yaml and mongodb.yaml files.
  • Go to the CloudQuery integrations page for Azure to MongoDB and follow it to create azure.yaml file. mongodb.yaml file should have the same content as the previous one.

Syncing Cloud Infrastructure Data

With CloudQuery and MongoDB configured, you can now begin fetching your cloud infrastructure data. Run the following command to start the process:

cloudquery sync aws.yaml azure.yaml mongodb.yaml

CloudQuery will now fetch the specified resources from your cloud providers and transfer the data to MongoDB. This process can take some time, depending on the number of resources you are fetching and the size of your cloud infrastructure.

Querying and Analyzing Cloud Infrastructure Data

Now that your cloud infrastructure data is stored in MongoDB, you can use the MongoDB query language (MQL) to analyze and visualize it. Here are some ideas for how you might use MongoDB to analyze your cloud infrastructure data:

  • Calculate the total cost of your EC2 instances or identify unused resources that could be terminated to save on costs
  • Use the MongoDB aggregation framework to group and summarize data
  • Use the MongoDB Compass GUI to visualize your data in a more intuitive manner

Example Queries

Here are some example MongoDB queries that you could use to analyze CloudQuery infrastructure data for AWS. Please note that these queries assume that you have already synced your AWS data using CloudQuery into MongoDB.

Count the total number of EC2 instances:

db.aws_ec2_instances.aggregate([
{ $group: { _id: null, count: { $sum: 1 } } }
])

Find EC2 instances with a specific instance type (e.g., "t3.medium"):

db.aws_ec2_instances.find({ instance_type: "t3.medium" })

Find security groups that allow unrestricted access to a specific port (e.g., port 22 for SSH):

db.aws_ec2_security_groups.find({
"ip_permissions": {
$elemMatch: {
FromPort: 22,
ToPort: 22,
IpRanges: {
$elemMatch: {
CidrIp: "0.0.0.0/0"
}
}
}
}
})

These are just a few examples of the types of queries you can perform using MongoDB to analyze your cloud infrastructure data from AWS. Depending on your specific needs and the resources you have synced, you can create custom queries to gain insights into your cloud infrastructure.

Conclusion

By combining the power of CloudQuery and MongoDB, developers can build a robust cloud infrastructure data lake that simplifies the management and monitoring of cloud resources. With this approach, organizations can gain valuable insights into their cloud infrastructure, optimize resource usage, and ultimately reduce costs.

In this short tutorial we just showed how to get your cloud infrastructure data from any supported CloudQuery source into MongoDB. The number of use cases around MongoDB is really infinite and it all depends on what you are trying to achieve and optimize for. CloudQuery with MongoDB is a powerful tool for analysis which ensures cheap storage and fast querying on large amount of data.

We hope you enjoyed this tutorial and found it useful. If you have any questions or feedback, please reach out to us on our Community or Twitter.

Want to see CloudQuery in action? Schedule a demo with our team or check out the platform documentation to learn more.

Want help getting started? Join the CloudQuery community to connect with other users and experts, or message our team directly here if you have any questions.

Try CloudQuery yourself

Self-hosted, open source, and free to start. Or schedule a demo to see the managed version.

Related posts

Tutorials

How to Automatically Update CloudQuery Integrations with Renovate

Configure Renovate to automatically update CloudQuery integration versions in your YAML files using the CloudQuery Hub API as a custom datasource.

Matthew M

Matthew M

May 20, 2026

AWS

Security

Tutorials

Finding AWS Unrestricted Outbound Access in 2026: EC2, Lambda, and Beyond

Find EC2, Lambda, and ECS Fargate resources with unrestricted outbound access in AWS using SQL. Why egress controls still break attack chains in 2026.

Joe Karlsson

Joe Karlsson

May 5, 2026

AWS

Security

Tutorials

AWS IAM Identity Center with Google Workspace as an IdP: 2026 Step-by-Step Guide

Set up AWS IAM Identity Center with Google Workspace: SAML authentication, SCIM user provisioning, and group sync with CLI or ssosync. Updated for 2026.

Joe Karlsson

Joe Karlsson

May 5, 2026

GCP

Security

Tutorials

Creating Cross-Project Service Accounts in GCP (Console + gcloud CLI)

Create a GCP service account that reads across all your projects with one credential. Covers gcloud CLI, org-level vs.

Joe Karlsson

Joe Karlsson

May 5, 2026

AWS

Cloud Infrastructure

Security

Tutorials

Integrate AbuseIPDB Threat Intelligence with CloudQuery

Enrich your cloud asset inventory with AbuseIPDB threat intelligence to identify public cloud IPs with known abuse reports.

Joe Karlsson

Joe Karlsson

April 20, 2026

AWS

Cloud Infrastructure

Tutorials

How to Query ipinfo.io Data with CloudQuery

Learn how to enrich your cloud asset inventory with ipinfo.io IP geolocation and threat intelligence data using CloudQuery.

Joe Karlsson

Joe Karlsson

April 20, 2026

AWS

Security

Tutorials

AWS IAM Identity Center (Formerly AWS SSO): The Complete 2026 Guide

AWS IAM Identity Center replaced AWS SSO in 2022. Learn how to set it up, configure permission sets, use the AWS CLI, and audit access with CloudQuery.

Joe Karlsson

Joe Karlsson

April 6, 2026

AWS

Tutorials

AWS Pinpoint Is Being Deprecated: How to Assess Your Migration Scope

AWS Pinpoint reaches end of support October 2026. Query your Pinpoint usage across all accounts to find what needs migrating before the deadline.

Joe Karlsson

Joe Karlsson

March 24, 2026

Tutorials

Moving Data from PostgreSQL to MotherDuck

How to sync PostgreSQL data to MotherDuck using CloudQuery, with both one-off batch and continuous CDC streaming options for serverless analytics.

Herman Schaaf

Herman Schaaf

March 13, 2026

Own your pipeline.
Own your data.