Tutorials

Add observability to your CloudQuery sync runs with Datadog

In this guide we will discuss how to leverage Datadog to add observability to CloudQuery sync jobs.
A common use case for CloudQuery is to run it as a scheduled job to sync cloud resources to data warehouse, or even multiple jobs each with a different configuration.
In order to get insights into runs of sync jobs, we can use Datadog to monitor the sync jobs and get alerts when something goes wrong.

Prerequisites

  • A Datadog account. See here for getting started
  • The Datadog agent installed in the machine(s) running the CloudQuery sync jobs. See here for more information
  • Log collection enabled in the Datadog agent configuration. See here for more information

Adding a Custom Log Source for CloudQuery Logs

First we recommended passing the --log-format json flag to the CloudQuery sync command. This ensures that the logs are in a structured format that Datadog can parse.
Datadog can collect logs in non JSON format as well, but using JSON format ensure that we can query the logs.

Now, let's create the custom log source in Datadog:

  1. Locate the Datadog conf.d directory. You can find the location of the directory based on our OS in the Datadog documentation here.
  2. Under the conf.d directory create a directory named cloudquery.d
  3. Under the cloudquery.d directory create a file named conf.yaml with the following content:
logs:
- type: 'file'
path: '' # Example path: /var/log/cloudquery/cloudquery.log
service: 'CloudQuery'
source: 'CloudQuery'
  1. Restart the Datadog agent for changes to take effect.
You can find more information on setting up a custom log source in Datadog here

Using Datadog to query CloudQuery logs

Datadog Logs Explorer showing CloudQuery sync log entries with timestamps and log levels

Once you run a CloudQuery sync job, you should see the logs in the Datadog logs explorer:

Datadog logs explorer time range

If you don't see any logs you might need to chose a different time range in the Datadog logs explorer:

By using JSON format we can immediately query the logs in Datadog.

Datadog Logs Explorer filtered to show CloudQuery tables with sync errors using the errors query

For example the @errors:>0 @table:* query filters all tables that had errors during the sync job:

Datadog Logs Explorer filtered to show CloudQuery tables with more than 10,000 synced resources

And the @resources:>10000 @table:* query filters all tables that had more than 10,000 resources during the sync job:

Summary and Next Steps

In this guide we created a custom log source in Datadog to collect CloudQuery logs and query them.

Datadog is a powerful tool for monitoring and observability and we only scratched the surface of what you can do with it.
We recommend reading the Datadog documentation to learn more about its capabilities, for example creating monitors and dashboards.

Want to see CloudQuery in action? Schedule a demo with our team or check out the platform documentation to learn more.

Want help getting started? Join the CloudQuery community to connect with other users and experts, or message our team directly here if you have any questions.

Try CloudQuery yourself

Self-hosted, open source, and free to start. Or schedule a demo to see the managed version.

Related posts

Tutorials

How to Automatically Update CloudQuery Integrations with Renovate

Configure Renovate to automatically update CloudQuery integration versions in your YAML files using the CloudQuery Hub API as a custom datasource.

Matthew M

Matthew M

May 20, 2026

AWS

Security

Tutorials

Finding AWS Unrestricted Outbound Access in 2026: EC2, Lambda, and Beyond

Find EC2, Lambda, and ECS Fargate resources with unrestricted outbound access in AWS using SQL. Why egress controls still break attack chains in 2026.

Joe Karlsson

Joe Karlsson

May 5, 2026

AWS

Security

Tutorials

AWS IAM Identity Center with Google Workspace as an IdP: 2026 Step-by-Step Guide

Set up AWS IAM Identity Center with Google Workspace: SAML authentication, SCIM user provisioning, and group sync with CLI or ssosync. Updated for 2026.

Joe Karlsson

Joe Karlsson

May 5, 2026

GCP

Security

Tutorials

Creating Cross-Project Service Accounts in GCP (Console + gcloud CLI)

Create a GCP service account that reads across all your projects with one credential. Covers gcloud CLI, org-level vs.

Joe Karlsson

Joe Karlsson

May 5, 2026

AWS

Cloud Infrastructure

Security

Tutorials

Integrate AbuseIPDB Threat Intelligence with CloudQuery

Enrich your cloud asset inventory with AbuseIPDB threat intelligence to identify public cloud IPs with known abuse reports.

Joe Karlsson

Joe Karlsson

April 20, 2026

AWS

Cloud Infrastructure

Tutorials

How to Query ipinfo.io Data with CloudQuery

Learn how to enrich your cloud asset inventory with ipinfo.io IP geolocation and threat intelligence data using CloudQuery.

Joe Karlsson

Joe Karlsson

April 20, 2026

AWS

Security

Tutorials

AWS IAM Identity Center (Formerly AWS SSO): The Complete 2026 Guide

AWS IAM Identity Center replaced AWS SSO in 2022. Learn how to set it up, configure permission sets, use the AWS CLI, and audit access with CloudQuery.

Joe Karlsson

Joe Karlsson

April 6, 2026

AWS

Tutorials

AWS Pinpoint Is Being Deprecated: How to Assess Your Migration Scope

AWS Pinpoint reaches end of support October 2026. Query your Pinpoint usage across all accounts to find what needs migrating before the deadline.

Joe Karlsson

Joe Karlsson

March 24, 2026

Tutorials

Moving Data from PostgreSQL to MotherDuck

How to sync PostgreSQL data to MotherDuck using CloudQuery, with both one-off batch and continuous CDC streaming options for serverless analytics.

Herman Schaaf

Herman Schaaf

March 13, 2026

Own your pipeline.
Own your data.