CloudQuery Product Updates #15

This is another issue of CloudQuery’s monthly newsletter featuring cloud security and data engineering tips, tools, tutorials, and a changelog of all the new updates rolled out this month in CloudQuery.

How To Run Efficient Embedded ELT CloudQuery Workflows Inside Dagster Learn to run CloudQuery, an open-source high-performance ELT framework, with Dagster orchestrator. You’ll leverage Dagster’s resource management and scheduling capabilities to run CloudQuery locally, avoiding additional cloud costs. This is possible due to CloudQuery’s design, which allows it to run as single binaries without third-party dependencies.

How to Build a CSPM with Grafana and CloudQuery Learn to build an extensible Cloud Security Posture Management (CSPM) solution using Grafana and CloudQuery. This tutorial guides you through setting up a local development environment with Docker Compose, integrating CloudQuery PostgreSQL DBT, and Grafana. You’ll create customizable dashboards for security compliance and monitoring, gaining practical experience in enhancing cloud security for both local and production environments.

Unmasking Adversary Cloud Defense Evasion Strategies: Modify Cloud Compute Infrastructure Part 2 To prevent abuse against the use of Snapshot Creation, Instance Creation, and Instance Deletion features within cloud environments, security teams and cyber-defenders must ensure that proper monitoring and logging services are enabled across all cloud providers they utilize.

Tactical Cloud Audit Log Analysis with DuckDB Using DuckDB to query Cloud Provider audit logs when you don't have a SIEM available.

Container security fundamentals part 6: seccomp A look at how seccomp is used in Linux and container systems.

AWS has new tables with reserved instances: ElasticSearch, MemoryDB, OpenSearch, RedShift. We have also added tables for AWS Health and RDS.

Azure has got tables with storage account access keys and key rotation policies.

GCP now supports tables with data from the Security Center.

Finally, new tables were added for secret and code-scanning alerts to our GitHub plugin.

These are just major highlights. To see all the changes described in detail, see the individual plugin changelogs in Hub.

We have added support for the overwrite-delete-stale write mode in the BigQuery plugin.

All destinations now support “Sync summary”. After syncing, a table named cloudquery_sync_summaries is created and includes the number of resources synced, the number of errors, and details about the plugins (both source and destination).

We have published the following plugins in preview:

Orca: Sync data about your assets, alerts, attack paths, or CVEs related to your infrastructure Wiz: Sync issues, vulnerability, and cloud infrastructure findings from Wiz. tempo.io: Sync data about time spent working on issues in JIRA (using Tempo.io app). BambooHR: Sync employees and time off requests, and connect them with data from other plugins to find owners of your resources, or employees on call that should handle an incident.

Note: The preview means the plugins are free to use but their schema may change in the next major version before they officially reach the “Generally Available” stage.

If you haven’t signed up for our product yet, now’s the perfect time! Get started with CloudQuery to harness the power of high-performance ELT for your cloud infrastructure.

Have questions, need support, or want to connect with other CloudQuery developers? Join our vibrant community on Discord. Our team is ready to help you succeed!

Interested in trying out CloudQuery Cloud? Reach out to us for more information!