CloudQuery Product Updates #13

This has been another exciting month at CloudQuery. There are new sources, metrics and destinations for you to try as well as a brand new feature for our cloud product.

New Beta Feature: CloudQuery Platform

You can now automate syncs from our cloud using either the command-line or UI without deploying your own infrastructure. This feature, CloudQuery Platform, is now in Open Beta. You can try it for the duration of the beta testing. Read how to get started in our feature announcement.

Pricing Changes

On March 1, we announced pricing changes for CloudQuery plugins. Read more about this in our announcement. To support those changes and the overall development of the product, some tables in various sources are now premium. This affects primarily the AWS, GCP and Azure plugins - starting with the latest major version of each plugin, the number of free tables is lower.

New Sources

First, we have released some brand-new plugins enabling you to grab data from even more places than before. New sources include:

• Apigee
• BigQuery with support for custom queries and incremental syncs
• ServiceNow
• S3 Source - read parquet files from S3
• Snowflake Config

Source Plugin Updates

AWS

A new major version has been released. This month, we have extended support for metrics from DynamoDB, CloudWatch, network rules (Route53) and AWS GuardDuty. This enables CloudQuery users to add more data to their Cloud Security Posture Management system (CSPM) without having to write code or transform data.

Read about how AWS Config pricing works and how you can get the same value 30 times cheaper with CloudQuery in Understanding AWS Config Cost.

New Tables

• aws_dynamodb_table_resource_policies
• aws_dynamodb_table_stream_resource_policies

Other changes

• Added support AWS AppConfig Profile tags
• Added support for AWS CloudWatch Metric Stream tags
• Added support for Route53 Resolver Rules and Endpoint tags
• Added support for resolving tags for AWS RDS DBSubnetGroup and OptionGroup
• Add table_option support for aws_guardduty_detectors and aws_guardduty_detector_findings
• Add incremental table support for aws_guardduty_detector_findings

GCP

A new major version has been released. This month, we have added support for GCP Identity Access Management (IAM) tables as well as the global address tables

New tables

• gcp_iam_organizational_roles
• gcp_iam_predefined_roles
• gcp_compute_global_addresses

Oracle

New environment variable is available for configuration: private_key. This is an alternative to providing a path to the file with the private key and makes it easier to run Oracle syncs in production and using the CloudQuery Platform.

Destinations

• CrateDB can be now used with PostgreSQL destination plugin.
• PostgreSQL breaking change: The plugin now supports decimal types for column. Previously, decimal types were stored as string.
• S3 Destination now supports SYNC_ID as a path variable.

SDK Improvements

Sync ID is now passed to plugins during their Init phase. This means it is available for both source and destination plugins and can be used for logging, or configuring the destination parameters.

Other changes

New Docker image with CloudQuery CLI based on Ubuntu makes it possible to run syncs with DuckDB plugin in a containerized environment.

In case you missed it…

In Pulling Stock Market Data with CloudQuery, Tim goes deep into SDK and walks you through the process of creating a new CloudQuery plugin to pull the delayed market data from London Stock Exchange

Yevgeny explains how AWS Config pricing works and how you can get the same value 30 times cheaper with CloudQuery in Understanding AWS Config Cost.

We have launched the CloudQuery Platform in beta last month so you can now build a CloudQuery powered CSPM completely without deploying any infrastructure. Read how in Building a serverless open source CSPM powered by CloudQuery.

Yevgeny walks you through counting words in post titles on HackerNews: Building Local and serverless ELT Pipelines with CloudQuery and MotherDuck

Want to see CloudQuery in action? Schedule a demo with our team or check out the platform documentation to learn more.

Got feedback or suggestions? Join the CloudQuery community to connect with other users and experts, or message our team directly here if you have any questions.