product
CloudQuery Product Updates #13
Michal Brutvan •
This has been another exciting month at CloudQuery. There are new sources, metrics and destinations for you to try as well as a brand new feature for our cloud product.
New Beta Feature: CloudQuery Cloud
You can now automate syncs from our cloud using either the command-line or UI without deploying your own infrastructure. This feature, CloudQuery Cloud, is now in Open Beta. You can try it for free for the duration of the beta testing. Read how to get started in our feature announcement.
Pricing Changes
On March 1, we announced pricing changes for CloudQuery plugins. Read more about this in our announcement. To support those changes and the overall development of the product, some tables in various sources are now premium. This affects primarily the AWS, GCP and Azure plugins - starting with the latest major version of each plugin, the number of free tables is lower.
New Sources
First, we have released some brand-new plugins enabling you to grab data from even more places than before. New sources include:
- BigQuery with support for custom queries and incremental syncs
- S3 Source - read parquet files from S3
Source Plugin Updates
AWS
A new major version has been released. This month, we have extended support for metrics from DynamoDB, CloudWatch, network rules (Route53) and AWS GuardDuty. This enables CloudQuery users to add more data to their Cloud Security Posture Management system (CSPM) without having to write code or transform data.
Read about how AWS Config pricing works and how you can get the same value 30 times cheaper with CloudQuery in Understanding AWS Config Cost.
New Tables
aws_dynamodb_table_resource_policiesaws_dynamodb_table_stream_resource_policies
Other changes
- Added support AWS AppConfig Profile tags
- Added support for AWS CloudWatch Metric Stream tags
- Added support for Route53 Resolver Rules and Endpoint tags
- Added support for resolving tags for AWS RDS DBSubnetGroup and OptionGroup
- Add
table_optionsupport foraws_guardduty_detectorsandaws_guardduty_detector_findings - Add incremental table support for
aws_guardduty_detector_findings
GCP
A new major version has been released. This month, we have added support for GCP Identity Access Management (IAM) tables as well as the global address tables
New tables
gcp_iam_organizational_rolesgcp_iam_predefined_rolesgcp_compute_global_addresses
Oracle
New environment variable is available for configuration:
private_key. This is an alternative to providing a path to the file with the private key and makes it easier to run Oracle syncs in production and using CloudQuery Cloud.Destinations
- CrateDB can be now used with PostgreSQL destination plugin.
- PostgreSQL breaking change: The plugin now supports decimal types for column. Previously, decimal types were stored as string.
- S3 Destination now supports
SYNC_IDas a path variable.
SDK Improvements
Sync ID is now passed to plugins during their Init phase. This means it is available for both source and destination plugins and can be used for logging, or configuring the destination parameters.
Other changes
New Docker image with CloudQuery CLI based on Ubuntu makes it possible to run syncs with DuckDB plugin in a containerized environment.
In case you missed it…
In Pulling Stock Market Data with CloudQuery, Tim goes deep into SDK and walks you through the process of creating a new CloudQuery plugin to pull the delayed market data from London Stock Exchange
Yevgeny explains how AWS Config pricing works and how you can get the same value 30 times cheaper with CloudQuery in Understanding AWS Config Cost.
We have launched the CloudQuery Cloud in beta last month so you can now build a CloudQuery powered CSPM completely without deploying any infrastructure. Read how in Building a serverless open source CSPM powered by CloudQuery.
Yevgeny walks you through counting words in post titles on HackerNews: Building Local and Serverless ELT pipelines with CloudQuery and MotherDuck
Feedback
We’re looking for feedback to help us improve our CloudQuery Cloud. Participate in user research and get a 50 USD Amazon gift card! Book a research call here.