Announcing Closed Beta of CloudQuery AWS Plugin with Event-based Sync

Michal Brutvan • Sep 06, 2023
What is it?
How it works

Service | Event |
---|---|
ec2.amazonaws.com | AssociateRouteTable |
ec2.amazonaws.com | AttachInternetGateway |
ec2.amazonaws.com | AuthorizeSecurityGroupEgress |
ec2.amazonaws.com | AuthorizeSecurityGroupIngress |
ec2.amazonaws.com | CreateImage |
ec2.amazonaws.com | CreateInternetGateway |
ec2.amazonaws.com | CreateNetworkInterface |
ec2.amazonaws.com | CreateSecurityGroup |
ec2.amazonaws.com | CreateSubnet |
ec2.amazonaws.com | CreateTags |
ec2.amazonaws.com | CreateVpc |
ec2.amazonaws.com | DeleteTags |
ec2.amazonaws.com | DetachInternetGateway |
ec2.amazonaws.com | ModifySubnetAttribute |
ec2.amazonaws.com | RevokeSecurityGroupEgress |
ec2.amazonaws.com | RevokeSecurityGroupIngress |
ec2.amazonaws.com | RunInstances |
iam.amazonaws.com | CreateGroup |
iam.amazonaws.com | CreateRole |
iam.amazonaws.com | CreateUser |
rds.amazonaws.com | CreateDBCluster |
rds.amazonaws.com | CreateDBInstance |
rds.amazonaws.com | ModifyDBCluster |
rds.amazonaws.com | ModifyDBInstance |
Getting Started
- Configure an AWS CloudTrail Trail to send management events to a Kinesis Data Stream via CloudWatch Logs. The most straight forward way to do this is to use the CloudQuery provided CloudFormation template.
aws cloudformation deploy --template-file ./streaming-deployment.yml --stack-name <STACK-NAME> --capabilities CAPABILITY_IAM --disable-rollback --region <DESIRED-REGION>
- Copy the ARN of the Kinesis stream. If you used the CloudFormation template you can run the following command:
aws cloudformation describe-stacks --stack-name <STACK-NAME> --query "Stacks[].Outputs" --region <DESIRED-REGION>
- Define a
config.yml
file like the one below
kind: source spec: name: "aws-event-based" registry: "local" path: <PATH/TO/BINARY> tables: - aws_ec2_instances - aws_ec2_internet_gateways - aws_ec2_security_groups - aws_ec2_subnets - aws_ec2_vpcs - aws_ecs_cluster_tasks - aws_iam_groups - aws_iam_roles - aws_iam_users - aws_rds_instances destinations: ["postgresql"] skip_tables: - aws_iam_group_last_accessed_details - aws_iam_role_last_accessed_details - aws_iam_user_last_accessed_details spec: event_based_sync: - account: local_profile: "<ROLE-NAME>" kinesis_stream_arn: "<OUTPUT-FROM-CLOUDFORMATION-STACK>"
- Sync the data!
cloudquery sync config.yml
Deploying in production
overwrite-delete-stale
write model. To delete stale data, you need to set up a recurrent task to run full table syncs. Additionally, you may need to set up another task with CloudQuery still running regular sync on tables that are currently not supported for the event-based sync. See the AWS Plugin documentation for the list of supported tables.