How to Manage Resilience Strategy and Visibility with AWS Backup and CloudQuery
- Manage AWS Backup at scale across multiple AWS accounts.
- Monitor status of AWS Backup and compliance with company strategies or requirements.
- Monitor overall health of AWS Backup including failures, issues, and successes.
- Manage AWS Backup settings and reduce data protection issues.
Walkthrough #
- AWS (Amazon Web Services) for infrastructure source
- CloudQuery for ETL
- PostgreSQL as the destination
- Grafana for a visualization layer
Prerequisites #
- AWS Account(s) with AWS Backup configured and running.
Step 1: Install or Deploy CloudQuery #
Step 2: Sync Data from AWS to PostgreSQL #
kind: source
spec:
# Source spec section
name: aws
path: cloudquery/aws
registry: cloudquery
version: "v28.8.0"
tables:
- aws_backup_*
- aws_s3_buckets
- aws_dynamodb_*
- aws_ec2_instances
destinations: ["postgresql"]
spec:
# AWS Spec section described below
regions:
- us-east-1
accounts:
- id: "account1"
local_profile: "account1-profile"
Step 3: Set up Grafana for Visualization #
- Self-hosted Grafana Official Guide
- SaaS/Managed Grafana: Grafana
- AWS Fully-Managed Grafana: Amazon Managed Grafana
Step 4: Run Queries #
SELECT *
FROM aws_backup_jobs
WHERE state='FAILED';
SELECT *
FROM aws_backup_protected_resources;
SELECT resource_type, COUNT(arn), account_id
FROM aws_backup_protected_resources
GROUP BY resource_type, account_id;
- Searching by tags if a tag-based backup strategy is being used.
- Querying the AWS Backup settings in each AWS account.
- Validating success of selected resources for AWS Backup.
Step 5: Create Custom Visualizations and Dashboards #
- Backup Compliance Numbers by resources for DynamoDB, EC2, and S3.
- Protected Resources by AWS Backup
- Protected Resources Summarized by Type
- Backup Job Health by Status showing Failed Jobs, Successful Jobs, and Jobs with Issues.
Summary #
Written by Jason Kao
Jason worked as Head of Security Research and Solutions at CloudQuery and was a Senior Data Engineer prior to taking on that role. He focused on multi-cloud environments and has particular expertise on AWS.