AWS to PostgreSQL

Prerequisites

Before starting this guide, make sure you have:

CloudQuery CLI installed — if you haven’t already, complete the quickstart guide for your platform
AWS credentials configured — CloudQuery uses the AWS SDK default credential chain. Configure credentials using the AWS CLI (aws configure) or by setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables
Docker — used in this guide to run a local PostgreSQL instance

Verify your AWS credentials are working before continuing:


aws sts get-caller-identity

You should see your account ID, user ID, and ARN. If you get an error, resolve your credentials before proceeding.

CloudQuery only requires read access to your AWS account. Follow the principle of least privilege and grant only the permissions needed for the tables you want to sync.

Start a local PostgreSQL database

Start a PostgreSQL instance using Docker:


docker run -d --name cloudquery-postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=cloudquery -p 5432:5432 postgres:15

If port 5432 is already in use (for example, by a local PostgreSQL installation), this command will fail. Either stop the existing service, or change the port mapping to something like -p 5433:5432 and update the connection string port accordingly.

If you’ve run this guide before and the container already exists, remove it first:


docker rm -f cloudquery-postgres

Verify the container is ready before continuing:


docker exec cloudquery-postgres pg_isready -U postgres

You should see: /var/run/postgresql:5432 - accepting connections

Create a sync configuration

This guide syncs aws_s3_buckets — a good starting table because it’s account-wide (not region-scoped), fast to sync, and requires only basic S3 read permissions. Once you have the pattern working, you can add more tables.

Generate a sync configuration file:


cloudquery init --source aws --destination postgresql --spec-path aws_to_postgresql.yaml

This creates aws_to_postgresql.yaml. Open the file to review it — you can add more tables later by editing the tables list.

Run the init command without arguments to pick from all available sources and destinations interactively. When logged in via cloudquery login, init launches an AI-assisted mode to help generate configurations based on your requirements.

Set the connection string

The generated configuration reads the PostgreSQL connection string from the POSTGRESQL_CONNECTION_STRING environment variable. Set it now:


export POSTGRESQL_CONNECTION_STRING="postgresql://postgres:postgres@localhost:5432/cloudquery?sslmode=disable"

The connection string above uses sslmode=disable, which is appropriate for local development. For production databases, use sslmode=require or sslmode=verify-full and provide the appropriate certificates.

Start syncing

Start the sync:


cloudquery sync aws_to_postgresql.yaml

You should see a progress spinner. A typical S3 sync completes in a few seconds and shows a summary like:


Sync completed successfully. Resources: 6, Errors: 0, Warnings: 0, Time: 4s

If your AWS account has no S3 buckets, the sync will complete successfully with Resources: 0. The setup is working correctly — try adding another table like aws_iam_users to the tables list in aws_to_postgresql.yaml and re-running the sync.

Examine the data

Query the synced data directly in PostgreSQL:


docker exec cloudquery-postgres psql -U postgres -d cloudquery -c "SELECT name, region, creation_date FROM aws_s3_buckets ORDER BY creation_date DESC LIMIT 5;"

Windows users: Run this command in PowerShell. Windows CMD has quote-handling limitations with the -c flag that can cause errors.

The output should look like this:


              name               |  region   |    creation_date
---------------------------------+-----------+---------------------
 my-app-assets                   | us-east-1 | 2025-12-10 14:22:01
 my-data-archive                 | us-west-2 | 2025-08-04 09:15:33
 my-backups-prod                 | us-east-1 | 2025-06-20 11:45:12
 cf-templates-abc123-us-east-1   | us-east-1 | 2025-02-11 16:03:44
 my-logs-bucket                  | eu-west-1 | 2024-11-30 08:12:05
(5 rows)

Next steps

Now that you have AWS data syncing into PostgreSQL:

Add more AWS tables by editing the tables list in aws_to_postgresql.yaml. Common additions include aws_ec2_instances, aws_iam_users, and aws_lambda_functions. See the full AWS table reference.
Connect psql, DBeaver, or any PostgreSQL-compatible BI tool to explore and visualize your data.
Deploy CloudQuery on a schedule when you’re ready to move to production.
Browse source integrations and destination integrations to sync from other sources or write to other destinations.

Clean up

To stop and remove the local PostgreSQL container when you’re done:


docker stop cloudquery-postgres && docker rm cloudquery-postgres