This syncs data from a CloudQuery source to a PostgreSQL-compatible database. Supports RDS, Prisma, Aiven, Neon, Supabase and many others. It also comes with a built-in support for pgvector extension that allows you to store, query, and index vectors.
Loading plugin documentation
CloudQuery's use of cookies
We use tracking cookies to understand how you use the product and help us improve it. Your consent is required before we can enable these cookies. You can opt out via the link in the footer.
POSTGRESQL_CONNECTION_STRING:kind: destination
spec:
name: "postgresql"
path: "cloudquery/postgresql"
registry: "cloudquery"
version: "v8.16.0"
write_mode: "overwrite-delete-stale"
send_sync_summary: true
# Learn more about the configuration options at https://cql.ink/postgresql_destination
spec:
# set the environment variable in DSN format like "user=postgres password=pass+0-[word host=localhost port=5432 dbname=postgres sslmode=disable"
# you can also specify it in URI format like "postgres://postgres:pass@localhost:5432/postgres?sslmode=disable". any special URI characters need to be percent-encoded
connection_string: "${POSTGRESQL_CONNECTION_STRING}"
# Optional parameters:
# pgx_log_level: error
# batch_size: 10000 # 10K entries
# batch_size_bytes: 100000000 # 100 MB
# batch_timeout: 60s
# create_performance_indexes: false #create indexes that help with performance when using `write_mode: overwrite-delete-stale`
batch_size and batch_size_bytes.connection_string (string) (required)pgxpool"user=user password=pass+0-[word host=localhost port=5432 dbname=mydb sslmode=disable" DSN format"postgres://user:pass@localhost:5432/mydb?sslmode=prefer" connect with tcp and prefer TLS"postgres://user:pass@localhost:5432/mydb?sslmode=disable&application_name=pgxtest&search_path=myschema&connect_timeout=5" be explicit with all options"postgres://localhost:5432/mydb?sslmode=disable" connect with os username cloudquery is being run as"postgres:///mydb?host=/tmp" connect over unix socket"dbname=mydb" unix domain socket, just specifying the db name - useful if you want to use peer authenticationpgx_log_level (string) (optional) (default: error)batch_size (integer) (optional) (default: 10000)batch_size_bytes (integer) (optional) (default: 100000000 (= 100 MB))batch_timeout (duration) (optional) (default: 60s (= 60 seconds))create_performance_indexes (boolean) (optional) (default: false)write_mode: overwrite-delete-stale.lakebase (object) (optional)connection_string still supplies the host, port, database name and user (the service principal client ID), and must use sslmode=require (or verify-ca/verify-full); TLS is required and enforced. See the Databricks Lakebase example below.endpoint (string) (required)projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id}.host (string) (optional)https://your-workspace.cloud.databricks.com. If empty, the Databricks SDK resolves it from the DATABRICKS_HOST environment variable (or other default Databricks configuration sources).client_id (string) (optional)DATABRICKS_CLIENT_ID environment variable (or other default Databricks configuration sources).client_secret (string) (optional)DATABRICKS_CLIENT_SECRET environment variable (or other default Databricks configuration sources).aws_iam_auth (object) (optional)connection_string still supplies the host, port, database name and user (the database user that has been granted IAM authentication), and must use TLS, which is enforced. Mutually exclusive with lakebase. See the Amazon RDS IAM authentication example below.service (string) (optional) (default: rds)rds (Amazon RDS and Aurora PostgreSQL).region (string) (optional)us-east-1. If empty, the region is resolved from the standard AWS configuration sources (the AWS_REGION environment variable, the shared config file, ...).endpoint (string) (optional)host or host:port. If empty, the host and port from connection_string are used; if only a host is given, the port from connection_string is used. Set this when the plugin connects through a different address than the database endpoint itself, such as an SSH tunnel or a CNAME.local_profile (string) (optional)role_arn (string) (optional)role_session_name (string) (optional)role_arn.external_id (string) (optional)role_arn.pgvector_config (object) (optional)_cq_id column on target tables if this is enabled.tables (array) (required)source_table_name (string) (required)target_table_name (string) (required)embedding vector column, a chunk text column, and the configured metadata columns. The _cq_id column is always included and indexed.embed_columns (array) (required)metadata_columns (array) (optional)_cq_id column will be added automatically and an index will be created on it.text_splitter (object) (optional)recursive_text (object) (required)chunk_size (integer) (required)chunk_overlap (integer) (required)openai_embedding (object) (required)dimensions (integer) (required)text-embedding-3-small, this is 1536. For text-embedding-3-large, this is 3072.api_key (string) (required)model_name (string) (required)text-embedding-3-small and text-embedding-3-large are supported.retry_on_deadlock (integer) (optional) (default: 0)40P01).lakebase block. The plugin authenticates with a Databricks service principal (OAuth M2M) and mints a fresh short-lived database credential for every new connection, so you do not put a password in the connection_string. The connection string supplies the Lakebase host, port, database, user (the service principal client ID) and must use sslmode=require.kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.16.0"
write_mode: "overwrite-delete-stale"
send_sync_summary: true
spec:
# No password in the connection string - it is generated per-connection from the Databricks credentials below.
connection_string: "host=${PGHOST} port=5432 dbname=databricks_postgres user=${DATABRICKS_CLIENT_ID} sslmode=require"
lakebase:
# projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id}
endpoint: "${LAKEBASE_ENDPOINT_NAME}"
host: "${DATABRICKS_HOST}"
client_id: "${DATABRICKS_CLIENT_ID}"
client_secret: "${DATABRICKS_CLIENT_SECRET}"
host, client_id and client_secret are optional. If omitted, the Databricks SDK resolves them from the standard DATABRICKS_HOST, DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET environment variables (or other default Databricks configuration sources).aws_iam_auth block and the plugin signs a fresh short-lived token for every new connection, so you do not put a password in the connection_string.rds_iam role:CREATE USER cloudquery;
GRANT rds_iam TO cloudquery;
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "rds-db:connect",
"Resource": "arn:aws:rds-db:us-east-1:123456789012:dbuser:db-ABCDEFGHIJKL01234/cloudquery"
}
]
}
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.16.0"
write_mode: "overwrite-delete-stale"
send_sync_summary: true
spec:
# No password in the connection string - a token is generated per-connection from the AWS credentials below.
connection_string: "host=${PGHOST} port=5432 dbname=postgres user=cloudquery sslmode=require"
aws_iam_auth:
service: "rds"
region: "us-east-1"
service defaults to rds. AWS credentials are resolved from the standard AWS sources (environment variables, the shared credentials file, an EC2 instance profile, an EKS service account, ...), and the region is resolved from AWS_REGION when region is not set. Set local_profile to pick a specific shared config profile, or role_arn (optionally with role_session_name and external_id) to assume a role before signing the token.pgx built-in logging and might output data/sensitive information to logs so make sure to not use it in production but only for debugging.kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.16.0"
send_sync_summary: true
spec:
connection_string: ${PG_CONNECTION_STRING}
pgx_log_level: debug # Available: error, warn, info, debug, trace. Default: "error"
connection_string field in the sync config file.postgres://postgres:pass@localhost:5432/postgres?sslmode=disable. Any special URI characters need to be percent-encoded."user=postgres password=pass+0-[word host=localhost port=5432 dbname=postgres sslmode=disable"\x00) are automatically stripped from string values for PostgreSQL compatibilityjsonb with null characters stripped from string valuestimestamp without time zone in UTC