The CloudQuery PostgreSQL plugin syncs your PostgreSQL database to any of the supported CloudQuery destinations
Publisher
cloudquery
Latest version
v6.13.8
Type
Source
Platforms
Date Published
Loading plugin documentation
CloudQuery's use of cookies
We use tracking cookies to understand how you use the product and help us improve it. Your consent is required before we can enable these cookies. You can opt out via the link in the footer.
cloudquery needs to be authenticated. Credentials can be inserted in the connection_string field.localhost:5432. The (top level) spec section is described in the Source Spec Reference.kind: source
spec:
name: "postgresql"
registry: "cloudquery"
path: "cloudquery/postgresql"
version: "v6.13.8"
tables: ["*"]
destinations: ["postgresql"]
# Learn more about the configuration options at https://cql.ink/postgresql_source
spec:
connection_string: "${POSTGRESQL_CONNECTION_STRING}" # set the environment variable in a format like postgres://postgres:pass@localhost:5432/postgres?sslmode=disable
# Optional parameters:
# cdc_id: "postgresql" # Set to a unique string per source to enable Change Data Capture mode (logical replication, or CDC)
# pgx_log_level: error
# rows_per_record: 500
connection_string (string) (required)pgxpool"postgres://jack:secret@localhost:5432/mydb?sslmode=prefer" connect with tcp and prefer TLS"postgres://jack:secret@localhost:5432/mydb?sslmode=disable&application_name=pgxtest&search_path=myschema&connect_timeout=5" be explicit with all options"postgres://localhost:5432/mydb?sslmode=disable" connect with os username cloudquery is being run as"postgres:///mydb?host=/tmp" connect over unix socket"dbname=mydb" unix domain socket, just specifying the db name - useful if you want to use peer authentication"user=jack password=jack\\'ssooper\\\\secret host=localhost port=5432 dbname=mydb sslmode=disable" DSN with escaped backslash and single quotepgx_log_level (string) (optional) (default: error)cdc_id (string) (optional)rows_per_record (integer) (optional) (default: 500)destination_table_name (string) (optional) (default: {{TABLE}}){{TABLE}} will be replaced with the table name{{UUID}} will be replaced with a random UUID in raw format to uniquely identify each table (*){{YEAR}} will be replaced with the current year in YYYY format (*){{MONTH}} will be replaced with the current month in MM format (*){{DAY}} will be replaced with the current day in DD format (*){{HOUR}} will be replaced with the current hour in HH format (*){{MINUTE}} will be replaced with the current minute in mm format (*){{TABLE}} or {{UUID}} is required. Dynamic placeholders (denoted with *) are not allowed in CDC mode (when cdc_id is set)UTC and will be the time sync started.SHOW wal_level;
SELECT setting FROM pg_settings WHERE name='wal_level'
replication but for CDC to work we need to set it to logical (This can only be done on the database startup).docker run -p 5432:5432 -e POSTGRES_PASSWORD=pass -d postgres:11 -c "wal_level=logical"
wal_level to logical in the postgresql.conf file and restart the database.rds.logical_replication to 1 and restart the database.rds.logical_replication to 1 and restart the database.cloudsql.logical_deconging configuration to on. See full documentation on GCP documentationCREATE TABLE, ALTER TABLE, etc)TRUNCATE statements (e.g. TRUNCATE TABLE table_name)DELETE statements (e.g. DELETE FROM table_name)user:pass in the connection string, the password needs to be longer than 112 bits.kind: source
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v6.13.8-fips"
...