Back to destination list
Official
Snowflake destination integration documentation
The snowflake plugin helps you sync data to your Snowflake data warehouse
Loading plugin documentation
The snowflake plugin helps you sync data to your Snowflake data warehouse
Loading plugin documentation
CloudQuery's use of cookies
We use tracking cookies to understand how you use the product and help us improve it. Your consent is required before we can enable these cookies. You can opt out via the link in the footer.
INSERT statements. All data is bulk-loaded via a stage. For each batch of records, the plugin:cq_plugin_stage, using PUT ... auto_compress=true.COPY INTO, which appends the rows.MERGE INTO, so matching rows are updated in place instead of duplicated.cq_plugin_json_format) are created with CREATE OR REPLACE the first time a sync writes data. Any files still present in cq_plugin_stage are therefore discarded when the next sync begins — the stage is scratch space, not durable storage, and should not be read by anything other than the plugin.SNOWFLAKE_CONNECTION_STRING environment variable:kind: destination
spec:
name: snowflake
path: cloudquery/snowflake
registry: cloudquery
version: "v5.2.14"
write_mode: "append"
send_sync_summary: true
# Learn more about the configuration options at https://cql.ink/snowflake_destination
spec:
connection_string: "${SNOWFLAKE_CONNECTION_STRING}"
# Optional parameters
# migrate_concurrency: 1
# batch_size: 5000 # 5K entries
# batch_size_bytes: 20971520 # 20 MiB
# leave_stage_files: false
batch_size and batch_size_bytes. Each batch becomes one staged file, so larger batches produce fewer and larger files.cq_plugin_stage is recreated on every sync (see How Data is Loaded), you should not point a Snowpipe at it — files can be removed before the pipe has ingested them.kind: destination
spec:
name: "s3"
path: "cloudquery/s3"
registry: "cloudquery"
version: "v7.10.13"
write_mode: "append"
spec:
bucket: "bucket_name"
region: "us-east-1"
path: "cloudquery/{{TABLE}}/{{UUID}}.{{FORMAT}}"
format: "parquet"
# Snowflake recommends files of roughly 100-250 MB compressed
batch_size_bytes: 209715200 # 200 MiB
create file format cq_parquet_format type = parquet;
create storage integration cq_s3_integration
type = external_stage
storage_provider = 's3'
storage_aws_role_arn = 'arn:aws:iam::123456789012:role/snowflake-cloudquery'
enabled = true
storage_allowed_locations = ('s3://bucket_name/cloudquery/');
create stage cq_external_stage
url = 's3://bucket_name/cloudquery/'
storage_integration = cq_s3_integration
file_format = cq_parquet_format;
show pipes) and configuring the bucket to publish to it:create pipe cq_aws_ec2_instances_pipe
auto_ingest = true
as
copy into aws_ec2_instances
from @cq_external_stage/aws_ec2_instances/
match_by_column_name = case_insensitive;
MERGE INTO this destination performs for tables with primary keys, and the object storage destinations write with append write_mode. Every sync adds a new set of rows, so deduplicate downstream — for example with a view that keeps the latest _cq_sync_time per _cq_id, or a scheduled task.batch_size_bytes accordingly rather than leaving it at its default.auto_ingest on internal stages is restricted. It is only available for Snowflake accounts hosted on AWS, which is another reason to use an external stage. Alternatives are the Snowpipe REST API or a scheduled alter pipe ... refresh.kind: destination
spec:
name: snowflake
send_sync_summary: true
...
spec:
connection_string: "user:pass@account/db/schema?warehouse=wh"
kind: destination
spec:
name: snowflake
send_sync_summary: true
...
spec:
connection_string: "user@account/database/schema?warehouse=wh"
private_key: |
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC2ajPRIbPtbxZ1
3DONLA02eZJuCzsgIkBWov/Me5TL6cKN0gnY+mbA8OnNCH+9HSzgiU9P8XhTUrIN
85diD+rj6uK+E0sSyxGk6HG17TyR5oBq8nz2hbZlbaNi/HO9qYoHQgAgMq908YBz
...
DUmOIrBYEMf2nDTlTu/QVcKb
-----END PRIVATE KEY-----
kind: destination
spec:
name: snowflake
send_sync_summary: true
...
spec:
connection_string: "user@account/database/schema?warehouse=wh"
private_key: "${file:./private.key}"
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC2ajPRIbPtbxZ1
3DONLA02eZJuCzsgIkBWov/Me5TL6cKN0gnY+mbA8OnNCH+9HSzgiU9P8XhTUrIN
85diD+rj6uK+E0sSyxGk6HG17TyR5oBq8nz2hbZlbaNi/HO9qYoHQgAgMq908YBz
...
DUmOIrBYEMf2nDTlTu/QVcKb
-----END PRIVATE KEY-----
kind: destination
spec:
name: snowflake
send_sync_summary: true
...
spec:
connection_string: "user:pass@account/db/schema?warehouse=wh&authenticator=oauth&token=token"
openssl pkcs8 -topk8 -nocrypt -in enc.key -out dec.key
connection_string (string) (required)connection_string.# user[:password]@account/database/schema?warehouse=user_warehouse[¶m1=value1¶mN=valueN]
# or
# user[:password]@account/database?warehouse=user_warehouse[¶m1=value1¶mN=valueN]
# or
# user[:password]@host:port/database/schema?account=user_account&warehouse=user_warehouse[¶m1=value1¶mN=valueN]
# or
# host:port/database/schema?account=user_account&warehouse=user_warehouse[¶m1=value1¶mN=valueN]
account - Name assigned to your Snowflake account. If you are not on us-west-2 or AWS deployment, append the region and platform to the end, e.g., <account>.<region> or <account>.<region>.<platform>.private_key (string) (optional)authenticator=snowflake_jwt&privateKey=... to the connection_string but
parses, validates, and correctly encodes the key for use with snowflake.migrate_concurrency (integer) (optional) (default: 1)1 or greater.batch_size (integer) (optional) (default: 5000)batch_size_bytes (integer) (optional) (default: 20971520 (= 20 MiB))leave_stage_files (boolean) (optional) (default: false)true, the intermediary files used to load data into the Snowflake stage are left in the local temp directory instead of being deleted after upload. This can be useful for debugging purposes.number type, which can handle arbitrary precisionfloat typearray type, including both regular and fixed-size listsvariant type for semi-structured data storagetimestamp_tz (timestamp with timezone)text representation