Back to plugin list
Official
ClickHouse
This destination plugin lets you sync data from a CloudQuery source to ClickHouse database
Publisher
cloudquery
Repositorygithub.com
Latest version
v3.4.14
Type
Destination
Platforms
Date Published
Mar 12, 2024
Price
Free
Overview
ClickHouse destination plugin
This destination plugin lets you sync data from a CloudQuery source to ClickHouse database.
It supports
append
write mode only.
Write mode selection is required through write_mode
.Supported database versions: >=
22.1.2
Configuration
Example
kind: destination
spec:
name: "clickhouse"
path: "cloudquery/clickhouse"
registry: "cloudquery"
version: "v3.4.14"
write_mode: "append"
spec:
connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}"
# Optional parameters
# cluster: ""
# ca_cert: ""
# engine:
# name: MergeTree
# parameters: []
#
# batch_size: 10000
# batch_size_bytes: 5242880 # 5 MiB
# batch_timeout: 20s
This example configures a ClickHouse instance, located at
localhost:9000
.
It expects CH_USER
, CH_PASSWORD
and CH_DATABASE
environment variables to be set.
The (top level) spec section is described in the Destination Spec Reference.ClickHouse spec
This is the (nested) spec used by the ClickHouse destination plugin.
connection_string
(string
) (required)Connection string to connect to the database. See SDK documentation for more details.cluster
(string
) (optional) (default: not used)Cluster name to be used for distributed DDL. If the value is empty, DDL operations will affect only the server the plugin is connected to.ca_cert
(string
) (optional) (default: not used)PEM-encoded certificate authorities. When set, a certificate pool will be created by appending the certificates to the system pool.See file variable substitution for how to read this value from a file.- Engine to be used for tables. Only
*MergeTree
family is supported at the moment. batch_size
(integer
) (optional) (default:10000
)Maximum number of items that may be grouped together to be written in a single write.batch_size_bytes
(integer
) (optional) (default:5242880
(= 5 MiB))Maximum size of items that may be grouped together to be written in a single write.batch_timeout
(duration
) (optional) (default:20s
)Maximum interval between batch writes.
ClickHouse table engine
This option allows to specify a custom table engine to be used.
name
(string
) (required)Name of the table engine. Only*MergeTree
family is supported at the moment.parameters
(array of parameters) (optional) (default: empty)Engine parameters. Currently, no restrictions are imposed on the parameter types.
kind: destination
spec:
name: "clickhouse"
path: "cloudquery/clickhouse"
registry: "cloudquery"
version: "v3.4.14"
write_mode: "append"
spec:
connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}"
engine:
name: ReplicatedMergeTree
parameters:
- "/clickhouse/tables/{shard}/{database}/{table}"
- "{replica}"
Verbose logging for debug
The ClickHouse destination can be run in debug mode.
To achieve this pass the
debug=true
option to connection_string
.
See SDK documentation for more details.Note: This will use SDK built-in logging
and might output data and sensitive information to logs.
Make sure not to use it in production environment.
kind: destination
spec:
name: "clickhouse"
path: "cloudquery/clickhouse"
registry: "cloudquery"
version: "v3.4.14"
write_mode: "append"
spec:
connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}?debug=true"