The CloudQuery transformer plugin provides basic transformation capabilities, such as removing columns, adding literal string columns, obfuscating string columns and renaming tables
Loading plugin documentation
CloudQuery's use of cookies
We use tracking cookies to understand how you use the product and help us improve it. Your consent is required before we can enable these cookies. You can opt out via the link in the footer.
{{.OldName}} to refer to the original name, see example below)kind: destination
spec:
name: "postgresql"
path: "cloudquery/postgresql"
registry: "cloudquery"
version: "v8.0.7"
write_mode: "overwrite-delete-stale"
migrate_mode: forced # optional
transformers:
- "basic"
spec:
connection_string: "postgresql://your.user:your.password@localhost:5432/db_name"
migrate_mode: forced setting might make sense if you plan on modifying the schema from a previous sync.kind: transformer
spec:
name: "basic"
path: "cloudquery/basic"
version: v2.7.0
spec:
transformations:
- kind: obfuscate_columns
tables: ["xkcd_comics"]
columns: ["safe_title", "title"]
include_sha: false
- kind: obfuscate_sensitive_columns
- kind: obfuscate_columns_except
tables: ["xkcd_comics"]
unmatched: drop
columns: ["num", "img"]
- kind: remove_columns
tables: ["xkcd_comics"]
columns: ["transcript", "news"]
- kind: add_column
tables: ["xkcd_comics"]
name: "source"
value: "xkcd"
- kind: add_primary_keys
tables: ["xkcd_comics"]
columns: ["_cq_source_name"]
- kind: add_current_timestamp_column
tables: ["xkcd_comics"]
name: "_record_processed_at"
- kind: change_table_names
tables: ["*"]
new_table_name_template: "cq_sync_{{.OldName}}"
- kind: rename_column
tables: ["xkcd_comics"]
name: img
value: img_url
- kind: uppercase
tables: ["xkcd_comics"]
columns: ["title"]
- kind: lowercase
tables: ["xkcd_comics"]
columns: ["title"]
- kind: drop_rows
tables: ["xkcd_comics"]
columns: ["year"]
value: "2023"
tags:{"foo":{"bar":["a","b","c"]},"hello":"world","kubectl.kubernetes.io/last-applied-configuration":"secrets"}
"a" and remove "b", "world", and "secrets" with:kind: transformer
spec:
name: "basic"
path: "cloudquery/basic"
registry: "cloudquery"
spec:
transformations:
- kind: obfuscate_columns
tables: ["example"]
columns: ["tags.foo.bar.0"]
- kind: remove_columns
tables: ["example"]
columns: ["tags.hello", "tags.foo.bar.1", "tags.kubectl\\.kubernetes\\.io\\/last-applied-configuration"]
example_column with value: {"top_foo":[{"foo": "baz0"},{"foo": "baz1"},{"foo": "baz2"}]} you can use the following syntax:kind: transformer
spec:
name: "basic"
path: "cloudquery/basic"
registry: "cloudquery"
spec:
transformations:
- kind: obfuscate_columns
tables: ["example"]
columns: ["example_column.top_foo.#.foo"]
#.foo syntax will cause all foo values to be replaced with the same obfuscated value {"top_foo":[{"foo": "Redacted by CloudQuery | XXX"},{"foo": "Redacted by CloudQuery | XXX"},{"foo": "Redacted by CloudQuery | XXX"}]}. (obfuscate_columns_except, described below, hashes each array element separately instead.)obfuscate_sensitive_columns transformation to automatically obfuscate all columns marked by the source plugin as sensitive and possibly containing secret information.obfuscate_columns_except #obfuscate_columns_except is the inverse of obfuscate_columns: instead of listing the columns to redact, you list the columns (and JSON sub-paths) to keep, and everything else is dropped or redacted according to unmatched. This is useful for compliance-sensitive syncs where only an explicitly approved set of fields may leave unredacted.columns is the allowlist. In every mode, CloudQuery internal columns (those prefixed with _cq_) pass through untouched, a bare column name (e.g. status_phase) keeps that whole column, and a dotted entry (e.g. spec_containers.#.image) keeps that JSON sub-path.unmatched:drop (default) — non-allowlisted fields are removed entirely: non-allowlisted top-level columns are dropped from the schema, and non-allowlisted leaves inside a kept JSON column are removed. Smallest output.collapse — each non-allowlisted object or array is replaced by a single redaction marker; top-level non-allowlisted columns are obfuscated to one marker. Keeps the shape without a marker per leaf.redact — every non-allowlisted leaf value is replaced by its own marker, preserving the full JSON structure (array leaves hashed individually). Most verbose; a top-level column whose type cannot be redacted (numbers, timestamps, lists, structs) is dropped.drop is recommended for allowlist/compliance syncs — non-approved data is simply absent. On a real Kubernetes pods table, drop cut the redacted spec_containers column from ~900 KB to ~10 KB and the table from 59 columns to 11 (collapse was the middle ground at ~200 KB).kind: transformer
spec:
name: "basic"
path: "cloudquery/basic"
registry: "cloudquery"
spec:
transformations:
- kind: obfuscate_columns_except
tables: ["k8s_core_pods"]
unmatched: drop
columns:
- uid
- status_phase
- context
- namespace
- "spec_containers.#.image"
- "spec_init_containers.#.image"
- "status_container_statuses.#.state.terminated.finishedAt"
uid is the primary key of k8s_core_pods, so it has to be on the allowlist — see the primary-key note below.obfuscate_columns. If a path segment traverses a JSON array, you must write # for that segment. For example, to keep the image of every container, the path is spec_containers.#.image (not spec_containers.image). A path that doesn't match — a missing #, a typo in a nested segment — does not error; it simply causes that field to be treated as non-allowlisted (dropped, collapsed, or redacted per unmatched) instead of kept, so double-check nested paths against your data.Note:obfuscate_columns_exceptonly redacts fields within the tables it is pointed at; it does not disable other tables. Restrict which tables are synced with the source plugin'stables/skip_tablesoptions so that no unapproved table is ever written unredacted.
unmatched: drop (the column is removed), under redact/collapse with include_sha: false (every row collapses to an identical marker), and for any primary-key type that cannot be redacted in place. Allowlist your primary-key column(s) — for the Kubernetes tables that is uid. Under redact/collapse with the default include_sha: true, a non-allowlisted primary key is redacted to a distinct, stable hash and upserts still work.redact and collapse, JSON object keys and array shape remain visible (only leaf values are redacted). Under drop, non-allowlisted keys and columns are removed entirely.TypeSchema), that metadata can describe a type the redacted value no longer matches — a nested timestamp leaf under redact, or a whole nested object/array replaced by one marker under collapse. Destinations that consume TypeSchema are expected to fall back to storing the value as a string when it does not match (the MongoDB destination does this). Use unmatched: drop to avoid the mismatch entirely, since non-allowlisted nested fields are then absent rather than retyped.Redacted by CloudQuery | <sha>) so that distinct values remain distinguishable. Set include_sha: false on any obfuscation transformation to omit the hash and emit a bare Redacted by CloudQuery marker instead. This applies to obfuscate_columns, obfuscate_sensitive_columns, and obfuscate_columns_except — though for obfuscate_columns_except it has no effect under the default unmatched: drop, since nothing is replaced by a marker in that mode. - kind: obfuscate_columns
tables: ["k8s_core_pods"]
include_sha: false
columns: ["annotations"]
include_sha: false makes every redacted value identical, avoid it on a column that is (or is part of) a primary key — obfuscate_columns_except will refuse to run rather than silently collapse rows (see the primary-keys note above).drop_rows transformation:nil values, configure value: null, value: ~ or drop the value configuration altogethertags has a value of {"foo": "bar"}, you should specify the value as {"foo":"bar"} without any whitespace.