Check the impact of a source upgrade
cloudquery upgrade check tells you what a new source integration version changes in your destinations, and what to do before you upgrade. It compares the version in your configuration file with the version you name, using your table selection, transformers, destination version and destination settings. It does not migrate, write, delete or upload anything.
Prerequisites
- A configuration file that you sync with today. The check reads the current source version and its destinations from it. The source can come from the CloudQuery Hub, a local binary, a gRPC server or a Docker image. See Check a source from a local binary, gRPC server or Docker image.
- The same credentials as a sync of that configuration file:
- A CloudQuery login (
cloudquery login) or aCLOUDQUERY_API_KEYenvironment variable, to download the integrations. - Every environment variable that the configuration file references. The CLI expands them before the check starts, the same as for a sync.
- Source credentials, if the source finds its tables by connecting to a service. See How the check gets the source tables.
- A CloudQuery login (
Run the check
Pass the configuration file and the version to upgrade to:
cloudquery upgrade check ./config.yml --to v7.0.0If the configuration file has more than one source, also pass the source name from the configuration file with --source:
cloudquery upgrade check ./config.yml --source okta --to v7.0.0For this configuration file:
kind: source
spec:
name: okta
path: cloudquery/okta
registry: cloudquery
version: "v6.8.2"
tables: ["okta_policy_rules", "okta_policy_mappings"]
destinations: ["postgresql"]
spec:
# Okta source spec
---
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.19.0"
write_mode: overwrite-delete-stale
spec:
connection_string: "${POSTGRESQL_CONNECTION_STRING}"The check prints one report for each destination of the source:
okta v6.8.2 → v7.0.0 | postgresql (cloudquery/postgresql@v8.19.0)
write_mode: overwrite-delete-stale | pk_mode: default | migrate_mode: safe
REVIEW REQUIRED — 1 table needs a manual migration, 1 new table
Changes
okta_policy_rules
+ policy_id text new column, part of the primary key (string)
+ actions jsonb new column (json)
+ conditions jsonb new column (json)
okta_policy_mappings new table
Next sync
migrate_mode: safe (your config)
✗ okta_policy_rules fails: safe mode cannot change a primary key
✓ okta_policy_mappings created
migrate_mode: forced
! okta_policy_rules dropped and recreated, existing rows deleted
✓ okta_policy_mappings created
Action: migrate okta_policy_rules manually before upgrading, or switch to migrate_mode: forced and accept losing its rows.
This check only previews the changes. It does not migrate, write, delete or upload anything.A report has these parts:
- Header: the source versions, the destination, and its
write_mode,pk_modeandmigrate_mode. - Result and a count of the affected tables. See Read the results. If the report lists more than 10 tables and some of them are breaking changes, a
Breaking:line under the result names the removed tables and the tables that need a manual migration or lose rows. - Changes: each new, removed or changed table and column, with the destination type and, in brackets, the source type. For Parquet, the destination type is the short logical type, for example
int64,list<string>ortimestamp (us, UTC), andrequiredmarks a column that cannot be null. Breaking changes come first. The tables are in this order: removed tables, tables that need a manual migration or rebuild, tables with a file schema change, other changed tables, and then new tables. In each table, removed columns and changes thatsafemode cannot apply come first, then new columns. Items in the same group are in name order. - Next sync: what the next sync does to each changed table, for both values of
migrate_mode.safeis the default.(your config)marks the value in your configuration file.✓means the change is applied,✗means the sync fails, and!means the sync deletes existing rows. In eachmigrate_mode, the tables that fail come first, then the tables that lose rows, then the others. - Action: what to do before you upgrade.
The result depends on the destination settings in your configuration file: migrate_mode, write_mode and pk_mode. For example, with write_mode: append, the same Okta upgrade is AUTOMATICALLY MIGRATABLE, because append mode ignores source primary keys.
For all flags, see the cloudquery upgrade check reference.
Read the results
Each report prints one result as its verdict, in the text and JSON output. If more than one applies, the report prints the first one in this table. The first column is the category name.
| Category | Text and JSON verdict | What it means | What to do |
|---|---|---|---|
| Manual migration/rebuild required | REVIEW REQUIRED | With your migrate_mode, the next sync fails for a table, or drops and recreates it and deletes its rows. | Migrate the table by hand before you upgrade, or use migrate_mode: forced and accept losing its rows. With forced, back up the rows first. |
| Selected table removed | SELECTED TABLES REMOVED | Your configuration selects the table, and the new version no longer has it. | Remove the tables entries that the action names: entries that name a removed table, and patterns that match no table in the new version. A sync fails while these entries are in tables. If a wildcard such as * selected the table, or the table is a dependent table, you do not need to change tables. In all cases, update the queries and consumers that read the table. The removed table stops syncing, and its existing data in the destination is kept but no longer updated. |
| File schema/output changed | FILE SCHEMA CHANGED | A file destination writes new files with a different schema: a column is removed, changes type or becomes required, a new column is required, or a CSV column is not at the end. | Update the readers that combine old and new files. |
| File schema/output changed | FILE OUTPUT CHANGED | A file destination writes a different output for at least one test value. The report shows the value under Output comparison, with OUTPUT DIFFERENCE DETECTED for some test values. | Update the readers that combine old and new files. |
| Unknown | UNKNOWN | The check could not fully assess a table, or could not list the source tables. Coverage gaps gives the reason. | Read the source changelog for the changes that the check could not assess, and review them by hand before you upgrade. |
| File schema extended | FILE SCHEMA EXTENDED | A file destination writes new files that only add columns that can be null and new tables. Readers that match Parquet or JSON columns by name, such as Athena, Glue, Spark, Trino and Snowflake external tables, read old and new files together. A new CSV column is in this category only when it is at the end of the row and the files have a header row (skip_header: false, the default). | Nothing. Update the readers that match columns by position. |
| Automatically migratable | AUTOMATICALLY MIGRATABLE | With your migrate_mode, the next sync applies every change without deleting rows, for example a new column or a new table. | Upgrade. The next sync applies the changes. |
| No destination schema change | NO OUTPUT DIFFERENCE DETECTED for equivalent test values | File destinations only. The schema changes, and the destination writes the same output for the test values with the old and the new schema. See Limits. | Nothing. |
| No destination schema change | No schema changes affect your selected tables. | The upgrade does not change the destination schema of any selected table. | Nothing. |
Coverage gaps at the end of a report lists what the check could not fully assess, and why. For example, a destination version that does not support the check makes each of its tables unknown, with the reason destination version does not support assessment.
Examples
Select a destination to see the output of real upgrades. The PostgreSQL example in Run the check is not repeated here.
Datadog v5.19.10 → v6.0.0: a column type change
Seven tag columns across six tables change from a list of strings to JSON, so text[] becomes jsonb. safe mode cannot change a column type, so the result is REVIEW REQUIRED and the exit code is 3.
Configuration file
kind: source
spec:
name: datadog
path: cloudquery/datadog
registry: cloudquery
version: "v5.19.10"
tables: ["*"]
destinations: [postgresql]
spec:
# Datadog source spec
---
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.19.0"
write_mode: overwrite-delete-stale
spec:
connection_string: "${POSTGRESQL_CONNECTION_STRING}"cloudquery upgrade check ./config.yml --source datadog --to v6.0.0datadog v5.19.10 → v6.0.0 | postgresql (cloudquery/postgresql@v8.19.0)
write_mode: overwrite-delete-stale | pk_mode: default | migrate_mode: safe
REVIEW REQUIRED — 6 tables need a manual migration
Changes
datadog_downtimes
~ monitor_tags text[] → jsonb type changed (list<string> → json)
datadog_global_variables
~ tags text[] → jsonb type changed (list<string> → json)
datadog_monitor_downtimes
~ monitor_tags text[] → jsonb type changed (list<string> → json)
datadog_monitors
~ tags text[] → jsonb type changed (list<string> → json)
datadog_slos
~ monitor_tags text[] → jsonb type changed (list<string> → json)
~ tags text[] → jsonb type changed (list<string> → json)
datadog_synthetics
~ tags text[] → jsonb type changed (list<string> → json)
Next sync
migrate_mode: safe (your config)
✗ datadog_downtimes fails: safe mode cannot change a column type
✗ datadog_global_variables fails: safe mode cannot change a column type
✗ datadog_monitor_downtimes fails: safe mode cannot change a column type
✗ datadog_monitors fails: safe mode cannot change a column type
✗ datadog_slos fails: safe mode cannot change a column type
✗ datadog_synthetics fails: safe mode cannot change a column type
migrate_mode: forced
! datadog_downtimes dropped and recreated, existing rows deleted
! datadog_global_variables dropped and recreated, existing rows deleted
! datadog_monitor_downtimes dropped and recreated, existing rows deleted
! datadog_monitors dropped and recreated, existing rows deleted
! datadog_slos dropped and recreated, existing rows deleted
! datadog_synthetics dropped and recreated, existing rows deleted
Action: migrate these 6 tables manually before upgrading, or switch to migrate_mode: forced and accept losing their rows.
This check only previews the changes. It does not migrate, write, delete or upload anything.Okta v6.8.2 → v7.0.0 with write_mode: overwrite
overwrite keeps the source primary keys, the same as overwrite-delete-stale in Run the check. The new primary key column needs a manual migration. The exit code is 3.
Configuration file
kind: source
spec:
name: okta
path: cloudquery/okta
registry: cloudquery
version: "v6.8.2"
tables: ["*"]
destinations: [postgresql]
spec:
# Okta source spec
---
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.19.0"
write_mode: overwrite
spec:
connection_string: "${POSTGRESQL_CONNECTION_STRING}"cloudquery upgrade check ./config.yml --source okta --to v7.0.0okta v6.8.2 → v7.0.0 | postgresql (cloudquery/postgresql@v8.19.0)
write_mode: overwrite | pk_mode: default | migrate_mode: safe
REVIEW REQUIRED — 1 table needs a manual migration, 1 new table
Changes
okta_policy_rules
+ policy_id text new column, part of the primary key (string)
+ actions jsonb new column (json)
+ conditions jsonb new column (json)
okta_policy_mappings new table
Next sync
migrate_mode: safe (your config)
✗ okta_policy_rules fails: safe mode cannot change a primary key
✓ okta_policy_mappings created
migrate_mode: forced
! okta_policy_rules dropped and recreated, existing rows deleted
✓ okta_policy_mappings created
Action: migrate okta_policy_rules manually before upgrading, or switch to migrate_mode: forced and accept losing its rows.
This check only previews the changes. It does not migrate, write, delete or upload anything.Okta v6.8.2 → v7.0.0 with write_mode: append
append ignores source primary keys, so policy_id is a new column like the others and safe mode applies every change. The exit code is 0.
Configuration file
kind: source
spec:
name: okta
path: cloudquery/okta
registry: cloudquery
version: "v6.8.2"
tables: ["*"]
destinations: [postgresql]
spec:
# Okta source spec
---
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.19.0"
write_mode: append
spec:
connection_string: "${POSTGRESQL_CONNECTION_STRING}"cloudquery upgrade check ./config.yml --source okta --to v7.0.0okta v6.8.2 → v7.0.0 | postgresql (cloudquery/postgresql@v8.19.0)
write_mode: append | pk_mode: default | migrate_mode: safe
AUTOMATICALLY MIGRATABLE — 1 changed table, 1 new table
Changes
okta_policy_rules
+ actions jsonb new column (json)
+ conditions jsonb new column (json)
+ policy_id text new column (string)
okta_policy_mappings new table
Next sync
migrate_mode: safe (your config)
✓ okta_policy_rules migrated in place
✓ okta_policy_mappings created
migrate_mode: forced — same as safe
Action: use safe migration.
This check only previews the changes. It does not migrate, write, delete or upload anything.GCP v22.1.2 → v23.0.0: selected tables removed
The configuration selects two tables that v23.0.0 no longer has. The check does not delete the existing tables. The exit code is 3.
Configuration file
kind: source
spec:
name: gcp
path: cloudquery/gcp
registry: cloudquery
version: "v22.1.2"
tables: ["gcp_aiplatform_specialist_pools", "gcp_aiplatform_specialistpool_locations"]
destinations: [postgresql]
spec:
# GCP source spec
---
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.19.0"
write_mode: overwrite-delete-stale
spec:
connection_string: "${POSTGRESQL_CONNECTION_STRING}"cloudquery upgrade check ./config.yml --source gcp --to v23.0.0gcp v22.1.2 → v23.0.0 | postgresql (cloudquery/postgresql@v8.19.0)
write_mode: overwrite-delete-stale | pk_mode: default
SELECTED TABLES REMOVED — 2 removed tables
Changes
gcp_aiplatform_specialist_pools removed table, the new source version no longer provides it
gcp_aiplatform_specialistpool_locations removed table, the new source version no longer provides it
Action: remove explicit selections (gcp_aiplatform_specialist_pools, gcp_aiplatform_specialistpool_locations) and update dependent consumers.
This check only previews the changes. It does not migrate, write, delete or upload anything.How the check gets the source tables
The check starts both source versions and asks each one for its tables. It never runs a sync of the source, and it reads no rows.
- Sources with a fixed list of tables, such as AWS or Datadog, return their tables without a connection. The check does not connect to the service and does not use the source credentials.
- Sources that find their tables in the service, such as PostgreSQL, MySQL, BigQuery or ClickHouse, return no tables without a connection. The check starts them again with their connection and asks for the tables again. This reads metadata only, such as table and column names and types.
Coverage gapsshowstables were listed with a connection (metadata only, no rows read)for each version it connected. - Sources that find their tables only during a sync, such as S3, return no tables with or without a connection. Their report is
UNKNOWN.
Because some sources connect, give the check the same source credentials and network access as a sync wherever it runs, including CI.
The check starts each destination without a connection. It does not connect to your database or storage.
Check a source from a local binary, gRPC server or Docker image
The check also works for sources with registry: local, registry: grpc or registry: docker. For these registries, the version field does not select the binary, so the check gets each side from a different place:
- Current version: the check starts the source exactly as configured: the local binary, the running gRPC server or the Docker image. The report shows the version that the source reports, or
unknownif the source does not report a version. - New version: the check downloads the
--toversion from the CloudQuery Hub. By default, the Hub path iscloudquery/<name>, where<name>is the name that the running source reports. This works for CloudQuery integrations.
For this configuration file:
kind: source
spec:
name: okta
path: /opt/plugins/okta
registry: local
tables: ["okta_policy_rules", "okta_policy_mappings"]
destinations: ["postgresql"]
spec:
# Okta source specRun the check the same as for a source from the CloudQuery Hub:
cloudquery upgrade check ./config.yml --source okta --to v7.0.0The first line of the report shows where each side comes from:
okta (local: /opt/plugins/okta, v6.8.2) → cloudquery/okta@v7.0.0 | postgresql (cloudquery/postgresql@v8.19.0)If the source does not report its name, or its Hub path is not cloudquery/<name>, use --to-path to set the Hub path as <team>/<name>:
cloudquery upgrade check ./config.yml --source okta --to v7.0.0 --to-path cloudquery/okta--to-path is only for sources from the local, grpc and docker registries. A gRPC source must be running when you start the check. To download the new version, the check needs a CloudQuery login or a CLOUDQUERY_API_KEY environment variable, also when no integration in the configuration file comes from the CloudQuery Hub.
Use the JSON output in CI
Add --output json to print the reports as JSON:
cloudquery upgrade check ./config.yml --source okta --to v7.0.0 --output json > report.jsonThe JSON goes to standard output. Download progress goes to standard error, so report.json contains only the JSON. The output has one entry in reports for each destination. For the Okta example:
{
"reports": [
{
"source": {
"name": "okta",
"from_version": "v6.8.2",
"to_version": "v7.0.0",
"from": { "registry": "cloudquery", "path": "cloudquery/okta", "version": "v6.8.2" },
"to": { "registry": "cloudquery", "path": "cloudquery/okta", "version": "v7.0.0" }
},
"destination": {
"name": "postgresql",
"registry": "cloudquery",
"path": "cloudquery/postgresql",
"version": "v8.19.0",
"write_mode": "overwrite-delete-stale",
"pk_mode": "default",
"migrate_mode": "safe"
},
"verdict": "REVIEW REQUIRED",
"summary": "1 table needs a manual migration, 1 new table",
"tables": [
{
"name": "okta_policy_rules",
"changes": [
{
"kind": "added",
"column": "policy_id",
"new_type": "text",
"new_source_type": "string",
"reason": "part of the primary key"
}
],
"outcomes": {
"forced": { "result": "data_loss", "text": "dropped and recreated, existing rows deleted" },
"safe": { "result": "fails", "text": "fails: safe mode cannot change a primary key" }
}
},
{
"name": "okta_policy_mappings",
"new_table": true,
"outcomes": {
"forced": { "result": "applied", "text": "created" },
"safe": { "result": "applied", "text": "created" }
}
}
],
"removed_tables": [],
"output_comparisons": [],
"coverage_gaps": [],
"action": "migrate okta_policy_rules manually before upgrading, or switch to migrate_mode: forced and accept losing its rows."
}
]
}The actions and conditions columns are left out of this example.
| Field | Contents |
|---|---|
source | The source name and both versions. from and to give the registry, path and version of each side. For a source from a local binary, gRPC server or Docker image, from is the configured source and to is the CloudQuery Hub integration. See Check a source from a local binary, gRPC server or Docker image. |
verdict | The result, with the same text as the text output. See Read the results. |
summary | The count of affected tables. Not present when no selected table changes. |
tables | Each new or changed table, in the same order as Changes in the text output. changes lists its columns: kind is added, removed or changed. For Parquet, old_type and new_type are the short types from the text output, and old_parquet_type and new_parquet_type are the full Parquet schema types. outcomes gives the next sync result for safe and forced: applied, fails or data_loss. Unchanged tables are not listed. |
removed_tables | The selected tables that the new version no longer has. |
output_comparisons | For file destinations, the test values written with the old and the new schema. Entries with a different output come first. See Limits. |
coverage_gaps | The same list as Coverage gaps in the text output. |
action | What to do before you upgrade. |
destination | null when the check could not list the source tables. The output then has one report, with no destination and the verdict UNKNOWN. |
Exit codes
The check prints the full report first, in text and JSON output, and then exits with one of these codes:
| Code | Meaning |
|---|---|
0 | No action needed. The result is FILE SCHEMA EXTENDED, AUTOMATICALLY MIGRATABLE, NO OUTPUT DIFFERENCE DETECTED for equivalent test values or No schema changes affect your selected tables. |
1 | The check failed with an error, for example a configuration file that is not valid or an integration that does not download. Other cloudquery commands use the same code. |
3 | Action needed. At least one table needs a manual migration or rebuild, a selected table is removed, or a file schema or output changes. |
4 | Unknown. The check could not assess a source or a destination, or the report lists a Coverage gaps entry for a table. For example, the destination version does not support the check, or the check could not list the source tables. |
A source that the check connects to only to list its tables does not make the code 4. The verdict shows the most urgent action, and the exit code shows whether the check is complete. If any table is unknown, the code is 4, even when the verdict is REVIEW REQUIRED. With several destinations, the check exits with the highest of 0, 3 and 4.
In CI, a step that runs the check fails on any code other than 0. Because the check writes the report before it exits, report.json from the JSON example is complete when the step fails.
Limits
- The check assesses the destination schema, and the output that a file destination writes for a schema. It does not check your data, the live schema of your database, or whether your queries still work after the upgrade.
- The check assumes that your destination tables match the current source version and the settings in your configuration file. If you created the tables with other settings, such as another
write_mode, or changed them by hand, the results can be wrong. - For file destinations, the check writes test values with the old and the new schema and compares the output.
output_comparisonsin the JSON output, andSynthetic value,BeforeandAfterunderOutput comparisonin the text output, show these test values. They are not your data. A file destination that writes the same output for the test values can still write a different output for your data.
Next steps
- How CloudQuery handles changes to existing tables - What
safeandforcedmigration modes do - Managing Versions - Choose and pin integration versions
- Destination Integrations - Configure
migrate_mode,write_modeandpk_mode
Last updated on