Check the impact of a source upgrade
cloudquery upgrade check tells you what a new source integration version changes in your destinations, and what to do before you upgrade. It compares the version in your configuration file with the version you name, using your table selection, transformers, destination version and destination settings. It does not migrate, write, delete or upload anything.
Prerequisites
- A configuration file that you sync with today. The check reads the current source version and its destinations from it.
- The same credentials as a sync of that configuration file:
- A CloudQuery login (
cloudquery login) or aCLOUDQUERY_API_KEYenvironment variable, to download the integrations. - Every environment variable that the configuration file references. The CLI expands them before the check starts, the same as for a sync.
- Source credentials, if the source finds its tables by connecting to a service. See How the check gets the source tables.
- A CloudQuery login (
Run the check
Pass the configuration file, the source name from the configuration file, and the version to upgrade to:
cloudquery upgrade check ./config.yml --source okta --to v7.0.0For this configuration file:
kind: source
spec:
name: okta
path: cloudquery/okta
registry: cloudquery
version: "v6.8.2"
tables: ["okta_policy_rules", "okta_policy_mappings"]
destinations: ["postgresql"]
spec:
# Okta source spec
---
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.17.0"
write_mode: overwrite-delete-stale
spec:
connection_string: "${POSTGRESQL_CONNECTION_STRING}"The check prints one report for each destination of the source:
okta v6.8.2 → v7.0.0 | postgresql (cloudquery/[email protected])
write_mode: overwrite-delete-stale | pk_mode: default | migrate_mode: safe
REVIEW REQUIRED — 1 table needs a manual migration, 1 new table
Changes
okta_policy_mappings new table
okta_policy_rules
+ policy_id text new column, part of the primary key (string)
+ actions jsonb new column (json)
+ conditions jsonb new column (json)
Next sync
migrate_mode: safe (your config)
✓ okta_policy_mappings created
✗ okta_policy_rules fails: safe mode cannot change a primary key
migrate_mode: forced
✓ okta_policy_mappings created
! okta_policy_rules dropped and recreated, existing rows deleted
Action: migrate okta_policy_rules manually before upgrading, or switch to migrate_mode: forced and accept losing its rows.
This check only previews the changes. It does not migrate, write, delete or upload anything.A report has these parts:
- Header: the source versions, the destination, and its
write_mode,pk_modeandmigrate_mode. - Result and a count of the affected tables. See Read the results.
- Changes: each new, removed or changed table and column, with the destination type and, in brackets, the source type.
- Next sync: what the next sync does to each changed table, for both values of
migrate_mode.safeis the default.(your config)marks the value in your configuration file.✓means the change is applied,✗means the sync fails, and!means the sync deletes existing rows. - Action: what to do before you upgrade.
The result depends on the destination settings in your configuration file: migrate_mode, write_mode and pk_mode. For example, with write_mode: append, the same Okta upgrade is AUTOMATICALLY MIGRATABLE, because append mode ignores source primary keys.
For all flags, see the cloudquery upgrade check reference.
Read the results
Each report prints one result as its verdict, in the text and JSON output. If more than one applies, the report prints the first one in this table. The first column is the category name.
| Category | Text and JSON verdict | What it means | What to do |
|---|---|---|---|
| Manual migration/rebuild required | REVIEW REQUIRED | With your migrate_mode, the next sync fails for a table, or drops and recreates it and deletes its rows. | Migrate the table by hand before you upgrade, or use migrate_mode: forced and accept losing its rows. With forced, back up the rows first. |
| Selected table removed | SELECTED TABLES REMOVED | Your configuration selects the table, and the new version no longer has it. | If tables names the table, remove it. Update the queries and consumers that read the table. The check does not delete it. |
| File schema/output changed | FILE SCHEMA CHANGED | A file destination writes new files with a different schema. | Update the readers that combine old and new files. |
| File schema/output changed | FILE OUTPUT CHANGED | A file destination writes a different output for at least one test value. The report shows the value under Output comparison, with OUTPUT DIFFERENCE DETECTED for some test values. | Update the readers that combine old and new files. |
| Unknown | UNKNOWN | The check could not fully assess a table, or could not list the source tables. Coverage gaps gives the reason. | Read the source changelog for the changes that the check could not assess, and review them by hand before you upgrade. |
| Automatically migratable | AUTOMATICALLY MIGRATABLE | With your migrate_mode, the next sync applies every change without deleting rows, for example a new column or a new table. | Upgrade. The next sync applies the changes. |
| No destination schema change | NO OUTPUT DIFFERENCE DETECTED for equivalent test values | File destinations only. The schema changes, and the destination writes the same output for the test values with the old and the new schema. See Limits. | Nothing. |
| No destination schema change | No schema changes affect your selected tables. | The upgrade does not change the destination schema of any selected table. | Nothing. |
Coverage gaps at the end of a report lists what the check could not fully assess, and why. For example, a destination version that does not support the check makes each of its tables unknown, with the reason destination version does not support assessment.
Examples
Select a destination to see the output of real upgrades. The PostgreSQL example in Run the check is not repeated here.
Datadog v5.19.10 → v6.0.0: a column type change
Seven tag columns across six tables change from a list of strings to JSON, so text[] becomes jsonb. safe mode cannot change a column type, so the result is REVIEW REQUIRED and the exit code is 3.
Configuration file
kind: source
spec:
name: datadog
path: cloudquery/datadog
registry: cloudquery
version: "v5.19.10"
tables: ["*"]
destinations: [postgresql]
spec:
# Datadog source spec
---
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.17.0"
write_mode: overwrite-delete-stale
spec:
connection_string: "${POSTGRESQL_CONNECTION_STRING}"cloudquery upgrade check ./config.yml --source datadog --to v6.0.0datadog v5.19.10 → v6.0.0 | postgresql (cloudquery/[email protected])
write_mode: overwrite-delete-stale | pk_mode: default | migrate_mode: safe
REVIEW REQUIRED — 6 tables need a manual migration
Changes
datadog_downtimes
~ monitor_tags text[] → jsonb type changed (list<string> → json)
datadog_global_variables
~ tags text[] → jsonb type changed (list<string> → json)
datadog_monitor_downtimes
~ monitor_tags text[] → jsonb type changed (list<string> → json)
datadog_monitors
~ tags text[] → jsonb type changed (list<string> → json)
datadog_slos
~ monitor_tags text[] → jsonb type changed (list<string> → json)
~ tags text[] → jsonb type changed (list<string> → json)
datadog_synthetics
~ tags text[] → jsonb type changed (list<string> → json)
Next sync
migrate_mode: safe (your config)
✗ datadog_downtimes fails: safe mode cannot change a column type
✗ datadog_global_variables fails: safe mode cannot change a column type
✗ datadog_monitor_downtimes fails: safe mode cannot change a column type
✗ datadog_monitors fails: safe mode cannot change a column type
✗ datadog_slos fails: safe mode cannot change a column type
✗ datadog_synthetics fails: safe mode cannot change a column type
migrate_mode: forced
! datadog_downtimes dropped and recreated, existing rows deleted
! datadog_global_variables dropped and recreated, existing rows deleted
! datadog_monitor_downtimes dropped and recreated, existing rows deleted
! datadog_monitors dropped and recreated, existing rows deleted
! datadog_slos dropped and recreated, existing rows deleted
! datadog_synthetics dropped and recreated, existing rows deleted
Action: migrate these 6 tables manually before upgrading, or switch to migrate_mode: forced and accept losing their rows.
This check only previews the changes. It does not migrate, write, delete or upload anything.Okta v6.8.2 → v7.0.0 with write_mode: overwrite
overwrite keeps the source primary keys, the same as overwrite-delete-stale in Run the check. The new primary key column needs a manual migration. The exit code is 3.
Configuration file
kind: source
spec:
name: okta
path: cloudquery/okta
registry: cloudquery
version: "v6.8.2"
tables: ["*"]
destinations: [postgresql]
spec:
# Okta source spec
---
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.17.0"
write_mode: overwrite
spec:
connection_string: "${POSTGRESQL_CONNECTION_STRING}"cloudquery upgrade check ./config.yml --source okta --to v7.0.0okta v6.8.2 → v7.0.0 | postgresql (cloudquery/[email protected])
write_mode: overwrite | pk_mode: default | migrate_mode: safe
REVIEW REQUIRED — 1 table needs a manual migration, 1 new table
Changes
okta_policy_mappings new table
okta_policy_rules
+ policy_id text new column, part of the primary key (string)
+ actions jsonb new column (json)
+ conditions jsonb new column (json)
Next sync
migrate_mode: safe (your config)
✓ okta_policy_mappings created
✗ okta_policy_rules fails: safe mode cannot change a primary key
migrate_mode: forced
✓ okta_policy_mappings created
! okta_policy_rules dropped and recreated, existing rows deleted
Action: migrate okta_policy_rules manually before upgrading, or switch to migrate_mode: forced and accept losing its rows.
This check only previews the changes. It does not migrate, write, delete or upload anything.Okta v6.8.2 → v7.0.0 with write_mode: append
append ignores source primary keys, so policy_id is a new column like the others and safe mode applies every change. The exit code is 0.
Configuration file
kind: source
spec:
name: okta
path: cloudquery/okta
registry: cloudquery
version: "v6.8.2"
tables: ["*"]
destinations: [postgresql]
spec:
# Okta source spec
---
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.17.0"
write_mode: append
spec:
connection_string: "${POSTGRESQL_CONNECTION_STRING}"cloudquery upgrade check ./config.yml --source okta --to v7.0.0okta v6.8.2 → v7.0.0 | postgresql (cloudquery/[email protected])
write_mode: append | pk_mode: default | migrate_mode: safe
AUTOMATICALLY MIGRATABLE — 1 changed table, 1 new table
Changes
okta_policy_mappings new table
okta_policy_rules
+ policy_id text new column (string)
+ actions jsonb new column (json)
+ conditions jsonb new column (json)
Next sync
migrate_mode: safe (your config)
✓ okta_policy_mappings created
✓ okta_policy_rules migrated in place
migrate_mode: forced — same as safe
Action: use safe migration.
This check only previews the changes. It does not migrate, write, delete or upload anything.GCP v22.1.2 → v23.0.0: selected tables removed
The configuration selects two tables that v23.0.0 no longer has. The check does not delete the existing tables. The exit code is 3.
Configuration file
kind: source
spec:
name: gcp
path: cloudquery/gcp
registry: cloudquery
version: "v22.1.2"
tables: ["gcp_aiplatform_specialist_pools", "gcp_aiplatform_specialistpool_locations"]
destinations: [postgresql]
spec:
# GCP source spec
---
kind: destination
spec:
name: postgresql
path: cloudquery/postgresql
registry: cloudquery
version: "v8.17.0"
write_mode: overwrite-delete-stale
spec:
connection_string: "${POSTGRESQL_CONNECTION_STRING}"cloudquery upgrade check ./config.yml --source gcp --to v23.0.0gcp v22.1.2 → v23.0.0 | postgresql (cloudquery/[email protected])
write_mode: overwrite-delete-stale | pk_mode: default
SELECTED TABLES REMOVED — 2 removed tables
Changes
gcp_aiplatform_specialist_pools removed table, the new source version no longer provides it
gcp_aiplatform_specialistpool_locations removed table, the new source version no longer provides it
Action: remove explicit selections and update dependent consumers.
This check only previews the changes. It does not migrate, write, delete or upload anything.How the check gets the source tables
The check starts both source versions and asks each one for its tables. It never runs a sync of the source, and it reads no rows.
- Sources with a fixed list of tables, such as AWS or Datadog, return their tables without a connection. The check does not connect to the service and does not use the source credentials.
- Sources that find their tables in the service, such as PostgreSQL, MySQL, BigQuery or ClickHouse, return no tables without a connection. The check starts them again with their connection and asks for the tables again. This reads metadata only, such as table and column names and types.
Coverage gapsshowstables were listed with a connection (metadata only, no rows read)for each version it connected. - Sources that find their tables only during a sync, such as S3, return no tables with or without a connection. Their report is
UNKNOWN.
Because some sources connect, give the check the same source credentials and network access as a sync wherever it runs, including CI.
The check starts each destination without a connection. It does not connect to your database or storage.
Use the JSON output in CI
Add --output json to print the reports as JSON:
cloudquery upgrade check ./config.yml --source okta --to v7.0.0 --output json > report.jsonThe JSON goes to standard output. Download progress goes to standard error, so report.json contains only the JSON. The output has one entry in reports for each destination. For the Okta example:
{
"reports": [
{
"source": { "name": "okta", "from_version": "v6.8.2", "to_version": "v7.0.0" },
"destination": {
"name": "postgresql",
"registry": "cloudquery",
"path": "cloudquery/postgresql",
"version": "v8.17.0",
"write_mode": "overwrite-delete-stale",
"pk_mode": "default",
"migrate_mode": "safe"
},
"verdict": "REVIEW REQUIRED",
"summary": "1 table needs a manual migration, 1 new table",
"tables": [
{
"name": "okta_policy_mappings",
"new_table": true,
"outcomes": {
"forced": { "result": "applied", "text": "created" },
"safe": { "result": "applied", "text": "created" }
}
},
{
"name": "okta_policy_rules",
"changes": [
{
"kind": "added",
"column": "policy_id",
"new_type": "text",
"new_source_type": "string",
"reason": "part of the primary key"
}
],
"outcomes": {
"forced": { "result": "data_loss", "text": "dropped and recreated, existing rows deleted" },
"safe": { "result": "fails", "text": "fails: safe mode cannot change a primary key" }
}
}
],
"removed_tables": [],
"output_comparisons": [],
"coverage_gaps": [],
"action": "migrate okta_policy_rules manually before upgrading, or switch to migrate_mode: forced and accept losing its rows."
}
]
}The actions and conditions columns are left out of this example.
| Field | Contents |
|---|---|
verdict | The result, with the same text as the text output. See Read the results. |
summary | The count of affected tables. Not present when no selected table changes. |
tables | Each new or changed table. changes lists its columns: kind is added, removed or changed. outcomes gives the next sync result for safe and forced: applied, fails or data_loss. Unchanged tables are not listed. |
removed_tables | The selected tables that the new version no longer has. |
output_comparisons | For file destinations, the test values written with the old and the new schema. See Limits. |
coverage_gaps | The same list as Coverage gaps in the text output. |
action | What to do before you upgrade. |
destination | null when the check could not list the source tables. The output then has one report, with no destination and the verdict UNKNOWN. |
Exit codes
The check prints the full report first, in text and JSON output, and then exits with one of these codes:
| Code | Meaning |
|---|---|
0 | No action needed. The result is AUTOMATICALLY MIGRATABLE, NO OUTPUT DIFFERENCE DETECTED for equivalent test values or No schema changes affect your selected tables. |
1 | The check failed with an error, for example a configuration file that is not valid or an integration that does not download. Other cloudquery commands use the same code. |
3 | Action needed. At least one table needs a manual migration or rebuild, a selected table is removed, or a file schema or output changes. |
4 | Unknown. The check could not assess a source or a destination, or the report lists a Coverage gaps entry for a table. For example, the destination version does not support the check, or the check could not list the source tables. |
A source that the check connects to only to list its tables does not make the code 4. The verdict shows the most urgent action, and the exit code shows whether the check is complete. If any table is unknown, the code is 4, even when the verdict is REVIEW REQUIRED. With several destinations, the check exits with the highest of 0, 3 and 4.
In CI, a step that runs the check fails on any code other than 0. Because the check writes the report before it exits, report.json from the JSON example is complete when the step fails.
Limits
- The check assesses the destination schema, and the output that a file destination writes for a schema. It does not check your data, the live schema of your database, or whether your queries still work after the upgrade.
- The check assumes that your destination tables match the current source version and the settings in your configuration file. If you created the tables with other settings, such as another
write_mode, or changed them by hand, the results can be wrong. - For file destinations, the check writes test values with the old and the new schema and compares the output.
output_comparisonsin the JSON output, andSynthetic value,BeforeandAfterunderOutput comparisonin the text output, show these test values. They are not your data. A file destination that writes the same output for the test values can still write a different output for your data.
Next steps
- How CloudQuery handles changes to existing tables - What
safeandforcedmigration modes do - Managing Versions - Choose and pin integration versions
- Destination Integrations - Configure
migrate_mode,write_modeandpk_mode
Last updated on