Skip to Content

Check the impact of a source upgrade

cloudquery upgrade check tells you what a new source integration version changes in your destinations, and what to do before you upgrade. It compares the version in your configuration file with the version you name, using your table selection, transformers, destination version and destination settings. It does not migrate, write, delete or upload anything.

Prerequisites

  • A configuration file that you sync with today. The check reads the current source version and its destinations from it.
  • The same credentials as a sync of that configuration file:
    • A CloudQuery login (cloudquery login) or a CLOUDQUERY_API_KEY environment variable, to download the integrations.
    • Every environment variable that the configuration file references. The CLI expands them before the check starts, the same as for a sync.
    • Source credentials, if the source finds its tables by connecting to a service. See How the check gets the source tables.

Run the check

Pass the configuration file, the source name from the configuration file, and the version to upgrade to:

cloudquery upgrade check ./config.yml --source okta --to v7.0.0

For this configuration file:

kind: source spec: name: okta path: cloudquery/okta registry: cloudquery version: "v6.8.2" tables: ["okta_policy_rules", "okta_policy_mappings"] destinations: ["postgresql"] spec: # Okta source spec --- kind: destination spec: name: postgresql path: cloudquery/postgresql registry: cloudquery version: "v8.17.0" write_mode: overwrite-delete-stale spec: connection_string: "${POSTGRESQL_CONNECTION_STRING}"

The check prints one report for each destination of the source:

okta v6.8.2 → v7.0.0 | postgresql (cloudquery/[email protected]) write_mode: overwrite-delete-stale | pk_mode: default | migrate_mode: safe REVIEW REQUIRED — 1 table needs a manual migration, 1 new table Changes okta_policy_mappings new table okta_policy_rules + policy_id text new column, part of the primary key (string) + actions jsonb new column (json) + conditions jsonb new column (json) Next sync migrate_mode: safe (your config) ✓ okta_policy_mappings created ✗ okta_policy_rules fails: safe mode cannot change a primary key migrate_mode: forced ✓ okta_policy_mappings created ! okta_policy_rules dropped and recreated, existing rows deleted Action: migrate okta_policy_rules manually before upgrading, or switch to migrate_mode: forced and accept losing its rows. This check only previews the changes. It does not migrate, write, delete or upload anything.

A report has these parts:

  • Header: the source versions, the destination, and its write_mode, pk_mode and migrate_mode.
  • Result and a count of the affected tables. See Read the results.
  • Changes: each new, removed or changed table and column, with the destination type and, in brackets, the source type.
  • Next sync: what the next sync does to each changed table, for both values of migrate_mode. safe is the default. (your config) marks the value in your configuration file. ✓ means the change is applied, ✗ means the sync fails, and ! means the sync deletes existing rows.
  • Action: what to do before you upgrade.

The result depends on the destination settings in your configuration file: migrate_mode, write_mode and pk_mode. For example, with write_mode: append, the same Okta upgrade is AUTOMATICALLY MIGRATABLE, because append mode ignores source primary keys.

For all flags, see the cloudquery upgrade check reference.

Read the results

Each report prints one result as its verdict, in the text and JSON output. If more than one applies, the report prints the first one in this table. The first column is the category name.

CategoryText and JSON verdictWhat it meansWhat to do
Manual migration/rebuild requiredREVIEW REQUIREDWith your migrate_mode, the next sync fails for a table, or drops and recreates it and deletes its rows.Migrate the table by hand before you upgrade, or use migrate_mode: forced and accept losing its rows. With forced, back up the rows first.
Selected table removedSELECTED TABLES REMOVEDYour configuration selects the table, and the new version no longer has it.If tables names the table, remove it. Update the queries and consumers that read the table. The check does not delete it.
File schema/output changedFILE SCHEMA CHANGEDA file destination writes new files with a different schema.Update the readers that combine old and new files.
File schema/output changedFILE OUTPUT CHANGEDA file destination writes a different output for at least one test value. The report shows the value under Output comparison, with OUTPUT DIFFERENCE DETECTED for some test values.Update the readers that combine old and new files.
UnknownUNKNOWNThe check could not fully assess a table, or could not list the source tables. Coverage gaps gives the reason.Read the source changelog for the changes that the check could not assess, and review them by hand before you upgrade.
Automatically migratableAUTOMATICALLY MIGRATABLEWith your migrate_mode, the next sync applies every change without deleting rows, for example a new column or a new table.Upgrade. The next sync applies the changes.
No destination schema changeNO OUTPUT DIFFERENCE DETECTED for equivalent test valuesFile destinations only. The schema changes, and the destination writes the same output for the test values with the old and the new schema. See Limits.Nothing.
No destination schema changeNo schema changes affect your selected tables.The upgrade does not change the destination schema of any selected table.Nothing.

Coverage gaps at the end of a report lists what the check could not fully assess, and why. For example, a destination version that does not support the check makes each of its tables unknown, with the reason destination version does not support assessment.

Examples

Select a destination to see the output of real upgrades. The PostgreSQL example in Run the check is not repeated here.

Datadog v5.19.10 → v6.0.0: a column type change

Seven tag columns across six tables change from a list of strings to JSON, so text[] becomes jsonb. safe mode cannot change a column type, so the result is REVIEW REQUIRED and the exit code is 3.

Configuration file

kind: source spec: name: datadog path: cloudquery/datadog registry: cloudquery version: "v5.19.10" tables: ["*"] destinations: [postgresql] spec: # Datadog source spec --- kind: destination spec: name: postgresql path: cloudquery/postgresql registry: cloudquery version: "v8.17.0" write_mode: overwrite-delete-stale spec: connection_string: "${POSTGRESQL_CONNECTION_STRING}"
cloudquery upgrade check ./config.yml --source datadog --to v6.0.0
datadog v5.19.10 → v6.0.0 | postgresql (cloudquery/[email protected]) write_mode: overwrite-delete-stale | pk_mode: default | migrate_mode: safe REVIEW REQUIRED — 6 tables need a manual migration Changes datadog_downtimes ~ monitor_tags text[] → jsonb type changed (list<string> → json) datadog_global_variables ~ tags text[] → jsonb type changed (list<string> → json) datadog_monitor_downtimes ~ monitor_tags text[] → jsonb type changed (list<string> → json) datadog_monitors ~ tags text[] → jsonb type changed (list<string> → json) datadog_slos ~ monitor_tags text[] → jsonb type changed (list<string> → json) ~ tags text[] → jsonb type changed (list<string> → json) datadog_synthetics ~ tags text[] → jsonb type changed (list<string> → json) Next sync migrate_mode: safe (your config) ✗ datadog_downtimes fails: safe mode cannot change a column type ✗ datadog_global_variables fails: safe mode cannot change a column type ✗ datadog_monitor_downtimes fails: safe mode cannot change a column type ✗ datadog_monitors fails: safe mode cannot change a column type ✗ datadog_slos fails: safe mode cannot change a column type ✗ datadog_synthetics fails: safe mode cannot change a column type migrate_mode: forced ! datadog_downtimes dropped and recreated, existing rows deleted ! datadog_global_variables dropped and recreated, existing rows deleted ! datadog_monitor_downtimes dropped and recreated, existing rows deleted ! datadog_monitors dropped and recreated, existing rows deleted ! datadog_slos dropped and recreated, existing rows deleted ! datadog_synthetics dropped and recreated, existing rows deleted Action: migrate these 6 tables manually before upgrading, or switch to migrate_mode: forced and accept losing their rows. This check only previews the changes. It does not migrate, write, delete or upload anything.

Okta v6.8.2 → v7.0.0 with write_mode: overwrite

overwrite keeps the source primary keys, the same as overwrite-delete-stale in Run the check. The new primary key column needs a manual migration. The exit code is 3.

Configuration file

kind: source spec: name: okta path: cloudquery/okta registry: cloudquery version: "v6.8.2" tables: ["*"] destinations: [postgresql] spec: # Okta source spec --- kind: destination spec: name: postgresql path: cloudquery/postgresql registry: cloudquery version: "v8.17.0" write_mode: overwrite spec: connection_string: "${POSTGRESQL_CONNECTION_STRING}"
cloudquery upgrade check ./config.yml --source okta --to v7.0.0
okta v6.8.2 → v7.0.0 | postgresql (cloudquery/[email protected]) write_mode: overwrite | pk_mode: default | migrate_mode: safe REVIEW REQUIRED — 1 table needs a manual migration, 1 new table Changes okta_policy_mappings new table okta_policy_rules + policy_id text new column, part of the primary key (string) + actions jsonb new column (json) + conditions jsonb new column (json) Next sync migrate_mode: safe (your config) ✓ okta_policy_mappings created ✗ okta_policy_rules fails: safe mode cannot change a primary key migrate_mode: forced ✓ okta_policy_mappings created ! okta_policy_rules dropped and recreated, existing rows deleted Action: migrate okta_policy_rules manually before upgrading, or switch to migrate_mode: forced and accept losing its rows. This check only previews the changes. It does not migrate, write, delete or upload anything.

Okta v6.8.2 → v7.0.0 with write_mode: append

append ignores source primary keys, so policy_id is a new column like the others and safe mode applies every change. The exit code is 0.

Configuration file

kind: source spec: name: okta path: cloudquery/okta registry: cloudquery version: "v6.8.2" tables: ["*"] destinations: [postgresql] spec: # Okta source spec --- kind: destination spec: name: postgresql path: cloudquery/postgresql registry: cloudquery version: "v8.17.0" write_mode: append spec: connection_string: "${POSTGRESQL_CONNECTION_STRING}"
cloudquery upgrade check ./config.yml --source okta --to v7.0.0
okta v6.8.2 → v7.0.0 | postgresql (cloudquery/[email protected]) write_mode: append | pk_mode: default | migrate_mode: safe AUTOMATICALLY MIGRATABLE — 1 changed table, 1 new table Changes okta_policy_mappings new table okta_policy_rules + policy_id text new column (string) + actions jsonb new column (json) + conditions jsonb new column (json) Next sync migrate_mode: safe (your config) ✓ okta_policy_mappings created ✓ okta_policy_rules migrated in place migrate_mode: forced — same as safe Action: use safe migration. This check only previews the changes. It does not migrate, write, delete or upload anything.

GCP v22.1.2 → v23.0.0: selected tables removed

The configuration selects two tables that v23.0.0 no longer has. The check does not delete the existing tables. The exit code is 3.

Configuration file

kind: source spec: name: gcp path: cloudquery/gcp registry: cloudquery version: "v22.1.2" tables: ["gcp_aiplatform_specialist_pools", "gcp_aiplatform_specialistpool_locations"] destinations: [postgresql] spec: # GCP source spec --- kind: destination spec: name: postgresql path: cloudquery/postgresql registry: cloudquery version: "v8.17.0" write_mode: overwrite-delete-stale spec: connection_string: "${POSTGRESQL_CONNECTION_STRING}"
cloudquery upgrade check ./config.yml --source gcp --to v23.0.0
gcp v22.1.2 → v23.0.0 | postgresql (cloudquery/[email protected]) write_mode: overwrite-delete-stale | pk_mode: default SELECTED TABLES REMOVED — 2 removed tables Changes gcp_aiplatform_specialist_pools removed table, the new source version no longer provides it gcp_aiplatform_specialistpool_locations removed table, the new source version no longer provides it Action: remove explicit selections and update dependent consumers. This check only previews the changes. It does not migrate, write, delete or upload anything.

How the check gets the source tables

The check starts both source versions and asks each one for its tables. It never runs a sync of the source, and it reads no rows.

  • Sources with a fixed list of tables, such as AWS or Datadog, return their tables without a connection. The check does not connect to the service and does not use the source credentials.
  • Sources that find their tables in the service, such as PostgreSQL, MySQL, BigQuery or ClickHouse, return no tables without a connection. The check starts them again with their connection and asks for the tables again. This reads metadata only, such as table and column names and types. Coverage gaps shows tables were listed with a connection (metadata only, no rows read) for each version it connected.
  • Sources that find their tables only during a sync, such as S3, return no tables with or without a connection. Their report is UNKNOWN.

Because some sources connect, give the check the same source credentials and network access as a sync wherever it runs, including CI.

The check starts each destination without a connection. It does not connect to your database or storage.

Use the JSON output in CI

Add --output json to print the reports as JSON:

cloudquery upgrade check ./config.yml --source okta --to v7.0.0 --output json > report.json

The JSON goes to standard output. Download progress goes to standard error, so report.json contains only the JSON. The output has one entry in reports for each destination. For the Okta example:

{ "reports": [ { "source": { "name": "okta", "from_version": "v6.8.2", "to_version": "v7.0.0" }, "destination": { "name": "postgresql", "registry": "cloudquery", "path": "cloudquery/postgresql", "version": "v8.17.0", "write_mode": "overwrite-delete-stale", "pk_mode": "default", "migrate_mode": "safe" }, "verdict": "REVIEW REQUIRED", "summary": "1 table needs a manual migration, 1 new table", "tables": [ { "name": "okta_policy_mappings", "new_table": true, "outcomes": { "forced": { "result": "applied", "text": "created" }, "safe": { "result": "applied", "text": "created" } } }, { "name": "okta_policy_rules", "changes": [ { "kind": "added", "column": "policy_id", "new_type": "text", "new_source_type": "string", "reason": "part of the primary key" } ], "outcomes": { "forced": { "result": "data_loss", "text": "dropped and recreated, existing rows deleted" }, "safe": { "result": "fails", "text": "fails: safe mode cannot change a primary key" } } } ], "removed_tables": [], "output_comparisons": [], "coverage_gaps": [], "action": "migrate okta_policy_rules manually before upgrading, or switch to migrate_mode: forced and accept losing its rows." } ] }

The actions and conditions columns are left out of this example.

FieldContents
verdictThe result, with the same text as the text output. See Read the results.
summaryThe count of affected tables. Not present when no selected table changes.
tablesEach new or changed table. changes lists its columns: kind is added, removed or changed. outcomes gives the next sync result for safe and forced: applied, fails or data_loss. Unchanged tables are not listed.
removed_tablesThe selected tables that the new version no longer has.
output_comparisonsFor file destinations, the test values written with the old and the new schema. See Limits.
coverage_gapsThe same list as Coverage gaps in the text output.
actionWhat to do before you upgrade.
destinationnull when the check could not list the source tables. The output then has one report, with no destination and the verdict UNKNOWN.

Exit codes

The check prints the full report first, in text and JSON output, and then exits with one of these codes:

CodeMeaning
0No action needed. The result is AUTOMATICALLY MIGRATABLE, NO OUTPUT DIFFERENCE DETECTED for equivalent test values or No schema changes affect your selected tables.
1The check failed with an error, for example a configuration file that is not valid or an integration that does not download. Other cloudquery commands use the same code.
3Action needed. At least one table needs a manual migration or rebuild, a selected table is removed, or a file schema or output changes.
4Unknown. The check could not assess a source or a destination, or the report lists a Coverage gaps entry for a table. For example, the destination version does not support the check, or the check could not list the source tables.

A source that the check connects to only to list its tables does not make the code 4. The verdict shows the most urgent action, and the exit code shows whether the check is complete. If any table is unknown, the code is 4, even when the verdict is REVIEW REQUIRED. With several destinations, the check exits with the highest of 0, 3 and 4.

In CI, a step that runs the check fails on any code other than 0. Because the check writes the report before it exits, report.json from the JSON example is complete when the step fails.

Limits

  • The check assesses the destination schema, and the output that a file destination writes for a schema. It does not check your data, the live schema of your database, or whether your queries still work after the upgrade.
  • The check assumes that your destination tables match the current source version and the settings in your configuration file. If you created the tables with other settings, such as another write_mode, or changed them by hand, the results can be wrong.
  • For file destinations, the check writes test values with the old and the new schema and compares the output. output_comparisons in the JSON output, and Synthetic value, Before and After under Output comparison in the text output, show these test values. They are not your data. A file destination that writes the same output for the test values can still write a different output for your data.

Next steps

Was this page helpful?

Last updated on