# CloudQuery — Complete Content > CloudQuery is a cloud infrastructure platform for platform engineering and cloud operations teams. It syncs configuration data from 70+ sources (AWS, GCP, Azure, SaaS APIs) into your own database and provides Policies, Automations, and AI features on top of that data. This file combines content from all CloudQuery web properties. Auto-generated at build time. ## Content Sections (in order) 1. **Documentation** — Platform guides, CLI reference, core concepts 2. **Integration Hub** — Source, destination, and transformation integration docs 3. **Blog** — Product updates, tutorials, and engineering posts (last 6 months) 4. **Learning Center** — Guides and best practices --- Source: https://www.cloudquery.io/docs/cli/advanced/arrow-string-representation # How CloudQuery represents some Arrow types as strings Some types in Arrow are highly specialized and often there isn't a one to one mapping to a database type. For example, Arrow has a `Date32` type which is a 32-bit integer representing the number of days since the UNIX epoch. Most databases do not support this type, so CloudQuery has to represent it as a string. Here is a list of all Arrow types and their string representation. | Arrow Type | String Representation | Example | | ------------------------------------ | ----------------------------------------------------------------- | ---------------------------------------------- | | Binary, FixedSizeBinary, LargeBinary | base64 StdEncoding | `YQ==` | | Boolean | `strconv.FormatBool` output | "true" | | Decimal128, Decimal256 | GetOneForMarshal.(string) | "12345" | | List, FixedSizeList | string(GetOneForMarshal.(json.RawMessage)) | `[ 1, 2, 3 ]` | | MonthInterval | int32 as string | "123" | | DayTimeInterval | JSON {"days":int32, "milliseconds":int32} | `{"days":1, "milliseconds":234}` | | MonthDayNanoInterval | JSON {"months":int32, "days":int32, "nanoseconds": int64} | `{"months":1, "days":2, "nanoseconds": 34567}` | | Struct | JSON | `{"key": ["values", "value2"]}` | | String, LargeString | As is | "foo" | | Uint8, Uint16, Uint32, Uint64 | strconv.FormatUint | "123" | | Int8, Int16, Int32, Int64 | strconv.FormatInt | "-123" | | Float16 | GetOneForMarshal.(string) | "123.45" | | Float32, Float64 | strconv.FormatFloat | "123.45" | | Timestamp | `YYYY-MM-DD HH:mm:ss.SSSSSSSSS`, no timezone info | "2006-01-02 15:04:05.999999999" | | Time32 | `HH:mm:ss` or `HH:mm:ss.SSS`, depending on precision | "15:04:05.000" | | Time64 | `HH:mm:ss.SSSSSS` or `HH:mm:ss.SSSSSSSSS`, depending on precision | "15:04:05.000000" | | Date32, Date64 | `YYYY-MM-DD` | "2006-01-02" | | Duration | Numeric amount and time unit, concatenated | "12345ms" | | SparseUnion, DenseUnion | GetOneForMarshal.(string) | | Null values (in nested types) are represented as `(null)` ## CloudQuery Extension Types | Extension Type | Underlying Arrow Type | String Representation | Example | | -------------- | --------------------- | --------------------- | -------------------------------------- | | UUID | FixedSizeBinary(16) | Formatted UUID | "123e4567-e89b-12d3-a456-426614174000" | | MAC Address | Binary | Formatted MAC Address | "01:23:45:67:89:ab" | | Inet | String | As is | "192.168.1.0/24" | | JSON | String | As is | `{"key": "value"}` | ## Next Steps - [CloudQuery Types](/cli/core-concepts/cloudquery-types) - Learn about CloudQuery's custom Arrow extension types - [Creating a New Integration](/cli/integrations/creating-new-integration) - Build integrations using the type system --- Source: https://www.cloudquery.io/docs/cli/advanced/build # Building From Source The preferred way to use CloudQuery is through the available distribution, see the [Getting Started](/cli/getting-started) guide To build CloudQuery CLI from source, follow the steps: 1. CloudQuery is developed in Go. Ensure you have a working [Go runtime](https://go.dev/) 2. Fork and clone the CloudQuery repository. If you’re not sure how to do this, watch [these videos](https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github). 3. From the cloned repository root, change directory to `./cli` and run `go build -o cloudquery` to build the CloudQuery CLI. The binary will be created in the same directory. Building an integration from source is similar. Most integrations have a makefile in their directory to make this easier. For example, to build the `aws` integration, run `make build` from the `./plugins/source/aws` directory. The resulting binary can be used by providing the path to it as the `path` parameter in a [plugin configuration](/cli/integrations/sources), together with the `local` registry. Python integrations have `make build-docker` to build a Docker image that can be used with the `docker` registry. ## Next Steps - [Creating a New Integration](/cli/integrations/creating-new-integration) - Build a source integration from scratch - [Running Locally](/cli/advanced/running-locally) - Test your integration locally during development - [Publishing an Integration](/cli/integrations/creating-new-integration/publishing) - Share your integration on the hub --- Source: https://www.cloudquery.io/docs/cli/advanced/codegen # Generating resources Adding resources to an integration can sometimes be a tedious task, some resources can have more than hundreds of fields and relations, and adding them all can take a long time. To remedy this issue, we provide utilities as part of our [plugin-sdk](https://github.com/cloudquery/plugin-sdk) to automatically infer columns from Go structs. In particular, see the [`transformers.TransformWithStruct()` method](https://github.com/cloudquery/plugin-sdk/blob/main/transformers/struct.go). ## Next Steps - [Creating a New Integration in Go](/cli/integrations/creating-new-integration/go-source) - Full guide to building a Go source integration - [CloudQuery Types](/cli/core-concepts/cloudquery-types) - Understand the Arrow-based type system - [Build](/cli/advanced/build) - Build integration binaries --- Source: https://www.cloudquery.io/docs/cli/advanced/instrumenting-a-paid-integration # Instrumenting a Paid Integration This page is aimed at integration developers. CloudQuery integrations can be published as free, open-core or premium. For rows to be counted as paid in an open-core or premium integration, you will need to add some additional instrumentation code. Instrumenting a paid integration to check quotas and count the number of rows synced can be done using the [`github.com/cloudquery/plugin-sdk/v4/premium`](https://github.com/cloudquery/plugin-sdk/tree/main/premium) package. ## Steps 1. Ensure that the integration’s team, name and kind are passed in. For example: ```go var ( Name = "your-integration-name" // TODO: replace with your integration name Kind = "source" // TODO: replace with your integration kind (source / destination) Team = "your-team-name" // TODO: replace with your team name Version = "development" ) func Plugin() *plugin.Plugin { return plugin.NewPlugin( Name, Version, Configure, plugin.WithKind(Kind), plugin.WithTeam(Team), ) } ``` 2. Inside `resources/plugin/client.go`, add `usage premium.UsageClient` to the `Client` struct. 3. Instantiate the `premium.UsageClient` inside `Configure`: ```go uc, err := premium.NewUsageClient( opts.PluginMeta, premium.WithLogger(logger), ) if err != nil { return nil, fmt.Errorf("failed to initialize usage client: %w", err) } return &Client{ // ... usage: uc, // ... } ``` 4. Add the following methods to the `Client`: ```go // OnBeforeSend increases the usage count for every message. If some messages should not be counted, // they can be ignored here. func (c *Client) OnBeforeSend(_ context.Context, msg message.SyncMessage) (message.SyncMessage, error) { if c.usage == nil { return msg, nil } if si, ok := msg.(*message.SyncInsert); ok { if err := c.usage.Increase(uint32(si.Record.NumRows())); err != nil { return msg, fmt.Errorf("failed to increase usage: %w", err) } } return msg, nil } // OnSyncFinish is used to ensure the final usage count gets reported func (c *Client) OnSyncFinish(_ context.Context) error { if c.usage != nil { return c.usage.Close() } return nil } ``` 5. Inside the `Client` `Sync` method, create a new context using `premium.WithCancelOnQuotaExceeded`. This will do two things: 1. stop the sync from happening if the user has no remaining quota, and 2. periodically check that the user still has remaining quota, canceling the context if not. ```go newCtx, err := premium.WithCancelOnQuotaExceeded(ctx, c.usage) if err != nil { return fmt.Errorf("failed to configure quota monitor: %w", err) } return c.scheduler.Sync(newCtx, schedulerClient, tt, res, scheduler.WithSyncDeterministicCQID(options.DeterministicCQID)) ``` If there is a `stateClient` the above block should read: ```go newCtx, err := premium.WithCancelOnQuotaExceeded(ctx, c.usage) if err != nil { return fmt.Errorf("failed to configure quota monitor: %w", err) } if err := c.scheduler.Sync(newCtx, schedulerClient, tt, res, scheduler.WithSyncDeterministicCQID(options.DeterministicCQID)); err != nil { return fmt.Errorf("failed to sync: %w", err) } return stateClient.Flush(ctx) ``` 6. If all tables are paid: `return premium.MakeAllTablesPaid(tables)` in `getTables`. If only some tables are paid: add `isPaid: true` to the relevant Table definitions. ## Next Steps - [Publishing an Integration](/cli/integrations/creating-new-integration/publishing) - Publish your paid integration to the hub - [Using an Offline License](/cli/advanced/using-an-offline-license) - Support customers with air-gapped environments - [Creating a New Integration](/cli/integrations/creating-new-integration) - Build an integration from scratch --- Source: https://www.cloudquery.io/docs/cli/advanced/managing-incremental-tables # Managing Incremental Tables Incremental tables are tables that fetch only the data that changed since the last sync. Tables that support this mode are marked as "incremental" in integration table documentation. Learn more about [sync modes](/cli/core-concepts/syncs) and [schema migrations](/cli/managing-cloudquery/migrations). When a sync runs on an incremental table, the table will first fetch the last known cursor state from the state backend, then resume syncing from that point. Incremental tables guarantee at-least-once delivery, which means that there should never be gaps in the data as a result of the cursor state being used, but there may be duplicates. If the destination uses `overwrite` or `overwrite-delete-stale` write mode, these duplicates will be handled automatically. But if the destination uses `append` mode, you will need to either exclude the duplicates at query time or run a deduplication process. To resume from a previous position, incremental tables store some state, known as the **cursor**. When using the CloudQuery CLI, the cursor state is stored in a **backend**. Any destination integration can be used as a backend. This state destination is often the same destination data is being written to for the sync, but it doesn't need to be. In some destinations that support only `append` mode, each state update will generate a new row (with a new version) in the state table and may need to be cleaned up periodically. Destinations supporting `overwrite` mode don't have this issue. ## Example Configuration Add a `backend_options` property to the source configuration to enable the use of a state backend for incremental tables. It accepts the following properties: - `table_name` (`string`, required): The name of the table to store the cursor state in. - `connection` (`string`, required): The connection string to the destination integration. This is a gRPC connection string, not the connection string to the destination database itself. See below for more details. For example, the following configuration shows how to enable the use of a PostgreSQL destination as the state backend for the AWS integration: ```yaml kind: source spec: name: aws path: cloudquery/aws registry: cloudquery version: "VERSION_SOURCE_AWS" tables: ["aws_cloudtrail_events"] destinations: ["postgresql"] backend_options: table_name: "cq_state_aws" connection: "@@plugins.postgresql.connection" spec: # AWS integration specific configuration # ... --- kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" registry: "cloudquery" version: "VERSION_DESTINATION_POSTGRESQL" write_mode: "overwrite-delete-stale" spec: connection_string: "${CONNECTION_STRING}" ``` A special `@@plugins..` syntax is used to reference a property from another integration. In this case, the `connection` property from the PostgreSQL integration is being referenced. Here, `connection` refers to the gRPC connection to the destination integration, automatically inferred after the destination integration is started, not the connection string to the destination database itself. Sometimes it may be useful to use a different state backend than the destination. For example, if the destination is a data warehouse that only supports `append` mode, a separate database can be used as the state backend. For example, the following configuration writes AWS data to BigQuery, but stores state in a local SQLite database: ```yaml kind: source spec: name: aws path: cloudquery/aws registry: cloudquery version: "VERSION_SOURCE_AWS" tables: ["aws_cloudtrail_events"] destinations: ["bigquery"] backend_options: table_name: "cq_state_aws" connection: "@@plugins.sqlite.connection" spec: # AWS integration specific configuration # ... --- kind: destination spec: name: bigquery path: cloudquery/bigquery registry: cloudquery version: "VERSION_DESTINATION_BIGQUERY" write_mode: "append" spec: project_id: ${PROJECT_ID} dataset_id: ${DATASET_ID} --- kind: destination spec: name: sqlite path: cloudquery/sqlite registry: cloudquery version: "VERSION_DESTINATION_SQLITE" spec: connection_string: ./db.sqlite ``` ## Next Steps - [Syncs](/cli/core-concepts/syncs) - Understand full vs incremental sync modes - [Performance Tuning](/cli/advanced/performance-tuning) - Optimize sync speed and resource usage - [Source Integrations](/cli/integrations/sources) - Configure source integrations with backend options --- Source: https://www.cloudquery.io/docs/cli/advanced/managing-versions # Managing Versions ## CLI Versions ### Downloading the CLI All CloudQuery CLI versions are available for download on the [releases page](https://github.com/cloudquery/cloudquery/releases?q=cli-&expanded=true). ### Homebrew To update the CLI via Homebrew, run: ```bash brew upgrade cloudquery/tap/cloudquery ``` To install a specific version of the CLI, run: ```bash brew install cloudquery/tap/cloudquery@ ``` (e.g. `brew install cloudquery/tap/cloudquery@2.3.10`) ## Integration Versions CloudQuery integrations are versioned independently of the CLI. Releases happen on a weekly schedule, using semantic versioning to indicate breaking schema changes as described in [Source Integration Release Stages](#source-integration-release-stages). We recommend pinning integration versions to avoid unexpected changes to your data model, and only upgrading to new versions when you need to take advantage of new features or bug fixes. That said, if you are okay with the risk of breaking changes (or able to use `migrate_mode: forced`), [this how-to guide](https://www.cloudquery.io/blog/update-plugins-using-renovate) describes how to keep integration versions up-to-date automatically using Renovate. In all cases, we recommend performing upgrades in a staging environment first before applying them to production. ### Semantic Versioning For version `Major.Minor.Patch`: - `Major` is incremented when there are breaking changes (e.g. breaking configuration spec structure, column type changes). - `Minor` is incremented when we add features in a backwards compatible way. - `Patch` is incremented when we fix bugs in a backwards compatible way. ## Source Integration Release Stages [Official source integrations](https://www.cloudquery.io/hub/plugins/source?authors=official) go through two release stages: Preview and Generally Available (GA). Both Preview and GA integrations follow [semantic versioning](#semantic-versioning). The main differences between the two stages are: 1. Preview integrations are experimental and may have frequent breaking changes. 2. Preview integrations might get deprecated due to lack of usage. 3. Premium integrations in Preview are free to use. 4. Long Term Support and bug fixes are only guaranteed for integrations that are Generally Available. ## Next Steps - [Source Integrations](/cli/integrations/sources) - Configure source integration versions - [Destination Integrations](/cli/integrations/destinations) - Configure destination integration versions - [Schema Migrations](/cli/managing-cloudquery/migrations) - Understand how version changes affect schemas --- Source: https://www.cloudquery.io/docs/cli/advanced/performance-tuning # Performance Tuning Improve the performance of `cloudquery sync` for large cloud estates with these techniques. For related topics, see [running syncs in parallel](/cli/managing-cloudquery/running-in-parallel) and [managing incremental tables](/cli/advanced/managing-incremental-tables). ## Identifying Slow Tables The first step in improving the performance of a sync is to identify which tables are taking the longest to sync. To identify slow tables, run the `cloudquery sync` command with the `--tables-metrics-location` flag. This flag takes a path to a file where the metrics will be written. The metrics are displayed as a human-readable table and refreshed as the sync progresses. To use this feature run `cloudquery sync --tables-metrics-location metrics.txt` and open the file `metrics.txt` in a text editor that supports live updates. Alternatively, you can use `watch cat metrics.txt` to monitor the file in real-time. You can see an example output below. Tables still in progress appear first with a `N/A` in the `END TIME` column, and the rest sorted by table name. The `CLIENT ID` column differs between integrations and can be used to identify which account, region, project, location, etc. is being synced (the terms differ between integrations). ```text +-----------------------------------------------------+-----------+-----------+--------+--------+ | TABLE | DURATION | RESOURCES | ERRORS | PANICS | +-----------------------------------------------------+-----------+-----------+--------+--------+ | gcp_compute_addresses | 3.151s | 8 | 0 | 0 | | gcp_compute_autoscalers | 3.289s | 0 | 0 | 0 | | gcp_compute_backend_buckets | 3.427s | 2 | 0 | 0 | | gcp_compute_backend_services | 3.601s | 18 | 0 | 0 | | gcp_compute_disk_types | 4.442s | 2682 | 0 | 0 | | gcp_compute_disks | 3.614s | 11 | 0 | 0 | | gcp_compute_external_vpn_gateways | 3.593s | 0 | 0 | 0 | | gcp_compute_firewalls | 3.658s | 10 | 0 | 0 | | gcp_compute_forwarding_rules | 3.961s | 3 | 0 | 0 | | gcp_compute_global_addresses | 3.928s | 6 | 0 | 0 | | gcp_compute_health_checks | 3.762s | 2 | 0 | 0 | | gcp_compute_images | 3.638s | 2 | 0 | 0 | | gcp_compute_instance_group_instances | 9.491s | 3 | 0 | 0 | | gcp_compute_instance_group_managers | 3.8s | 3 | 0 | 0 | | gcp_compute_instance_group_regional_instances | 9.567s | 0 | 0 | 0 | | gcp_compute_instance_groups | 15.042s | 3 | 0 | 0 | | gcp_compute_instance_tag_bindings | 10.4s | 0 | 0 | 0 | | gcp_compute_instances | 14.903s | 5 | 0 | 0 | | gcp_compute_interconnect_attachments | 2m21.925s | 0 | 0 | 0 | | gcp_compute_interconnect_locations | 0s | 200 | 0 | 0 | | gcp_compute_interconnect_remote_locations | 3.894s | 148 | 0 | 0 | | gcp_compute_interconnects | 3.903s | 0 | 0 | 0 | | gcp_compute_machine_types | 2m10.787s | 7390 | 0 | 0 | | gcp_compute_network_endpoint_groups | 4.121s | 74 | 0 | 0 | | gcp_compute_networks | 4.072s | 3 | 0 | 0 | | gcp_compute_osconfig_inventories | 2m12.575s | 4 | 0 | 0 | | gcp_compute_osconfig_os_patch_deployments | 4.057s | 1 | 0 | 0 | | gcp_compute_osconfig_os_patch_jobs | 14.06s | 6 | 0 | 0 | | gcp_compute_osconfig_os_patch_jobs_instance_details | 10.296s | 0 | 0 | 0 | | gcp_compute_osconfig_os_policy_assignment_reports | 2m13.269s | 0 | 0 | 0 | | gcp_compute_osconfig_os_policy_assignments | 2m12.514s | 3 | 0 | 0 | | gcp_compute_osconfig_os_vulnerability_reports | 2m9.722s | 4 | 0 | 0 | | gcp_compute_packet_mirrorings | 4.303s | 0 | 0 | 0 | | gcp_compute_projects | 4.482s | 2 | 0 | 0 | | gcp_compute_router_nat_mapping_infos | 901ms | 3 | 0 | 0 | | gcp_compute_routers | 6.805s | 1 | 0 | 0 | | gcp_compute_routes | 4.407s | 103 | 0 | 0 | | gcp_compute_security_policies | 4.741s | 1 | 0 | 0 | | gcp_compute_snapshots | 4.51s | 34 | 0 | 0 | | gcp_compute_ssl_certificates | 4.648s | 4 | 0 | 0 | | gcp_compute_ssl_policies | 4.794s | 3 | 0 | 0 | | gcp_compute_subnetworks | 5.404s | 90 | 0 | 0 | | gcp_compute_target_grpc_proxies | 5.531s | 0 | 0 | 0 | | gcp_compute_target_http_proxies | 5.071s | 1 | 0 | 0 | | gcp_compute_target_https_proxies | 5.69s | 4 | 0 | 0 | | gcp_compute_target_instances | 5.207s | 0 | 0 | 0 | | gcp_compute_target_pools | 5.557s | 0 | 0 | 0 | | gcp_compute_target_ssl_proxies | 5.839s | 0 | 0 | 0 | | gcp_compute_target_tcp_proxies | 5.722s | 0 | 0 | 0 | | gcp_compute_target_vpn_gateways | 6.011s | 0 | 0 | 0 | | gcp_compute_url_maps | 6.097s | 5 | 0 | 0 | | gcp_compute_vpn_gateways | 6.192s | 0 | 0 | 0 | | gcp_compute_vpn_tunnels | 6.194s | 0 | 0 | 0 | | gcp_compute_zones | 0s | 200 | 0 | 0 | +-----------------------------------------------------+-----------+-----------+--------+--------+ ``` This feature is available starting from CLI version [v5.25.0](https://github.com/cloudquery/cloudquery/releases/tag/cli-v5.25.0) and integrations released on July 10th 2024 or later. When debugging performance across distributed syncs, use the `--invocation-id` flag to correlate CLI traces with integration traces in your [OpenTelemetry monitoring setup](/cli/managing-cloudquery/monitoring/overview#cli-level-opentelemetry). ## Use Wildcard Matching Sometimes the easiest way to improve the performance of the `sync` command is to limit the number of tables that get synced. The `tables` and `skip_tables` source configuration options both support wildcard matching. This means that you can use `*` anywhere in a name to match multiple tables. For example, when using the `aws` source integration, it is possible to use a wildcard pattern to match all tables related to AWS EC2: ```yaml copy tables: - aws_ec2_* ``` This can also be combined with `skip_tables`. For example, to include all EC2 tables but not EBS-related ones: ```yaml copy tables: - "aws_ec2_*" skip_tables: - "aws_ec2_ebs_*" ``` The CloudQuery CLI will warn if a wildcard pattern does not match any known tables. ## Tune Concurrency The `concurrency` setting, available for all source integrations as part of the [source spec](/cli/integrations/sources#concurrency), controls the approximate number of concurrent requests that will be made while performing a sync. Setting this to a low number will reduce the number of concurrent requests, reducing the memory used and making the sync less likely to hit rate limits. The trade-off is that syncs will take longer to complete. Learn more about [sync modes](/cli/core-concepts/syncs) and [running in parallel](/cli/managing-cloudquery/running-in-parallel). ## Adjust Batch Size Most destination integrations have batching related settings that can be adjusted to improve performance. Tuning these can improve performance, but it can also increase the memory usage of the sync process. Here are the batching related settings you will come across: - `batch_size`: The number of rows that are inserted into the destination at once. The default value for this setting is usually between 1000 to 10000 rows, depending on the destination integration. - `batch_size_bytes`: Maximum size of items that may be grouped together to be written in a single write. This is useful for limiting the memory usage of the sync process. The default value for this varies between 4 MB to 100 MB, depending on the destination integration. - `batch_timeout`: Maximum interval between batch writes. Even if data stops coming in, the batch will be written after this interval. The default value for this setting is usually between 10 seconds and 1 minute, depending on the destination integration. Some destination integrations (such as file or S3 destinations) start a new object or file for every batch, and some buffer the data in memory to be written at once. You should check the documentation for the destination integration you are using to see what the default values are and consider how they can be adjusted to suit your use case. Here's a conservative example for the PostgreSQL destination integration that reduces the overall memory usage, but may also increase the time it takes to sync: ```yaml kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" registry: "cloudquery" version: "VERSION_DESTINATION_POSTGRESQL" spec: connection_string: "postgres://user:pass@localhost:5432/mydb?sslmode=disable" # replace with your connection string batch_size: 10000 # 10000 rows, default batch_size_bytes: 4194304 # 4 MB, dramatically tuned down from the 100 MB default batch_timeout: "30s" # 30 seconds, tuned down from 60 seconds ``` With this configuration, the PostgreSQL destination integration will write 10,000 rows at a time, or 4 MB of data at a time, or every 30 seconds, whichever comes first. ## Use a Different Scheduler By default, CloudQuery syncs will fetch all tables in parallel, writing data to the destination(s) as they come in. However, the `concurrency` setting, mentioned above, places a limit on how many **table-clients** can be synced at a time. What "table-client" means depends on the source integration and the table. In AWS, for example, a client is usually a combination of account and region. Get all the combinations of accounts and regions for all tables, and you have all the table-clients for a sync. For the GCP source integration, clients generally map to projects. The default CloudQuery scheduler, known as `dfs`, will sync up to `concurrency / 100` table-clients at a time (we are ignoring child relations for the purposes of this discussion). Take an example GCP cloud estate with 5000 projects, syncing 100 tables. This makes for approximately 500,000 table-client pairs, and a concurrency of 10,000 will allow 100 table-client pairs to be synced at a time. The `dfs` scheduler will start with the first table and its first 100 projects, and then move on to finish all projects for that table before moving on to the next table. This means, in practice, only one table is really being synced at a time! Usually this works out fine, as long as the cloud platform's rate limits are aligned with the clients. But if rate limits are applied per-table, rather than per-project, `dfs` can be suboptimal. A better strategy in this case would be to choose the first client for every table before moving on to the next client. This is what the `round-robin` scheduler does. Only some integrations support this setting. The following example configuration enables `round-robin` scheduling for the GCP source integration: ```yaml kind: source spec: name: "gcp" path: "cloudquery/gcp" registry: "cloudquery" version: "VERSION_SOURCE_GCP" tables: ["gcp_storage_*", "gcp_compute_*"] destinations: ["postgresql"] spec: scheduler: "round-robin" project_ids: ... ``` Finally, the `shuffle` strategy aims to provide a balance between `dfs` and `round-robin` by randomizing the order in which table-client pairs are chosen. The following example enables `shuffle` for the GCP integration, which can help reduce the likelihood of hitting rate limits by randomly mixing the underlying services to which API calls that are made concurrently, rather than hitting a single API with all calls at once: ```yaml kind: source spec: name: "gcp" path: "cloudquery/gcp" registry: "cloudquery" version: "VERSION_SOURCE_GCP" tables: ["gcp_storage_*", "gcp_compute_*"] destinations: ["postgresql"] spec: project_ids: ... scheduler: "shuffle" # ... ``` The `shuffle` scheduler is the **default** for the AWS source integration. ## Avoid `skip_dependent_tables: false` Starting with version [v6.0.0](https://github.com/cloudquery/cloudquery/releases/tag/cli-v6.0.0) of the CloudQuery CLI `skip_dependent_tables` is set to `true` by default, to avoid new tables implicitly being synced when added to integrations. This can be overridden by setting `skip_dependent_tables: false` in the source configuration. When setting `skip_dependent_tables: false`, all tables that depend on other tables will be synced by default. When syncing dependent tables multiple API calls need to be made for every row in the parent table. This can lead to thousands of API calls, increasing the time it takes to sync. Consider three tables: `A`, `B` and `C`. `A` is the top-level table. `B` depends on it, and `C` depends on `B`: ```text copy A ↳ B ↳ C ``` By default only `A` will be synced. If you set `skip_dependent_tables: false`, `B` and `C` will also be synced. This can be a problem if `A` has a large number of rows, as it will result in a large number of API calls to sync `B` and `C`. To avoid setting `skip_dependent_tables: false` and still get dependent tables synced, you can explicitly list the dependent tables in the source configuration, or [use wildcard matching](#use-wildcard-matching). ## Next Steps - [Managing Incremental Tables](/cli/advanced/managing-incremental-tables) - Use incremental syncs for faster data extraction - [Running Syncs in Parallel](/cli/managing-cloudquery/running-in-parallel) - Distribute workloads across multiple processes - [Monitoring](/cli/managing-cloudquery/monitoring/overview) - Set up observability to track sync performance - [Deployment Options](/cli/managing-cloudquery/deployments/overview) - Choose the right production environment for your syncs --- Source: https://www.cloudquery.io/docs/cli/advanced/publishing-an-addon-to-the-hub # Publishing an Addon to the CloudQuery Hub With the announcement of [CloudQuery Hub](https://www.cloudquery.io/blog/announcing-cloudquery-new-hub), we are excited to see the community contribute addons, such as transformations and dashboards, to the Hub. Publish an addon to the Hub by following these steps. ## Prerequisites - You have created a [CloudQuery Platform](https://cloud.cloudquery.io/) account and completed the onboarding process to create a team - You have created the addon you'd like to publish on under the relevant team - You have the [CloudQuery CLI](/cli/getting-started) installed (version >= `v3.27.1`) - You are authenticated to [CloudQuery Platform](https://cloud.cloudquery.io/) using the `cloudquery login` command or an API key ## Publishing an Addon 1. (Optional, recommended) In the root directory of your addon repository run `git tag v1.0.0` to tag the version you're about to publish (replace `v1.0.0` with the version you'd like to publish) 2. (Optional, recommended) Run `git push origin v1.0.0` to push the tag 3. Create a `manifest.json` file that describes the addon, the path to a zip containing its files and the path to its documentation and changelog in markdown format. Here is an example: ```json copy filename="manifest.json" { "schema_version": 1, "type": "addon", "team_name": "my_team", "addon_name": "example", "addon_type": "visualization", "addon_format": "zip", "message": "./changelog.md", "doc": "./readme.md", "path": "./test.zip", "plugin_deps": ["cloudquery/source/test@v1.0.0"], "addon_deps": [] } ``` The `plugin_deps` field describes the plugins that this addon depends on, if any. The format is `//@`. Similarly, the `addon_deps` field describes the addons that this addon depends on, if any. The format is `//@`. 4. Run `cloudquery addon publish /path/to/manifest.json v1.0.0` to publish a draft version of the addon (replacing `v1.0.0` with the version you want to publish). The version will show up under the versions tab of your addon in . As long as the version is in draft it's mutable and you can re-package the addon and publish it again. 5. Once you're ready, run `cloudquery addon publish /path/to/manifest.json v1.0.0 --finalize` to publish a non-draft version of the addon. This version will be immutable and will show up in [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source). Allow up to 1 hour for the Hub to reflect the changes, and Allow time for the CloudQuery team to review the addon before it's published. ## Next Steps - [Publishing an Integration](/cli/integrations/creating-new-integration/publishing) - Publish source and destination integrations - [Transformations](/cli/core-concepts/transformations) - Learn about official dbt and SQL transformations - [Dashboards & Visualizations](/cli/core-concepts/dashboards) - Use Grafana dashboards from the hub --- Source: https://www.cloudquery.io/docs/cli/advanced/running-locally # Running CloudQuery Integrations Locally CloudQuery CLI normally invokes integrations as separate processes. However, for development purposes, it is possible to run integrations directly from the command line. You can run a single integration or multiple integrations locally. It's also possible to have some integrations (from other registries such as `github`) managed and run by the CloudQuery CLI, and have some running locally simultaneously. ## Required Settings You can run an integration locally yourself and tell the CLI to connect to it, or you can tell the CLI to run the integration locally from a filesystem location for you. ### Getting the CLI to run your binary Set the `registry` in the spec file to be `local`. `path` then becomes file path to the local binary: ```yaml copy kind: source spec: name: "cloudwidgets" registry: "local" path: "/home/user/path/to/plugin/binary" tables: ['*'] # other settings like tables, etc. ``` In this mode, the CLI will run the binary for you and connect to it. ### Running the integration yourself This is useful if you want to run the integration in a debugger, or if you want to run the integration in a different way than the CLI would run it. First of all, run your integration with the `serve` argument: ```bash copy /path/to/plugin serve ``` If you are running multiple integrations this way simultaneously, you will need to specify a different port for each one. You can do this with the `--address` flag: ```bash copy /path/to/plugin serve --address localhost:7778 ``` > `1:16PM INF Source plugin server listening address=127.0.0.1:7778` After the integration is running, you can tell the CLI to connect to it by setting the `registry` to `grpc` and the `path` to the listen address of the integration: ```yaml copy kind: source spec: name: "cloudwidgets" registry: "grpc" path: "localhost:7778" tables: ['*'] # other settings like tables, etc. ``` When you run CloudQuery CLI with this configuration, it connects to the integration as specified. ## Next Steps - [Creating a New Integration](/cli/integrations/creating-new-integration) - Build a custom integration - [Publishing an Integration](/cli/integrations/creating-new-integration/publishing) - Share your integration on the hub - [Configuration Guide](/cli/core-concepts/configuration) - Configure source and destination integrations --- Source: https://www.cloudquery.io/docs/cli/advanced/telemetry # Telemetry CloudQuery CLI and integrations collect both anonymous and identifiable usage data. We use the anonymous and identifiable usage data to help us improve the product, while we only use the identifiable information for billing. This page describes what data is collected and how to control it. ## Identifiable Data As part of the commercial offering, Premium CloudQuery integrations periodically send data to the CloudQuery licensing server to validate that the user or team has a valid subscription. The data in these requests include: - `plugin_name`: the full name of the integration (including the team that owns the integration) - `plugin_kind`: whether the integration is a source or destination integration - `table_name`: the name of the table being synced. This is only sent for sources whose tables are statically defined. Any source that has dynamic table names (for example all database sources) will not send this field. - `resource_count`: the number of rows being synced You cannot disable this data collection; the commercial offering requires it. If you have a use case that requires this data not be sent, please contact the [Sales team](https://www.cloudquery.io/contact-us) to discuss your requirements. ## Telemetry Data By default, the CloudQuery CLI collects usage data. There are two types of data we collect: **errors** and **stats**. These are described below. The [Controlling what is sent](#controlling-what-is-sent) section describes how to control what you send to CloudQuery. ### Errors Errors are stack traces sent whenever a panic occurs in the CLI or an official integration. Having this data allows us to be notified when there is a bug that needs to be prioritized. ### Stats #### Anonymous Stats Anonymous stats are numbers about the sync that was performed, such as the number of errors and number of resources fetched, as well as the integration versions used. These are sent at the end of a sync. They contain no identifying information. We use this data to understand which integrations are being used and how much, which helps guide our roadmap and development efforts. We also anonymously track commands that run, and if they result in an error. This helps us understand how the CLI is being used and how many errors are being encountered. #### Identifiable Stats Identifiable stats are the same as anonymous stats, but they also include a unique identifier for the user or team. Identifiable stats are collected when the CLI is authenticated via `cloudquery login` or using an [API key](/cli/managing-cloudquery/deployments/generate-api-key). We track the following user event types, `Login`, `Sync Started` and `Sync Finished`. We use this information to understand usage patterns, which helps us improve the product and our support. ## Controlling what is sent The CLI supports two methods of controlling the telemetry that gets sent to CloudQuery. ### Command-line Flag The `--telemetry-level` can be passed to the `cloudquery` CLI. It supports four options: `none`, `stats`, `errors`, `all` (default). ### Environment Variable A `CQ_TELEMETRY_LEVEL` environment variable can also be used to control the telemetry being sent. It supports the same options as the `--telemetry-level` flag. > **Deprecated:** The `CQ_NO_TELEMETRY` environment variable is deprecated. If you are using it, switch to `CQ_TELEMETRY_LEVEL=none` instead. Setting `CQ_NO_TELEMETRY` to any value currently behaves the same as `CQ_TELEMETRY_LEVEL=none`, but it will be removed in a future release. ## More Information See the [`cloudquery` command reference](/cli/cli-reference/cloudquery) for all available command-line options. ## Next Steps - [Monitoring](/cli/managing-cloudquery/monitoring/overview) - Set up OpenTelemetry observability for syncs - [Performance Tuning](/cli/advanced/performance-tuning) - Optimize sync speed and resource usage - [Security](/cli/managing-cloudquery/security) - Learn about CloudQuery's security practices --- Source: https://www.cloudquery.io/docs/cli/advanced/using-an-offline-license # Offline Licensing If you're using CloudQuery in an environment that doesn't have internet access, you can use an offline license with CloudQuery. Obtain and configure an offline license with these steps. ## Obtaining an Offline License To obtain an offline license, you'll need to contact our sales team by filling out the form in our [pricing page](https://www.cloudquery.io/pricing). The license will be assigned to your organization and will be valid for a specific period of time. If you need to use CloudQuery in a different organization or after the license has expired, you'll need to obtain a new license. ## Offline License File Once you've obtained an offline license, you'll receive a file with a `.cqlicense` extension. Using this file you will be able to run [cloudquery sync](/cli/cli-reference/cloudquery_sync) and [cloudquery migrate](/cli/cli-reference/cloudquery_migrate) commands without a connection to the CloudQuery API. ## Using the Offline License To use the offline license, you'll need to place the `.cqlicense` file (or files, as you may have more than one) into a common directory. Then, when running `migrate` or `sync` commands, include the `--license` flag with the path to the directory containing the license files, or directly point it to a single license file. For example: ```bash mv mycompany.cqlicense /path/to/license/directory/ cloudquery sync --license /path/to/license/directory ./aws.yml ./pg.yml ``` or: ```bash mv mycompany.cqlicense ~ cloudquery sync --license ~/mycompany.cqlicense ./aws.yml ./pg.yml ``` ## Limitations of Using an Offline License The offline license may be used only for [sync](/cli/cli-reference/cloudquery_sync) and [migrate](/cli/cli-reference/cloudquery_migrate) commands. If you are setting up a new environment, you will need to have the integrations downloaded into a `.cq` directory: automatic integration downloads will **not** work with an offline license. You will need to enable internet access and [login](/cli/cli-reference/cloudquery_login) (or, [generate](/cli/managing-cloudquery/deployments/generate-api-key) and use an API key) and run `cloudquery plugin install` manually to install the integrations first. You may also just run a `sync` or `migrate` which will download the integrations, and then you can use the offline license for subsequent runs. ## Inspecting the License File The license file is a JSON object and will contain a `license` field, which is base64 encoded data, and a `signature` field to ensure the authenticity of the file contents. You can check the details of the license file using this command (requires the `jq` tool to decode JSON): ```bash cat mycompany.cqlicense | jq -r .license | base64 -d | jq . ``` The result will look like this: ```json { "licensed_to": "Your Company Name", "plugins": [ "cloudquery/*" ], "issued_at": "2024-03-06T12:00:00Z", "valid_from": "2024-03-06T12:00:00Z", "expires_at": "2024-09-06T12:00:00Z" } ``` If you have any questions about the contents of the license file, contact our support team. ## Next Steps - [Instrumenting a Paid Integration](/cli/advanced/instrumenting-a-paid-integration) - Add licensing to your integration - [Generate API Key](/cli/managing-cloudquery/deployments/generate-api-key) - Create API keys for authentication - [Deployment Options](/cli/managing-cloudquery/deployments/overview) - Deploy CloudQuery in production environments --- Source: https://www.cloudquery.io/docs/cli/advanced/using-cloud-query-docker-registry-integrations-inside-a-containerized-environment # Using CloudQuery Docker Registry Integrations Inside a Containerized Environment CloudQuery CLI uses the [Docker CLI](https://docs.docker.com/engine/reference/commandline/cli/) and [Engine API](https://docs.docker.com/engine/api/) to run Docker integrations. When using the CloudQuery CLI Docker image, Docker integrations don't work out of the box, as the Docker CLI and Engine API are not available in the container by default. Run Docker integrations with the CloudQuery CLI Docker image using Docker Compose. ## Prerequisites - [Docker installed (with Docker Compose)](https://docs.docker.com/get-docker/) - [CloudQuery CLI](/cli/getting-started) ## Setup 1. Run `cloudquery login` to authenticate with the CloudQuery registry. 2. Create a file named `spec.yml` with the configuration for the Docker integration. We will use this spec file to pull the Docker integration image locally from the private CloudQuery Docker registry. The `airtable` and `postgresql` integrations are used as an example. ```yaml filename="spec.yml" kind: source spec: name: airtable path: cloudquery/airtable version: VERSION_SOURCE_AIRTABLE tables: ["*"] destinations: ["postgresql"] --- kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" version: "VERSION_DESTINATION_POSTGRESQL" ``` 3. Run `cloudquery plugin install spec.yml` to pull the Docker integration image locally. ## Running a Sync 1. [Create a CloudQuery API Key](/cli/managing-cloudquery/deployments/generate-api-key) to be used with the Docker Compose file. 2. Create a `docker-compose.yml` file with the following content. The file configures the CLI docker image, the Docker integration image, a PostgreSQL database and a configuration spec that sets up a connection between the CLI and the Docker integration using gRPC. ```yaml filename="docker-compose.yml" version: '3.1' services: cli: container_name: cli image: ghcr.io/cloudquery/cloudquery:latest command: ["sync", "/spec.yml", "--log-console", "--log-format", "json"] environment: CLOUDQUERY_API_KEY: ${CLOUDQUERY_API_KEY} # We reference this environment variable in the `spec.yml` config block below # Other plugins will require other environment variables AIRTABLE_ACCESS_TOKEN: ${AIRTABLE_ACCESS_TOKEN} configs: - spec.yml depends_on: airtable: condition: service_healthy postgres: condition: service_healthy airtable: container_name: airtable image: docker.cloudquery.io/cloudquery/source-airtable:VERSION_SOURCE_AIRTABLE # We use `cloudquery login` and `cloudquery plugin install spec.yml` to pull the image locally pull_policy: never restart: always healthcheck: # Docker plugins always run on port 7777 test: ["CMD", "bash", "-c", "echo -n '' > /dev/tcp/localhost/7777"] interval: 5s timeout: 30s retries: 5 postgres: container_name: postgres image: postgres:15 restart: always environment: POSTGRES_PASSWORD: pass ports: - "5432:5432" healthcheck: test: ["CMD", "pg_isready", "-U", "postgres"] interval: 5s timeout: 30s retries: 5 configs: spec.yml: content: | kind: source spec: name: airtable registry: grpc # Notice we use the container name as the host to connect via Docker internal DNS path: airtable:7777 tables: ["*"] destinations: ["postgresql"] spec: access_token: "${AIRTABLE_ACCESS_TOKEN}" --- kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" version: "VERSION_DESTINATION_POSTGRESQL" spec: # Notice we use the container name as the host to connect via Docker internal DNS connection_string: "postgresql://postgres:pass@postgres:5432/postgres?sslmode=disable" ``` 3. Run `CLOUDQUERY_API_KEY= AIRTABLE_ACCESS_TOKEN= docker compose up -d` 4. You can check the logs of the CLI container to see the sync process. Run `docker logs -f cli` or `docker logs -f airtable` to see the logs. 5. To see the results, you can connect to the PostgreSQL database using your favorite client, for example, `psql`. ## Cleanup Run `docker compose down` to stop and remove the containers. ## Next Steps - [Docker Deployment](/cli/managing-cloudquery/deployments/docker) - Deploy CloudQuery with Docker - [Docker Offline Installation](/cli/managing-cloudquery/deployments/docker-offline-installation) - Install Docker images without internet access - [Source Integrations](/cli/integrations/sources) - Configure the Docker registry for source integrations --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery # cloudquery CloudQuery CLI ## Synopsis CloudQuery CLI High performance data integration at scale. Find more information at: https://www.cloudquery.io ## Options ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") -h, --help help for cloudquery --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery addon](/cli/cli-reference/cloudquery_addon) - Addon commands * [cloudquery init](/cli/cli-reference/cloudquery_init) - Generate a configuration file for a sync * [cloudquery login](/cli/cli-reference/cloudquery_login) - Login to CloudQuery Hub. * [cloudquery logout](/cli/cli-reference/cloudquery_logout) - Log out of CloudQuery Hub. * [cloudquery migrate](/cli/cli-reference/cloudquery_migrate) - Update schema of your destinations based on the latest changes in sources from your configuration * [cloudquery plugin](/cli/cli-reference/cloudquery_plugin) - Plugin commands * [cloudquery switch](/cli/cli-reference/cloudquery_switch) - Switches between teams. * [cloudquery sync](/cli/cli-reference/cloudquery_sync) - Sync resources from configured source plugins to destinations * [cloudquery tables](/cli/cli-reference/cloudquery_tables) - Generate documentation for all supported tables of source plugins specified in the spec(s) * [cloudquery test-connection](/cli/cli-reference/cloudquery_test-connection) - Test plugin connections to sources and/or destinations * [cloudquery validate-config](/cli/cli-reference/cloudquery_validate-config) - Validate config - [Getting Started](/cli/getting-started) - Install and run your first sync - [Configuration Guide](/cli/core-concepts/configuration) - Configure source and destination integrations --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_addon # cloudquery addon Addon commands ## Options ``` -h, --help help for addon ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery](/cli/cli-reference/cloudquery) - CloudQuery CLI * [cloudquery addon download](/cli/cli-reference/cloudquery_addon_download) - Download addon from CloudQuery Hub. * [cloudquery addon publish](/cli/cli-reference/cloudquery_addon_publish) - Publish to CloudQuery Hub. - [Transformations](/cli/core-concepts/transformations) - Official dbt and SQL transformations - [Dashboards & Visualizations](/cli/core-concepts/dashboards) - Grafana dashboards from the hub --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_addon_download # cloudquery addon download Download addon from CloudQuery Hub. ## Synopsis Download addon from CloudQuery Hub. This downloads an addon from CloudQuery Hub to local disk. ``` cloudquery addon download addon-team/addon-type/addon-name@v1.0.0 [-t directory] [flags] ``` ## Examples ``` # Download an addon to local disk cloudquery addon download //@v1.0.0 # Further example cloudquery addon download cloudquery/transformation/aws-compliance-premium@v1.9.0 ``` ## Options ``` -h, --help help for download -t, --target string Download to specified directory. Use - for stdout (default ".") ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery addon](/cli/cli-reference/cloudquery_addon) - Addon commands - [Transformations](/cli/core-concepts/transformations) - Available transformations and policies - [Dashboards & Visualizations](/cli/core-concepts/dashboards) - Grafana dashboards from the hub --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_addon_publish # cloudquery addon publish Publish to CloudQuery Hub. ## Synopsis Publish to CloudQuery Hub. This publishes an addon version to CloudQuery Hub from a manifest file. ``` cloudquery addon publish manifest.json v1.0.0 [--finalize] [flags] ``` ## Examples ``` # Publish an addon version from a manifest file cloudquery addon publish /path/to/manifest.json v1.0.0 ``` ## Options ``` -f, --finalize Finalize the addon version after publishing. If false, the addon version will be marked as draft. -h, --help help for publish ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery addon](/cli/cli-reference/cloudquery_addon) - Addon commands - [Publishing an Addon](/cli/advanced/publishing-an-addon-to-the-hub) - Full addon publishing guide - [Transformations](/cli/core-concepts/transformations) - Official transformations and policies --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_init # cloudquery init Generate a configuration file for a sync ## Synopsis Generate a configuration file for a sync ## Modes The `init` command operates in one of three modes depending on your authentication state and flags: **AI-assisted mode** (default when logged in) Activates when you are logged in to a team (`cloudquery login`) and don't specify `--source` or `--destination`. Launches an interactive AI chat session that walks you through the setup process — selecting integrations, generating YAML configuration files, testing connections, and giving you some example queries. Type `exit` or `quit` to end the conversation. Use `--resume-conversation` to continue a previous session instead of starting a new one. **Basic interactive mode** Activates when you pass `--disable-ai`, or as a fallback if the AI assistant is unavailable. Presents a searchable picker to select source and destination integrations, then generates a configuration file from their default templates. **Non-interactive mode** Activates when both `--source` and `--destination` are specified. Generates the configuration file directly without prompts. Authentication via `cloudquery login` is required for AI-assisted and basic interactive modes. ``` cloudquery init [flags] ``` ## Examples ``` # Display prompts to select source and destination plugins and generate a configuration file from them cloudquery init # Generate a configuration file for a sync from aws to bigquery cloudquery init --source aws --destination bigquery # Display a prompt to select a source plugin and generate a configuration file for a sync from it to bigquery cloudquery init --destination bigquery # Display a prompt to select a destination plugin and generate a configuration file for a sync from aws to it cloudquery init --source aws # Accept all defaults and generate a configuration file for a sync from the first source and destination plugins cloudquery init --yes ``` ## Options ``` --destination string Destination plugin name or path --disable-ai Disable AI assistant --disable-platform Skip CloudQuery Platform sync scaffolding -h, --help help for init --resume-conversation Resume existing AI conversation instead of starting a new one --source string Source plugin name or path --spec-path string Output spec file path --yes Accept all defaults ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery](/cli/cli-reference/cloudquery) - CloudQuery CLI - [Getting Started](/cli/getting-started) - Full quickstart guide using the init command - [Configuration Guide](/cli/core-concepts/configuration) - Understand the generated configuration files --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_login # cloudquery login Login to CloudQuery Hub. ## Synopsis Login to CloudQuery Hub. This is required to download plugins from CloudQuery Hub. Local plugins and different registries don't need login. ``` cloudquery login [flags] ``` ## Examples ``` # Log in to CloudQuery Hub cloudquery login # Log in to a specific team cloudquery login --team my-team ``` ## Options ``` -h, --help help for login -t, --team string Team to login to. Specify the team name, e.g. 'my-team' (not the display name) ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery](/cli/cli-reference/cloudquery) - CloudQuery CLI - [Generate API Key](/cli/managing-cloudquery/deployments/generate-api-key) - Create API keys for headless authentication - [Getting Started](/cli/getting-started) - Install and run your first sync --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_logout # cloudquery logout Log out of CloudQuery Hub. ## Synopsis Log out of CloudQuery Hub. ``` cloudquery logout [flags] ``` ## Options ``` -h, --help help for logout ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery](/cli/cli-reference/cloudquery) - CloudQuery CLI - [Generate API Key](/cli/managing-cloudquery/deployments/generate-api-key) - Manage API keys for authentication - [Security](/cli/managing-cloudquery/security) - CloudQuery security best practices --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_migrate # cloudquery migrate Update schema of your destinations based on the latest changes in sources from your configuration ## Synopsis Update schema of your destinations based on the latest changes in sources from your configuration ``` cloudquery migrate [files or directories] [flags] ``` ## Examples ``` # Run migration for plugins specified in directory cloudquery migrate ./directory # Run migration for plugins specified in directory and config files cloudquery migrate ./directory ./aws.yml ./pg.yml ``` ## Options ``` -h, --help help for migrate --license string set offline license file ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery](/cli/cli-reference/cloudquery) - CloudQuery CLI - [Schema Migrations](/cli/managing-cloudquery/migrations) - How CloudQuery handles schema changes - [Destination Integrations](/cli/integrations/destinations) - Configure migration modes --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_plugin # cloudquery plugin Plugin commands ## Options ``` -h, --help help for plugin ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery](/cli/cli-reference/cloudquery) - CloudQuery CLI * [cloudquery plugin install](/cli/cli-reference/cloudquery_plugin_install) - Install required plugin images from your configuration * [cloudquery plugin publish](/cli/cli-reference/cloudquery_plugin_publish) - Publish to CloudQuery Hub. - [Integration Concepts](/cli/core-concepts/integrations) - How integrations work - [Managing Versions](/cli/advanced/managing-versions) - Integration versioning --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_plugin_install # cloudquery plugin install Install required plugin images from your configuration ## Synopsis Install required plugin images from your configuration ``` cloudquery plugin install [files or directories] [flags] ``` ## Examples ``` # Install required plugins specified in directory cloudquery plugin install ./directory # Install required plugins specified in directory and config files cloudquery plugin install ./directory ./aws.yml ./pg.yml ``` ## Options ``` -h, --help help for install ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery plugin](/cli/cli-reference/cloudquery_plugin) - Plugin commands - [Managing Versions](/cli/advanced/managing-versions) - Understand version management - [Source Integrations](/cli/integrations/sources) - Available source integrations --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_plugin_publish # cloudquery plugin publish Publish to CloudQuery Hub. ## Synopsis Publish to CloudQuery Hub. This publishes a plugin version to CloudQuery Hub from a local dist directory. ``` cloudquery plugin publish [-D dist] [flags] ``` ## Examples ``` # Publish a plugin version from a local dist directory cloudquery plugin publish ``` ## Options ``` -D, --dist-dir string Path to the dist directory (default "dist") -f, --finalize Finalize the plugin version after publishing. If false, the plugin version will be marked as draft. -h, --help help for publish -U, --ui-dir string Path to the built plugin UI directory ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery plugin](/cli/cli-reference/cloudquery_plugin) - Plugin commands - [Publishing an Integration](/cli/integrations/creating-new-integration/publishing) - Full publishing guide - [Creating a New Integration](/cli/integrations/creating-new-integration) - Build an integration first --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_switch # cloudquery switch Switches between teams. ## Synopsis Switches between teams. ``` cloudquery switch [team] [flags] ``` ## Examples ``` # Switch to a different team cloudquery switch my-team ``` ## Options ``` -h, --help help for switch ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery](/cli/cli-reference/cloudquery) - CloudQuery CLI - [Managing Versions](/cli/advanced/managing-versions) - Understand integration versioning - [Source Integrations](/cli/integrations/sources) - Configure source integration versions --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_sync # cloudquery sync Sync resources from configured source plugins to destinations ## Synopsis Sync resources from configured source plugins to destinations ## Flag Details ### --summary-location When set, a JSON summary of each sync is appended to the specified file after the sync completes. The file uses JSONL format (one JSON object per line). When syncing to multiple destinations, a separate entry is written for each destination. Example usage: ```bash cloudquery sync config.yml --summary-location summary.jsonl ``` The summary contains the following fields: | Field | Type | Description | | ------------------------ | -------- | ----------------------------------------------------------------- | | `cli_version` | string | CloudQuery CLI version | | `sync_id` | string | Unique sync identifier (from `--invocation-id` or auto-generated) | | `sync_time` | string | Sync start time (RFC 3339) | | `sync_duration_ms` | number | Sync duration in milliseconds | | `resources` | number | Total resources synced | | `source_name` | string | Source integration name | | `source_path` | string | Source integration path | | `source_version` | string | Source integration version | | `source_errors` | number | Errors from the source integration | | `source_warnings` | number | Warnings from the source integration | | `source_tables` | string[] | Tables synced from source | | `destination_name` | string | Destination integration name | | `destination_path` | string | Destination integration path | | `destination_version` | string | Destination integration version | | `destination_errors` | number | Errors from the destination integration | | `destination_warnings` | number | Warnings from the destination integration | | `sync_group_id` | string | Rendered sync group ID (if configured) | | `shard_num` | number | Shard number (if using `--shard`) | | `shard_total` | number | Total shards (if using `--shard`) | | `resources_per_table` | object | Resource count per table | | `errors_per_table` | object | Error count per table | | `durations_per_table_ms` | object | Duration per table in milliseconds | ### --invocation-id A UUID that uniquely identifies a sync invocation. If not provided, a random UUID is automatically generated. This flag serves three purposes: 1. **OpenTelemetry correlation**: The UUID is attached to all logs and traces, allowing you to correlate CLI activity with integration activity in your [monitoring setup](/cli/managing-cloudquery/monitoring/overview). 2. **Sync summary**: The UUID is stored as the `sync_id` field in the [sync summary](#--summary-location). 3. **`sync_group_id` template**: When a destination's [`sync_group_id`](/cli/integrations/destinations#sync_group_id) uses the `{{SYNC_ID}}` placeholder, it is replaced with this UUID at runtime. Example: using a fixed invocation ID for repeatable tracing: ```bash cloudquery sync config.yml --invocation-id 550e8400-e29b-41d4-a716-446655440000 ``` ``` cloudquery sync [files or directories] [flags] ``` ## Examples ``` # Sync resources from configuration in a directory cloudquery sync ./directory # Sync resources from directories and files cloudquery sync ./directory ./aws.yml ./pg.yml # Log tables metrics to a file cloudquery sync ./directory ./aws.yml ./pg.yml --tables-metrics-location metrics.txt # Shard the sync process into 4 shards and run the first shard cloudquery sync spec.yml --shard 1/4 ``` ## Options ``` -h, --help help for sync --license string Set offline license file. --no-migrate Disable auto-migration before sync. By default, sync runs a migration before syncing resources. --shard string Allows splitting the sync process into multiple shards. This feature is in Preview. Please provide feedback to help us improve it. For a list of supported plugins visit https://www.cloudquery.io/docs/cli/managing-cloudquery/running-in-parallel --summary-location string Sync summary file location. --tables-metrics-location string Tables metrics file location. This feature is in Preview. Please provide feedback to help us improve it. Works with plugins released on 2024-07-10 or later. ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery](/cli/cli-reference/cloudquery) - CloudQuery CLI - [Syncs](/cli/core-concepts/syncs) - Understand full and incremental sync modes - [Configuration Guide](/cli/core-concepts/configuration) - Set up sync configurations - [Performance Tuning](/cli/advanced/performance-tuning) - Optimize sync performance --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_tables # cloudquery tables Generate documentation for all supported tables of source plugins specified in the spec(s) ## Synopsis Generate documentation for all supported tables of source plugins specified in the spec(s) ``` cloudquery tables [files or directories] [flags] ``` ## Examples ``` # Generate documentation for all supported tables of source plugins specified in the spec(s) cloudquery tables ./directory # The default format is JSON, you can override it with --format cloudquery tables ./directory --format markdown # You can also specify an output directory. The default is ./cq-docs cloudquery tables ./directory --output-dir ./docs # You can also filter which tables are included in the output. The default is all, use --filter=spec to include only tables referenced in the spec cloudquery tables ./directory --filter spec ``` ## Options ``` --filter string Filter tables. One of: all, spec (default "all") --format string Output format. One of: json, markdown (default "json") -h, --help help for tables --output-dir string Base output directory for generated files (default "cq-docs") ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery](/cli/cli-reference/cloudquery) - CloudQuery CLI - [Source Integrations](/cli/integrations/sources) - Configure which tables to sync - [Integration Concepts](/cli/core-concepts/integrations) - How source integrations define tables --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_test-connection # cloudquery test-connection Test plugin connections to sources and/or destinations ## Synopsis Test plugin connections to sources and/or destinations ``` cloudquery test-connection [files or directories] [flags] ``` ## Examples ``` # Test plugin connections to sources and/or destinations cloudquery test-connection ./directory # Test plugin connections from directories and files cloudquery test-connection ./directory ./aws.yml ./pg.yml ``` ## Options ``` -h, --help help for test-connection ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery](/cli/cli-reference/cloudquery) - CloudQuery CLI - [Source Integrations](/cli/integrations/sources) - Configure source connections - [Destination Integrations](/cli/integrations/destinations) - Configure destination connections - [Troubleshooting](/cli/getting-support/troubleshooting) - Debug connection issues --- Source: https://www.cloudquery.io/docs/cli/cli-reference/cloudquery_validate-config # cloudquery validate-config Validate config ## Synopsis Validate configuration without running a sync. For `registry: cloudquery` plugins, the spec JSON schema is fetched from the CloudQuery Hub API (https://api.cloudquery.io). This avoids downloading the plugin binary and works for public plugins without authentication; if a CloudQuery API token is available (via login or CLOUDQUERY_API_KEY) it is propagated so private plugins resolve too. For other registries (`local`, `grpc`, `docker`) the plugin is still spawned locally to obtain its schema, identical to the previous behaviour. The tables list is not validated against the source — this validation is stricter than the validation done during `sync`, so a config passing here will also pass sync's validation. ``` cloudquery validate-config [files or directories] [flags] ``` ## Examples ``` # Validate configs cloudquery validate-config ./directory # Validate configs from directories and files cloudquery validate-config ./directory ./aws.yml ./pg.yml ``` ## Options ``` -h, --help help for validate-config --license cloudquery sync --license Set offline license file. When provided, the Hub API is bypassed and plugins are spawned locally (mirrors cloudquery sync --license). ``` ## Options inherited from parent commands ``` --cq-dir string directory to store cloudquery files, such as downloaded plugins (default ".cq") --invocation-id uuid useful for when using Open Telemetry integration for tracing and logging to be able to correlate logs and traces through many services (default ) --log-console enable console logging --log-file-name string Log filename (default "cloudquery.log") --log-file-overwrite Overwrite log file on each run instead of appending. Use this if your filesystem does not support append mode (e.g. FUSE-mounted cloud storage). --log-format string Logging format (json, text) (default "text") --log-level string Logging level (trace, debug, info, warn, error) (default "info") --no-log-file Disable logging to file --telemetry-level string Telemetry level (none, errors, stats, all) (default "all") ``` ## See Also * [cloudquery](/cli/cli-reference/cloudquery) - CloudQuery CLI - [Configuration Guide](/cli/core-concepts/configuration) - Configuration format and options - [Environment Variables](/cli/managing-cloudquery/environment-variables) - Variable substitution in configuration files --- Source: https://www.cloudquery.io/docs/cli/core-concepts/architecture # Architecture CloudQuery works by connecting to your cloud providers and SaaS apps, extracting configuration data through their APIs, and loading it into a database you control. The CLI is the orchestrator - it manages this pipeline by coordinating independent components called [integrations](/cli/core-concepts/integrations) that each handle one part of the job. This page covers how those components fit together. If you're building a custom integration, understanding this architecture is especially relevant. For day-to-day usage, start with [Configuration](/cli/core-concepts/configuration) instead. CloudQuery uses [gRPC](https://grpc.io/docs/languages/go/basics/) for communication between the CLI and integrations. ## Data Flow The CLI orchestrates all communication between integrations. Integrations do not talk to each other directly - every record flows through the CLI. ```text Source --> CLI --> Transformer 1 --> Transformer 2 --> ... --> Destination 1 ^ | Destination 2 | + adds _cq_sync_time, _cq_source_name, Destination N | _cq_sync_group_id to every record External APIs ``` Each connection between components uses gRPC. The transformer chain is optional and configured per destination - each destination can have its own transformers (or none). A single source can feed multiple destinations simultaneously without extra API calls. During a sync: 1. The **source integration** fetches data from external APIs and streams records to the CLI over gRPC. 2. The **CLI** applies an internal transformation to each record, adding columns like `_cq_sync_time`, `_cq_source_name`, and `_cq_sync_group_id`. 3. If **transformer integrations** are configured for a destination, records pass through them sequentially. Each transformer receives records from the previous one and forwards its output to the next. The order matches the `transformers` list in the destination spec. 4. The **destination integration** receives the final records and writes them to the target database or storage. ## CloudQuery CLI Responsibilities - Main entry point and CLI for the user. - Reading CloudQuery configuration files. - Downloading, verifying, and running integrations. - Orchestrating the sync pipeline between source, transformer, and destination integrations. - Applying internal transformations (adding `_cq_*` columns to every record). ## CloudQuery Integration Responsibilities - Intended to be run only by the CloudQuery CLI. - Communicates with the CLI over gRPC to receive commands and stream data. - **Source integrations**: initialization, authentication, and fetching data via third-party cloud/SaaS APIs. - **Destination integrations**: authentication, schema migration, and data insertion. - **Transformer integrations**: receiving records, applying transformations, and forwarding results. ## SDK CloudQuery integrations use the [Integration SDK](https://pkg.go.dev/github.com/cloudquery/plugin-sdk/v4), which abstracts most of the TL (in ETL, extract-transform-load). As a developer, you only need to implement the "E" (extract) - initializing, authentication, and fetching data via the third-party APIs. The SDK takes care of transforming the data and loading it into the destination. ## Next Steps - [Integrations](/cli/core-concepts/integrations) - learn about the three integration types and their responsibilities - [Syncs](/cli/core-concepts/syncs) - understand how syncs work, including write modes and incremental tables - [Configuration](/cli/core-concepts/configuration) - set up your first source and destination configuration - [CloudQuery Types](/cli/core-concepts/cloudquery-types) - understand the Arrow-based type system used by integrations - [Creating a new integration](/cli/integrations/creating-new-integration) - build your own source, destination, or transformer integration - [Publishing integrations to the hub](/cli/integrations/creating-new-integration/publishing) - share your integration with the community - [Integration SDK documentation](https://pkg.go.dev/github.com/cloudquery/plugin-sdk/v4) - SDK API reference --- Source: https://www.cloudquery.io/docs/cli/core-concepts/cloudquery-types # CloudQuery Types When CloudQuery syncs data from a cloud provider, every column in every table has a defined type - timestamps, strings, IP addresses, JSON blobs, and so on. Understanding the type system matters when you're writing SQL queries against synced data, building custom integrations, or troubleshooting why a column looks different in your destination than you expected. CloudQuery uses [Apache Arrow](https://arrow.apache.org/docs/index.html) to represent data internally. Source integrations define columns in terms of Arrow types, and destinations convert from Arrow to their own native types (e.g. Arrow `Timestamp` becomes `TIMESTAMP` in PostgreSQL or `DateTime` in ClickHouse). Apart from the native Arrow types (strings, integers, floats, booleans, timestamps, lists, structs, etc.), CloudQuery defines four custom types as Arrow extensions: | Type | Extension Name | Underlying Arrow Storage | Description | | -------- | -------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------- | | **JSON** | `json` | Binary | A valid JSON object. Used for unstructured or semi-structured data like tags, metadata, and API responses. | | **Inet** | `inet` | Binary | An IP address or CIDR block (e.g. `192.168.1.0/24`). Stored as binary-encoded `net.IPNet` values. | | **MAC** | `mac` | Binary | A hardware MAC address (e.g. `01:23:45:67:89:ab`). Stored as binary-encoded `net.HardwareAddr` values. | | **UUID** | `uuid` | FixedSizeBinary(16) | A 16-byte UUID (e.g. `123e4567-e89b-12d3-a456-426614174000`). The only extension type that uses fixed-size storage. | These extension types carry semantic meaning that destinations can use for more accurate type mapping. For example, a destination can map `UUID` to a native `UUID` column type rather than treating it as raw binary. ## String Representation When Arrow types need to be represented as strings (for CSV output, debugging, or destinations that lack native type support), CloudQuery uses specific formatting conventions. For the full mapping of all Arrow types to their string representations, see [Arrow String Representation](/cli/advanced/arrow-string-representation). ## Next Steps - [Arrow String Representation](/cli/advanced/arrow-string-representation) - full mapping of Arrow types to string representations - [Integrations](/cli/core-concepts/integrations) - how source and destination integrations define and consume typed data - [Schema Migrations](/cli/managing-cloudquery/migrations) - how CloudQuery handles type changes between syncs - [Creating a new integration](/cli/integrations/creating-new-integration) - define tables and columns using Arrow types in your own integration - [Integration SDK documentation](https://pkg.go.dev/github.com/cloudquery/plugin-sdk/v4) - SDK API reference including type definitions --- Source: https://www.cloudquery.io/docs/cli/core-concepts/configuration # CloudQuery Integration Configuration A CloudQuery sync fetches data from cloud accounts (sources) and writes it to one or more destinations. A sync requires at least one source- and one destination configuration. Configuration files are specified in YAML format and can be either split across multiple files or combined. Learn more about [how syncs work](/cli/core-concepts/syncs) and [integration architecture](/cli/core-concepts/integrations). ## Example using multiple files One option is to maintain configuration for your source and destination integrations in separate files. Here is an example with only one source and one destination integration: ```yaml copy filename="aws.yml" kind: source spec: name: aws path: cloudquery/aws registry: cloudquery version: "VERSION_SOURCE_AWS" tables: ["aws_s3_buckets"] destinations: ["postgresql"] ``` ```yaml copy filename="postgresql.yml" kind: destination spec: name: postgresql path: cloudquery/postgresql registry: cloudquery version: "VERSION_DESTINATION_POSTGRESQL" spec: connection_string: ${PG_CONNECTION_STRING} ``` With these two files, we can run a sync using: ```shell copy cloudquery sync aws.yml postgresql.yml ``` ### Adding another source To add a `gcp` source as well, create its configuration in a new file: ```yaml copy filename="gcp.yml" kind: source spec: name: gcp path: cloudquery/gcp registry: cloudquery version: "VERSION_SOURCE_GCP" tables: ["gcp_storage_buckets"] destinations: ["postgresql"] ``` And now sync both `aws` and `gcp` to `postgresql` in a single command: ```shell copy cloudquery sync aws.yml gcp.yml postgresql.yml ``` ## Example using one file You can also combine sources and destinations into a single file by separating the sections with `---`: ```yaml copy filename="config.yml" kind: source spec: name: aws path: cloudquery/aws registry: cloudquery version: "VERSION_SOURCE_AWS" tables: ["aws_s3_buckets"] destinations: ["postgresql"] --- kind: destination spec: name: postgresql path: cloudquery/postgresql registry: cloudquery version: "VERSION_DESTINATION_POSTGRESQL" spec: connection_string: ${PG_CONNECTION_STRING} ``` Now we can run a sync using: ```shell copy cloudquery sync config.yml ``` This example shows only two integration sections, but a configuration file is allowed to contain any number of integration sections. ## Source spec fields These are the top-level fields available under `spec` for a `kind: source` block. For full documentation including examples, see [Source Integration Reference](/cli/integrations/sources). | Field | Type | Default | Description | |---|---|---|---| | `name` | string | N/A | **Required.** Unique name for this source. | | `path` | string | N/A | **Required.** Plugin path, e.g. `cloudquery/aws`. | | `registry` | string | `cloudquery` | Plugin registry: `cloudquery`, `github`, `local`, `grpc`, `docker`. | | `version` | string | N/A | Plugin version (required for `cloudquery`/`github` registries). | | `tables` | list | N/A | **Required.** Tables to sync. Accepts wildcards, e.g. `["aws_ec2_*"]`. | | `skip_tables` | list | `[]` | Tables to skip. Useful with wildcards. Skipping a parent also skips its children. | | `skip_dependent_tables` | bool | `true` | When `true`, dependent tables are only synced if explicitly listed in `tables`. Set to `false` to restore pre-v6.0.0 behavior where matched parents pulled in all descendants. | | `destinations` | list | N/A | **Required.** Names of destination plugins to write to. | | `backend_options` | object | N/A | Enables incremental (stateful) syncs. See [Managing Incremental Tables](/cli/advanced/managing-incremental-tables). | | `deterministic_cq_id` | bool | `false` | When `true`, generates `_cq_id` as a hash of primary keys instead of a random UUID. Useful for deduplication across syncs. | | `otel_endpoint` | string | N/A | *(Preview)* OpenTelemetry OTLP/HTTP endpoint for sync traces. See [Monitoring](/cli/managing-cloudquery/monitoring/overview). | | `otel_endpoint_insecure` | bool | `false` | *(Preview)* Skip TLS verification for the OTel endpoint. | | `spec` | object | N/A | Plugin-specific configuration. See each plugin's documentation. | | `docker_registry_auth_token` | string | N/A | Auth token for private Docker registries. See [Source Integration Reference](/cli/integrations/sources#docker_registry_auth_token). | ### backend_options fields The `backend_options` object enables state tracking for incremental syncs: ```yaml backend_options: table_name: cq_state_aws # table to store incremental state connection: "@@plugins.postgresql.connection" # destination to use for state ``` | Field | Type | Description | |---|---|---| | `table_name` | string | **Required.** Table name for storing incremental state. | | `connection` | string | **Required.** `@@plugins..connection` or a gRPC address. | See [Managing Incremental Tables](/cli/advanced/managing-incremental-tables) for full examples. ## Destination spec fields These are the top-level fields available under `spec` for a `kind: destination` block. For full documentation, see [Destination Integration Reference](/cli/integrations/destinations). | Field | Type | Default | Description | |---|---|---|---| | `name` | string | N/A | **Required.** Unique name for this destination. | | `path` | string | N/A | **Required.** Plugin path, e.g. `cloudquery/postgresql`. | | `registry` | string | `cloudquery` | Plugin registry: `cloudquery`, `github`, `local`, `grpc`, `docker`. | | `version` | string | N/A | Plugin version (required for `cloudquery`/`github` registries). | | `write_mode` | string | `overwrite-delete-stale` | How to handle existing rows: `overwrite-delete-stale`, `overwrite`, `append`. | | `migrate_mode` | string | `safe` | Schema migration behavior: `safe` (non-destructive) or `forced` (allows column drops). | | `pk_mode` | string | `default` | Primary key behavior: `default` (use plugin PKs) or `cq-id-only` (use `_cq_id` only). | | `sync_group_id` | string | N/A | Groups multiple syncs together. Supports placeholders: `{{SYNC_ID}}`, `{{YEAR}}`, `{{MONTH}}`, `{{DAY}}`, `{{HOUR}}`, `{{MINUTE}}`. | | `send_sync_summary` | bool | `false` | When `true`, writes a sync summary row to the `cloudquery_sync_summaries` table after each sync. | | `transformers` | list | `[]` | Names of transformer plugins to run on data before writing. See [Transformer Integrations](/cli/integrations/transformers). | | `spec` | object | N/A | Plugin-specific configuration. See each plugin's documentation. | | `docker_registry_auth_token` | string | N/A | Auth token for private Docker registries. See [Destination Integration Reference](/cli/integrations/destinations#docker_registry_auth_token). | ### sync_group_id example `sync_group_id` is useful for identifying which sync run produced a set of rows, or for partitioning data by time: ```yaml kind: destination spec: name: postgresql # ... sync_group_id: "aws-sync-{{YEAR}}-{{MONTH}}-{{DAY}}" ``` ## Removed source fields The following source fields were removed in CLI v3.6.0. They no longer have any effect and will cause a validation error if used. Remove them from any older configuration files. | Removed field | Replacement | |---|---| | `concurrency` | Configure concurrency at the plugin level. See the plugin's own documentation. | | `table_concurrency` | Configure concurrency at the plugin level. | | `resource_concurrency` | Configure concurrency at the plugin level. | | `scheduler` | Configure the scheduler at the plugin level. See the plugin's own documentation. | | `backend` | Use `backend_options` instead. | | `backend_spec` | Use `backend_options` instead. | ## Additional configuration features - **Transformer integrations**: In addition to `kind: source` and `kind: destination`, you can also configure `kind: transformer` integrations that process data between sources and destinations. See [Transformer Integrations](/cli/integrations/transformers) for details. - **Registry options**: The `registry` field supports multiple values beyond `cloudquery`, including `local`, `grpc`, and `docker`. See [Source Integrations](/cli/integrations/sources#registry) for all available options. - **Variable substitution**: Configuration files support environment variables (`${ENV_VAR}`), file contents (`${file:./path}`), and time-based values (`${time:5 days ago}`). See [Environment Variables](/cli/managing-cloudquery/environment-variables) for details. ## Next Steps - [AWS to PostgreSQL guide](/cli/getting-started/aws-to-postgresql) - end-to-end walkthrough using the AWS and PostgreSQL configuration shown above - [Syncs](/cli/core-concepts/syncs) - understand how syncs work, including write modes and incremental tables - [Source integration reference](/cli/integrations/sources) - full configuration options for source specs - [Destination integration reference](/cli/integrations/destinations) - full configuration options for destination specs - [Transformer integration reference](/cli/integrations/transformers) - configure pre-load transformations - [Managing Incremental Tables](/cli/advanced/managing-incremental-tables) - stateful syncs with `backend_options` - [Environment Variables](/cli/managing-cloudquery/environment-variables) - variable substitution, file references, and time-based values - [`cloudquery sync` CLI reference](/cli/cli-reference/cloudquery_sync) - command-line options and flags - [`cloudquery validate-config` CLI reference](/cli/cli-reference/cloudquery_validate-config) - validate configuration files before running a sync - [Deployment guides](/cli/managing-cloudquery/deployments) - run CloudQuery in Docker, Kubernetes, ECS, GitHub Actions, and more --- Source: https://www.cloudquery.io/docs/cli/core-concepts/dashboards # Dashboards & Visualizations When using the CLI, CloudQuery data lands in the destination of your choice (PostgreSQL, BigQuery, Snowflake, etc.), where you can visualize it with external BI tools. ## Grafana dashboards CloudQuery provides official Grafana dashboards that work with data synced by the CLI. Browse and download them from the [CloudQuery Hub](https://www.cloudquery.io/hub/templates). These dashboards can be combined with [CloudQuery SQL transformations](https://www.cloudquery.io/hub/addons/transformation) for security, compliance, DevOps, and cost optimization use cases. ## Other BI tools You can also build your own dashboards with any BI tool that connects to your destination database, including [Apache Superset](https://www.cloudquery.io/blog/cloud-asset-inventory-cloudquery-apache-superset), [AWS QuickSight](https://www.cloudquery.io/blog/cloud-asset-inventory-cloudquery-aws-quicksight), [Microsoft Power BI](https://www.cloudquery.io/blog/cloud-asset-inventory-cloudquery-microsoft-power-bi), and others. ## Built-in reports on CloudQuery Platform If you prefer built-in reporting without managing a separate BI tool, [CloudQuery Platform](/platform/features/reports) offers 40+ [report templates](/platform/features/reports/built-in-report-templates) covering security, compliance, asset inventory, cost optimization, and more. See [Platform vs CLI](/platform/introduction/platform-vs-cli) for a full comparison. ## Next Steps - [Transformations](/cli/core-concepts/transformations) - post-load dbt transformations that create the views dashboards rely on - [Browse Grafana dashboards on the Hub](https://www.cloudquery.io/hub/templates) - download official dashboards for your data - [Browse transformations on the Hub](https://www.cloudquery.io/hub/addons/transformation) - find dbt packages for asset inventory, cost, and compliance - [Platform Reports](/platform/features/reports) - built-in reporting on CloudQuery Platform - [Platform SQL Console](/platform/features/sql-console) - run ad-hoc queries on CloudQuery Platform --- Source: https://www.cloudquery.io/docs/cli/core-concepts/integrations # Integrations **Using CloudQuery Platform?** See [Platform Integrations](/platform/core-concepts/integrations) for how integrations work in the managed platform, including the default ClickHouse destination and multi-destination support. CloudQuery has an integration-based architecture, with integrations communicating over [gRPC](https://github.com/cloudquery/plugin-pb). An integration can be a source, destination, or transformer. - **Source integration** - Extracts and transforms configuration from cloud providers, SaaS apps, and other APIs ([Available source integrations](https://www.cloudquery.io/hub/plugins/source)). - **Destination integration** - Writes data from source integrations to databases, message queues, and storage ([Available destination integrations](https://www.cloudquery.io/hub/plugins/destination)). - **Transformer integration** - Optional component that modifies data in flight between a source and destination, such as renaming tables, obfuscating sensitive columns, or flattening JSON ([Available transformer integrations](https://www.cloudquery.io/hub/addons/transformation)). All integrations are split into official (maintained by CloudQuery) and community. You can also [build your own integration](/cli/integrations/creating-new-integration). ``` Sources Destinations ┌──────────┐ ┌──────────────┐ │ AWS │──┐ ┌──│ PostgreSQL │ ├──────────┤ │ ┌──────────┐ │ ├──────────────┤ │ GCP │──┼──│ CLI sync │─┼──│ BigQuery │ ├──────────┤ │ └──────────┘ │ ├──────────────┤ │ Azure │──┤ │ │ Snowflake │ ├──────────┤ │ │ ├──────────────┤ │ GitHub │──┘ └──│ S3 / MySQL │ ├──────────┤ └──────────────┘ │ 70+ │ │ more │ └──────────┘ ``` ## Source Integration The core responsibilities of a source integration: - Define the schema (tables). - Authenticate with the supported APIs, SaaS services and/or cloud providers. - Extract data from the supported APIs and transform it into the defined schema. - Send the data via [protobuf](https://github.com/cloudquery/plugin-pb-go/tree/main/pb) to the CLI for further processing and storage at the defined destination integrations. ### Popular Source Integrations | Integration | Description | | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | [AWS](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs) | EC2, S3, IAM, Lambda, RDS, and 300+ other AWS services | | [GCP](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs) | Compute, Storage, IAM, BigQuery, GKE, and more | | [Azure](https://www.cloudquery.io/hub/plugins/source/cloudquery/azure/latest/docs) | VMs, Storage, Active Directory, AKS, and more | | [GitHub](https://www.cloudquery.io/hub/plugins/source/cloudquery/github/latest/docs) | Repositories, teams, members, actions, and security advisories | | [Kubernetes](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) | Pods, deployments, services, nodes, and cluster configuration | | [Cloudflare](https://www.cloudquery.io/hub/plugins/source/cloudquery/cloudflare/latest/docs) | DNS, WAF, workers, and zone configuration | | [GitLab](https://www.cloudquery.io/hub/plugins/source/cloudquery/gitlab/latest/docs) | Projects, pipelines, merge requests, and group members | | [Oracle](https://www.cloudquery.io/hub/plugins/source/cloudquery/oracle/latest/docs) | Compute, networking, storage, and identity resources | Browse all source integrations on the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source). For the full source configuration reference, see [Source Integrations](/cli/integrations/sources). ## Destination Integration The core responsibilities of a destination integration: - Authenticate with the destination (such as database, message queue, storage). - Auto-migrate the schemas defined by the source integrations. - Save each incoming object in the appropriate table. ### Popular Destination Integrations | Destination | Use case | | ------------------------------------------------------------------------------------------------- | ------------------------------------------- | | [PostgreSQL](https://www.cloudquery.io/hub/plugins/destination/cloudquery/postgresql/latest/docs) | Relational queries, existing data pipelines | | [S3](https://www.cloudquery.io/hub/plugins/destination/cloudquery/s3/latest/docs) | Data lake storage, archival | | [BigQuery](https://www.cloudquery.io/hub/plugins/destination/cloudquery/bigquery/latest/docs) | Large-scale analytics in GCP | | [MySQL](https://www.cloudquery.io/hub/plugins/destination/cloudquery/mysql/latest/docs) | Relational database, application backends | | [Neo4j](https://www.cloudquery.io/hub/plugins/destination/cloudquery/neo4j/latest/docs) | Graph-based infrastructure analysis | | [Snowflake](https://www.cloudquery.io/hub/plugins/destination/cloudquery/snowflake/latest/docs) | Cloud data warehouse analytics | Browse all destination integrations on the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/destination). See [Configuration Reference](/cli/integrations/destinations#destination-configuration-reference) and learn more about [CloudQuery configuration](/cli/core-concepts/configuration). ## Transformer Integration The core responsibilities of a transformer integration: - Receive records from the source integration via the CLI. - Apply configured transformations to each record (renaming, filtering, obfuscation, etc.). - Deliver the transformed records to the destination integration. Transformers are optional - if none are configured, data flows directly from source to destination. See [Transformer Integrations](/cli/integrations/transformers) for configuration details and [Transformations](/cli/core-concepts/transformations) for an overview of all available transformations. ## Next Steps - [Configuration](/cli/core-concepts/configuration) - set up source and destination configuration files - [Source integration reference](/cli/integrations/sources) - full configuration options for source integrations - [Destination integration reference](/cli/integrations/destinations) - full configuration options for destination integrations - [Transformer integration reference](/cli/integrations/transformers) - configuration and available transformers - [Architecture](/cli/core-concepts/architecture) - how the CLI orchestrates the data pipeline between integrations - [Creating a new integration](/cli/integrations/creating-new-integration) - build your own source, destination, or transformer - [Browse source integrations](https://www.cloudquery.io/hub/plugins/source) - find integrations for your cloud providers and SaaS apps - [Browse destination integrations](https://www.cloudquery.io/hub/plugins/destination) - find integrations for your databases and storage --- Source: https://www.cloudquery.io/docs/cli/core-concepts/syncs # Syncs **Using CloudQuery Platform?** See [Platform Syncs](/platform/core-concepts/syncs) for how the platform manages write modes, incremental state, and table views automatically. When you run `cloudquery sync `, the CloudQuery CLI fetches data from all the source [integrations](/cli/core-concepts/integrations) matched by the config and delivers it to the matched destination integrations. This might mean fetching data from [AWS](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs), [GCP](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs) and [Azure](https://www.cloudquery.io/hub/plugins/source/cloudquery/azure/latest/docs) and delivering it to [PostgreSQL](https://www.cloudquery.io/hub/plugins/destination/cloudquery/postgresql/latest/docs), or it could mean fetching data from AWS, [Cloudflare](https://www.cloudquery.io/hub/plugins/source/cloudquery/cloudflare/latest/docs) and [GitLab](https://www.cloudquery.io/hub/plugins/source/cloudquery/gitlab/latest/docs) and delivering it to [BigQuery](https://www.cloudquery.io/hub/plugins/destination/cloudquery/bigquery/latest/docs), [S3](https://www.cloudquery.io/hub/plugins/destination/cloudquery/s3/latest/docs) and [MySQL](https://www.cloudquery.io/hub/plugins/destination/cloudquery/mysql/latest/docs). It all depends on the configuration provided, and there is a near-endless array of possible combinations that grows every time a new source or destination is created. (Configuration is described in the [Configuration section](/cli/core-concepts/configuration).) Browse all available source and destination integrations on the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source). CloudQuery streams data to the destination as it arrives. As soon as data is received for a source integration resource, it is delivered to the destination integration. Destination integrations may batch writes for performance reasons, but generally data will be delivered to the destination as the sync progresses. ## Table Sync Modes Table syncs come in two flavors: `full` and `incremental`. A single `cloudquery sync` command invocation can combine both types, and which type is used for a particular table depends on the table definition. ### Full Table Syncs This is the normal mode of operation for most tables. For tables in this mode, the CLI fetches a snapshot of all data from the corresponding APIs on every sync. Depending on the destination write mode, the data is then appended (`write_mode`: `append`), overwritten while keeping stale rows from previous syncs (`write_mode`: `overwrite`) or overwritten and rows from previous syncs deleted at the end of the sync (`write_mode`: `overwrite-delete-stale`). Learn more about [schema migrations](/cli/managing-cloudquery/migrations) and how CloudQuery handles schema changes. ### Incremental Table Syncs Some APIs lend themselves to being synced incrementally. Rather than fetch all past data on every sync, an incremental table will only fetch data that has changed since the last sync. This is done by storing some metadata in a state **backend**. The metadata is known as a **cursor**, and it marks where the last sync ended, so that the next sync can resume from the same point. Incremental syncs can be vastly more efficient than full syncs, especially for tables with large amounts of data. This is because only the data that's changed since the last sync needs to be retrieved, and in many cases this is a small subset of the overall dataset. Incremental tables are always clearly marked as "incremental" in integration table documentation, along with an indication of which columns are used for the value of the cursor. Because they use state, incremental tables require a state backend configured via `backend_options` in the source spec: ```yaml kind: source spec: # ... backend_options: table_name: cq_state_aws connection: "@@plugins.postgresql.connection" ``` For more details, see [Managing Incremental Tables](/cli/advanced/managing-incremental-tables). ## Destination Modes Destinations have three configuration modes that affect how syncs write data. These are set in the destination spec, not per-sync. ### Write Mode Controls what happens to existing data when new data arrives. Defaults to `overwrite-delete-stale`. - **`overwrite-delete-stale`** - new data replaces existing data, and rows from previous syncs that are no longer present in the source are deleted after the sync completes. - **`overwrite`** - new data replaces existing data, but stale rows from previous syncs are kept. - **`append`** - new rows are added alongside existing data from previous syncs. No data is deleted or replaced. ### Migrate Mode Controls how schema changes (new columns, type changes) are handled when a source integration updates its table definitions. Defaults to `safe`. - **`safe`** - only backward-compatible schema changes are applied (adding new columns, for example). Changes that would require dropping and recreating a table are skipped to avoid data loss. - **`forced`** - all schema changes are applied, including destructive ones like dropping and recreating tables. This is useful when you're using transformers that modify the schema and need the destination to match. ### PK Mode Controls how primary keys are set on destination tables. Defaults to `default`. - **`default`** - uses the primary keys defined by the source integration. - **`cq-id-only`** - uses only the `_cq_id` column as the primary key. This can help when the source-defined primary keys cause conflicts or when you want to preserve all rows including duplicates. ## Scaling Syncs For large cloud estates, you can split syncs across multiple processes or machines using sharding. See [Running Syncs in Parallel](/cli/managing-cloudquery/running-in-parallel) for details on how to distribute sync workloads. ## Next Steps - [AWS to PostgreSQL guide](/cli/getting-started/aws-to-postgresql) - follow a complete AWS to PostgreSQL sync from setup to querying data - [Configuration](/cli/core-concepts/configuration) - set up source and destination configuration files for your first sync - [Managing Incremental Tables](/cli/advanced/managing-incremental-tables) - configure state backends and cursors for incremental syncs - [Schema Migrations](/cli/managing-cloudquery/migrations) - understand how CloudQuery handles schema changes between syncs - [Running Syncs in Parallel](/cli/managing-cloudquery/running-in-parallel) - shard syncs across multiple processes for large cloud estates - [Performance Tuning](/cli/advanced/performance-tuning) - optimize sync speed and resource usage - [Monitoring with OpenTelemetry](/cli/managing-cloudquery/monitoring/overview) - track sync progress and performance - [Destination integration reference](/cli/integrations/destinations) - configure write mode, migrate mode, and PK mode - [`cloudquery sync` CLI reference](/cli/cli-reference/cloudquery_sync) - full command-line options and flags - Browse available integrations on the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source) --- Source: https://www.cloudquery.io/docs/cli/core-concepts/transformations # Transformations Not all data needs to land in your destination exactly as it comes from the source. You might need to strip sensitive columns before they reach the database, rename tables to match your naming conventions, or build aggregate views for reporting after the data lands. Transformations handle both of these scenarios. There are two categories, depending on when they run relative to data being written to your destination: - **Pre-load transformations** - transformer integrations that modify data _in flight_, between the source and destination. - **Post-load transformations** - dbt packages that run _after_ data has landed in your destination to create analytics-ready views. All transformations are available in [CloudQuery Hub](https://www.cloudquery.io/hub/addons/transformation). ## Pre-Load Transformations Pre-load transformations are **transformer integrations** that sit between a source and destination in your sync pipeline. They intercept records after they leave the source and modify them before they are written to the destination. This is useful for shaping data on the fly - renaming tables, removing sensitive columns, or filtering out unwanted rows - without touching the source or destination configuration. Transformer integrations are configured with `kind: transformer` in your CloudQuery configuration. You reference them in your destination spec under the `transformers` key, and the CLI chains them automatically during a sync. For full configuration details, see the [Transformer Integrations](/cli/integrations/transformers) page. ### Available Pre-Load Transformers | Transformer | Description | | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | [Basic](https://www.cloudquery.io/hub/plugins/transformer/cloudquery/basic/latest/docs) | Rename tables, add or remove columns, obfuscate sensitive data, normalize casing, drop rows by value, and add timestamps or literal columns. | | [JSON Flattener](https://www.cloudquery.io/hub/plugins/transformer/cloudquery/jsonflattener/latest/docs) | Flatten single-level JSON object fields into typed destination columns while preserving the original JSON column. | ### Example: Using the Basic Transformer The following configuration adds the Basic transformer to a PostgreSQL destination. It renames all tables with a `cq_sync_` prefix and obfuscates sensitive columns: ```yaml copy kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" registry: "cloudquery" version: "v8.0.7" write_mode: "overwrite-delete-stale" migrate_mode: forced transformers: - "basic" spec: connection_string: "postgresql://your.user:your.password@localhost:5432/db_name" --- kind: transformer spec: name: "basic" path: "cloudquery/basic" registry: "cloudquery" version: "VERSION_TRANSFORMER_BASIC" spec: transformations: - kind: obfuscate_sensitive_columns - kind: change_table_names tables: ["*"] new_table_name_template: "cq_sync_{{.OldName}}" ``` #### Basic Transformer Capabilities The Basic transformer supports the following transformation kinds: - **`remove_columns`** - Remove columns from specified tables. Supports JSON paths for nested fields. - **`add_column`** - Add a literal string column to specified tables. - **`add_current_timestamp_column`** - Add a column with the timestamp the record was processed. - **`add_primary_keys`** - Add additional primary key columns. - **`obfuscate_columns`** - Replace column values with a redacted placeholder. Supports JSON paths including array syntax (`#.field`). - **`obfuscate_sensitive_columns`** - Automatically obfuscate all columns the source has marked as sensitive. - **`change_table_names`** - Rename tables using a Go template (`{{.OldName}}`). - **`rename_column`** - Rename a single column on specified tables. - **`uppercase` / `lowercase`** - Normalize column values to upper or lower case. - **`drop_rows`** - Drop rows where a column matches a given value. > **Note:** Transformations are applied sequentially. If you rename tables, subsequent transformations must reference the new table names. ## Post-Load Transformations Post-load transformations are **dbt packages** that run after data has been synced to your destination. They create analytics-ready models, views, and dashboards on top of the raw synced data. These are useful for asset inventory reporting, cost analysis, and backup compliance - scenarios where you want to aggregate, join, or reshape data that has already landed. Post-load transformations are standalone dbt projects. You install them with dbt, configure a profile for your destination database, and run them with `dbt run` after a CloudQuery sync completes. ### Available Post-Load Transformations | Transformation | Description | | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | [AWS Asset Inventory](https://www.cloudquery.io/hub/addons/transformation/cloudquery/aws-asset-inventory) | Automated line-item listing of all active resources in your AWS environment. Creates an `aws_resources` view with ARN, account, region, tags, and more. | | [AWS Cost](https://www.cloudquery.io/hub/addons/transformation/cloudquery/aws-cost) | Analyze AWS expenses across services and accounts. Provides cost breakdowns for budgeting and optimization. | | [AWS Data Resilience (Backup)](https://www.cloudquery.io/hub/addons/transformation/cloudquery/aws-data-resilience) | Evaluate backup and disaster recovery coverage across your AWS resources. | | [Azure Asset Inventory](https://www.cloudquery.io/hub/addons/transformation/cloudquery/azure-asset-inventory) | Catalog Azure resources across subscriptions with a unified resource view. | | [GCP Asset Inventory](https://www.cloudquery.io/hub/addons/transformation/cloudquery/gcp-asset-inventory) | Catalog Google Cloud resources across projects with a unified resource view. | These transformations can be paired with BI tools like [Grafana](https://www.cloudquery.io/blog/open-source-cspm), [Apache Superset](https://www.cloudquery.io/blog/cloud-asset-inventory-cloudquery-apache-superset), [QuickSight](https://www.cloudquery.io/blog/cloud-asset-inventory-cloudquery-aws-quicksight), [PowerBI](https://www.cloudquery.io/blog/cloud-asset-inventory-cloudquery-microsoft-power-bi), and others to visualize and monitor cloud infrastructure. ## Creating Custom Transformers If the available transformers don't cover your use case, you can build your own. See [Creating a New Integration](/cli/integrations/creating-new-integration) for details on implementing a custom transformer integration. ## Next Steps - [Transformer integration reference](/cli/integrations/transformers) - full configuration spec for pre-load transformer integrations - [Dashboards](/cli/core-concepts/dashboards) - visualize transformation output with Grafana dashboards and BI tools - [Configuration](/cli/core-concepts/configuration) - set up source, destination, and transformer configuration files - [Browse all transformations on the Hub](https://www.cloudquery.io/hub/addons/transformation) - find pre-load and post-load transformations - [Creating a new integration](/cli/integrations/creating-new-integration) - build a custom transformer integration --- Source: https://www.cloudquery.io/docs/cli/getting-started/aws-to-postgresql # AWS to PostgreSQL Select your operating system: - [macOS](/cli/getting-started/aws-to-postgresql/macOS) - [Linux](/cli/getting-started/aws-to-postgresql/linux) - [Windows](/cli/getting-started/aws-to-postgresql/windows) --- Source: https://www.cloudquery.io/docs/cli/getting-started/aws-to-postgresql/linux # AWS to PostgreSQL --- Source: https://www.cloudquery.io/docs/cli/getting-started/aws-to-postgresql/macOS # AWS to PostgreSQL --- Source: https://www.cloudquery.io/docs/cli/getting-started/aws-to-postgresql/windows # AWS to PostgreSQL --- Source: https://www.cloudquery.io/docs/cli/getting-started # Quickstart See the following platform-specific guides: - [Linux](/cli/getting-started/quickstart/linux) - [macOS](/cli/getting-started/quickstart/macOS) - [Windows](/cli/getting-started/quickstart/windows) Once you've completed the quickstart, sync AWS into PostgreSQL, the most common production use case: - [AWS to PostgreSQL - macOS](/cli/getting-started/aws-to-postgresql/macOS) - [AWS to PostgreSQL - Linux](/cli/getting-started/aws-to-postgresql/linux) - [AWS to PostgreSQL - Windows](/cli/getting-started/aws-to-postgresql/windows) ## Next Steps After completing your first sync, learn more about CloudQuery's core concepts: - [Integration Architecture](/cli/core-concepts/integrations) - Understand how source and destination integrations work - [Configuration Guide](/cli/core-concepts/configuration) - Learn how to configure CloudQuery syncs - [Sync Modes](/cli/core-concepts/syncs) - Understand full vs incremental syncs - [Available Integrations](/cli/integrations) - Browse all source and destination integrations --- Source: https://www.cloudquery.io/docs/cli/getting-started/quickstart # Quickstart Select your operating system to get started: - [macOS](/cli/getting-started/quickstart/macOS) - [Linux](/cli/getting-started/quickstart/linux) - [Windows](/cli/getting-started/quickstart/windows) --- Source: https://www.cloudquery.io/docs/cli/getting-started/quickstart/linux --- Source: https://www.cloudquery.io/docs/cli/getting-started/quickstart/macOS --- Source: https://www.cloudquery.io/docs/cli/getting-started/quickstart/windows --- Source: https://www.cloudquery.io/docs/cli/getting-support/faq # CLI FAQ ## Installation and Setup ### How do I install CloudQuery CLI? CloudQuery CLI runs on macOS, Linux, and Windows. See the getting started guide for your platform: - [macOS](/cli/getting-started/quickstart/macOS) - [Linux](/cli/getting-started/quickstart/linux) - [Windows](/cli/getting-started/quickstart/windows) ### Do I need to authenticate with CloudQuery? You only need to run `cloudquery login` if you're using: - Commercial or premium integrations - CloudQuery Platform - Private integrations from GitHub Free community integrations don't require authentication. ### Do I need a license for air-gapped or offline environments? Yes. If you're using CloudQuery in an air-gapped environment, you'll need an offline license for commercial integrations. Contact our [sales team](https://www.cloudquery.io/pricing) to obtain an offline license file. ## Configuration ### How do I configure a sync? Syncs are defined in YAML configuration files. Each file specifies a source integration (where data comes from), a destination integration (where data goes), and which tables to sync. See the [configuration guide](/cli/core-concepts/configuration) for the full reference. ### What write modes are available? There are three write modes for destination integrations: - **`overwrite-delete-stale`** (default): Upserts rows based on primary keys and deletes stale data from the previous sync with the same source configuration. - **`overwrite`**: Upserts rows based on primary keys but never deletes old data. You'll need to handle stale data cleanup yourself. - **`append`**: Adds new rows on every sync. Old rows are never deleted or updated. In `overwrite` and `append` mode, you can distinguish rows from different syncs by inspecting the `_cq_sync_time` column. ### Can I run multiple syncs in parallel? Yes, but each integration configuration must have a unique `name`. If names aren't unique, different integrations may overwrite each other's data. No two integrations should fetch the same account, table, and region combination. See [Running in Parallel](/cli/managing-cloudquery/running-in-parallel) for details. ## Data and Security ### Does CloudQuery access my application data? No. CloudQuery cloud provider integrations like AWS, GCP, and Azure generally only access metadata and configuration data. Some tables like `aws_cloudwatch_metrics` and `aws_cloudwatch_logs` can sync log and metric data, but only if you select those tables. ### Where does my data go? Your data never leaves your infrastructure. CloudQuery CLI runs on your systems and sends data only to the destinations you configure (PostgreSQL, BigQuery, Snowflake, S3, etc.). No cloud data is sent to CloudQuery servers. We collect only: - Anonymous usage statistics (can be disabled via [telemetry settings](/cli/advanced/telemetry)) - Error reports to help improve the product - Licensing validation data for commercial integrations ### What permissions does CloudQuery need? CloudQuery requires read-only access to your cloud resources. We recommend: - **AWS**: IAM roles with read-only policies - **GCP**: Service accounts with Viewer role - **Azure**: Service principals with Reader role ## Performance ### Why is my sync slow? Large cloud estates take time to sync. A few things to try: - Specify table names instead of wildcards (`*`) - Skip slow tables you don't need (see the [skip_tables list](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs) for AWS) - Adjust concurrency settings in your configuration file - See the [performance tuning guide](/cli/advanced/performance-tuning) for more options If syncs are running locally and seem to stall, it could be DNS-related. Large integrations like AWS make a DNS query per region per service. Less performant DNS servers (like home routers) may not keep up. Try pointing your machine to a [public DNS server](https://developers.google.com/speed/public-dns). For more troubleshooting steps, see the [Troubleshooting](/cli/getting-support/troubleshooting) page. ### What should I do if I get authentication errors? Common solutions: - Verify your credentials are correct and have read-only permissions - Check that your authentication method is properly configured - Ensure your credentials haven't expired - Run with `--log-level debug` for detailed error information ## Integrations ### Which cloud providers are supported? CloudQuery supports 70+ integrations including: - **Cloud Providers**: AWS, Google Cloud, Azure, DigitalOcean, Linode - **SaaS Applications**: GitHub, GitLab, Slack, Jira, Salesforce - **Databases**: PostgreSQL, MySQL, MongoDB, Redis - **Security Tools**: Okta, Auth0, CrowdStrike, SentinelOne See all available integrations at [www.cloudquery.io/hub](https://www.cloudquery.io/hub/plugins/source). ### Can I sync data from multiple accounts/regions? Yes! You can sync from multiple: - AWS accounts and regions - GCP projects and regions - Azure subscriptions and regions Each configuration should have a unique name and target different resources. ### What destinations are supported? CloudQuery supports 50+ destinations including: - **Databases**: PostgreSQL, MySQL, SQLite, ClickHouse - **Data Warehouses**: BigQuery, Snowflake, Redshift, Databricks - **Data Lakes**: S3, GCS, Azure Blob Storage - **Analytics**: Elasticsearch, Grafana, Tableau See all destinations at [www.cloudquery.io/hub](https://www.cloudquery.io/hub/plugins/destination). ## Getting Help ### Where can I get help? - **Community Forum**: [community.cloudquery.io](https://community.cloudquery.io) - **GitHub Issues**: [github.com/cloudquery/cloudquery](https://github.com/cloudquery/cloudquery) - **CLI Documentation**: Browse the sidebar for guides on configuration, syncs, and advanced topics - **Platform FAQ**: For questions about CloudQuery Platform, see the [Platform FAQ](/platform/introduction/faq) ### How do I report a bug? 1. Check if the issue already exists on [GitHub](https://github.com/cloudquery/cloudquery/issues) 2. If not, create a new issue with: - Steps to reproduce the problem - Expected vs actual behavior - Logs with `--log-level debug` - Your configuration file (remove sensitive data) ## Next Steps - [Troubleshooting](/cli/getting-support/troubleshooting) - Debug common sync issues - [Getting Started](/cli/getting-started) - Install and run your first sync - [CLI vs Platform](/cli/cli-vs-platform) - Compare CLI and Platform options --- Source: https://www.cloudquery.io/docs/cli/getting-support # Getting Support CloudQuery offers multiple ways to get help and support, from our community to dedicated support tiers for enterprise customers. ## Community Support Our community is the best place to start for most questions and issues. Join thousands of CloudQuery users sharing knowledge, solutions, and best practices. [Visit CloudQuery Community →](https://community.cloudquery.io/) ## Support Tiers ### Community Support (Free) - Access to our community forum - Community-driven answers and solutions - Public GitHub issues and discussions - Documentation and troubleshooting guides ### Business Support - Private support channels - Direct access to CloudQuery team - SLA-backed response times - Best practices guidance ### Enterprise Support - Dedicated support channels - SLA-backed response times - Custom integration assistance - Dedicated technical account manager time - Architecture reviews and optimization ## Self-Service Resources ### Telemetry Guide Learn about CloudQuery's telemetry data collection, how to configure it, and how to use it for debugging. For performance issues, see [performance tuning](/cli/advanced/performance-tuning). For configuration problems, check the [configuration guide](/cli/core-concepts/configuration). [View Telemetry Guide →](/cli/advanced/telemetry) ### FAQ Frequently asked questions covering common CloudQuery topics and use cases. [View FAQ →](/cli/getting-support/faq) ### Open Source Contribution Want to contribute to CloudQuery? Check out our contribution guides and help improve the project. [Contribute to CloudQuery →](https://github.com/cloudquery/cloudquery/tree/main/contributing) ## Getting Started If you're new to CloudQuery, start with our getting started guides: - [macOS Getting Started](../getting-started/quickstart/macOS) - [Linux Getting Started](../getting-started/quickstart/linux) - [Windows Getting Started](../getting-started/quickstart/windows) ## Service Status Before opening a support ticket, check the [CloudQuery Status Page](https://status.cloudquery.io/) for any ongoing incidents or scheduled maintenance that may be affecting CloudQuery services. ## Still Need Help? If you can't find what you're looking for in our documentation or community: 1. **Check service status** at [status.cloudquery.io](https://status.cloudquery.io/) for known incidents 2. **Search existing issues** on [GitHub](https://github.com/cloudquery/cloudquery/issues) 3. **Post in our community** at [community.cloudquery.io](https://community.cloudquery.io/) 4. **Open a GitHub issue** for bugs or feature requests 5. **Contact our support team** for enterprise customers ## Next Steps - [Troubleshooting](/cli/getting-support/troubleshooting) - Common issues and solutions - [FAQ](/cli/getting-support/faq) - Frequently asked questions - [Getting Started](/cli/getting-started) - Install and run your first sync --- Source: https://www.cloudquery.io/docs/cli/getting-support/troubleshooting # Troubleshooting ## Help Channels ### Service Status If you're experiencing connection timeouts, API errors, or unexpected failures, first check the [CloudQuery Status Page](https://status.cloudquery.io/) for any ongoing incidents or maintenance that may be affecting CloudQuery services. ### CloudQuery Community Join our [Community](https://community.cloudquery.io)! ### GitHub Issues There are a couple of ways to get help for any CloudQuery-related issues or questions. 1. Check out previous issues at [https://github.com/cloudquery/cloudquery](https://github.com/cloudquery/cloudquery) and open a new one if no previous one has been opened or resolved. 2. Check out previous threads on our [Community](https://community.cloudquery.io) and open a new one if no previous one matches your issue. ## Debugging ### Verbose Logging To debug an issue, start by running `cloudquery` with `--log-level debug` to enable verbose logging. The CLI may print out environment variables, including sensitive information, when verbose logging is enabled. ### Error: "failed to migrate source"… If you see an error such as `failed to migrate source`, it means that, while upgrading an integration, the migration of the SQL schema failed. CloudQuery makes a best-effort attempt to automatically and transparently manage the schemas of integrations, but this can sometimes fail during version upgrades. The easiest solution is to drop and recreate the database or schema (or less destructively, all the integration's tables, such as `aws_*`). ### I am running `cloudquery sync` with multiple source integrations (or multiple `cloudquery sync`s), but some tables are empty / some rows are duplicated When running `cloudquery sync` with multiple source integrations (or multiple `cloudquery sync`s in parallel), it is important that every integration configuration has a unique `name`. If the names are not unique, the different integrations may overwrite/delete each others data. It is also important that every integration configuration fetches different data (i.e. no two integrations are fetching the same account/table/region combination). You can read more about this [here](/cli/managing-cloudquery/running-in-parallel). ### My AWS sync is taking a long time. What can I do to speed it up? A few specific tables in AWS are quite slow to sync. You can try skipping them if you don't need this data. Take a look at the [skip_tables list](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs). If syncs are still taking a long time, you can also take a look at our [performance tuning guide](/cli/advanced/performance-tuning). ### I am running `cloudquery sync` locally, but it is taking a long time / doesn't seem to finish (If running the AWS integrations, try the `skip_tables` solution from the previous paragraph first). Large CloudQuery integrations, such as AWS, make a lot of DNS queries (for example, the AWS integrations makes a DNS query per `num_regions * num_services`). Some less performant DNS servers may not be able to handle this load (e.g. home routers). You can try pointing your machine to a different DNS server (such as the reliable [Google DNS Server](https://developers.google.com/speed/public-dns)). ## Next Steps - [FAQ](/cli/getting-support/faq) - Frequently asked questions - [Monitoring](/cli/managing-cloudquery/monitoring/overview) - Set up observability to diagnose issues - [Performance Tuning](/cli/advanced/performance-tuning) - Resolve sync performance problems --- Source: https://www.cloudquery.io/docs/cli # CloudQuery CLI The CloudQuery CLI is a self-hosted, open-source tool that syncs cloud infrastructure data into your own databases and data warehouses. You run it on your own infrastructure with full control over configuration, scheduling, and data storage. **Looking for the managed platform?** [CloudQuery Platform](/platform/introduction) provides a web UI, built-in asset inventory, policies, automations, and AI-powered analysis. No infrastructure to manage. See [Platform vs CLI](/platform/introduction/platform-vs-cli) for a full comparison. ## How It Works The CLI uses an integration-based architecture. You configure [source integrations](https://www.cloudquery.io/hub/plugins/source) (AWS, GCP, Azure, GitHub, Okta, and hundreds more) to extract cloud configuration data, and [destination integrations](https://www.cloudquery.io/hub/plugins/destination) (PostgreSQL, BigQuery, Snowflake, S3, and others) to load it. Configuration is defined in YAML files and syncs are triggered via the command line. ``` cloudquery sync config.yml ``` Once synced, you query your infrastructure directly with SQL in whatever database or warehouse you chose. ## Get Started - [CLI Getting Started](/cli/getting-started) - Install the CLI and run your first sync - [AWS to PostgreSQL](/cli/getting-started/aws-to-postgresql) - Sync your AWS infrastructure into PostgreSQL, the most common production setup - [Platform vs CLI](/platform/introduction/platform-vs-cli) - Understand the differences and decide which path is right for your team ## Documentation - [Core Concepts](/cli/core-concepts/integrations) - Integrations, syncs, and destinations - [Configuration](/cli/core-concepts/configuration) - Configure your syncs - [Available Integrations](/cli/integrations) - Browse source and destination integrations - [Managing CloudQuery](/cli/managing-cloudquery/deployments/overview) - Deployment, monitoring, and upgrades - [CLI Reference](/cli/cli-reference/cloudquery) - Command reference ## Next Steps - Learn the CloudQuery [core concepts](/cli/core-concepts/integrations) - Explore [available integrations](/cli/integrations) for your data sources and destinations - Read how to build an [AWS Cloud Asset Inventory](https://www.cloudquery.io/blog/building-cloud-asset-inventory-with-aws) - Browse [step-by-step tutorials](https://www.cloudquery.io/blog/category/tutorials) - Join the [CloudQuery Community](https://community.cloudquery.io/) --- Source: https://www.cloudquery.io/docs/cli/integrations/creating-new-integration/go-destination # Creating a New Destination Integration in Go This guide builds a SQLite destination integration from scratch in stages. At each stage you'll have something you can actually run, starting with a gRPC server that serves but does nothing, then adding the ability to create tables, then write rows, then handle schema changes. The same patterns apply to any destination backend: PostgreSQL, BigQuery, a custom database, or an external API that accepts writes. The [official SQLite destination](https://github.com/cloudquery/cloudquery/tree/main/plugins/destination/sqlite) follows this exact structure and is a good reference once you've finished this guide. Before starting, review [core concepts](/cli/integrations/creating-new-integration#core-concepts) and the [Creating a New Integration overview](/cli/integrations/creating-new-integration). The terminology (tables, syncs, write modes) is introduced there. **Prerequisites:** - Go 1.21+ installed ([Go Tutorial](https://go.dev/doc/tutorial/getting-started)) - CloudQuery CLI installed - No CGO or C compiler required for this guide ## How Destinations Differ from Sources A source integration _produces_ data: it fetches records from a third-party API and sends them to CloudQuery. A destination integration _consumes_ data: it receives a stream of write messages from the CloudQuery engine and stores them somewhere. When a user runs `cloudquery sync`, data flows into your destination like this: ```text cloudquery sync │ ▼ [CloudQuery Engine] ──▶ Write(ctx, res <-chan WriteMessage) │ [batchwriter] ← SDK helper: buffers and routes ┌─────────┼──────────────────┐ ▼ ▼ ▼ MigrateTables() WriteTableBatch() DeleteStale() before rows batched rows after rows │ │ │ └────┬────┴───────────────────┘ ▼ [Your Database] ``` Your destination receives one call to `Write`. You pass that channel straight to the SDK's **batchwriter** helper, which takes care of buffering and routing. It calls three methods on your client: | Method | When called | What it does | |---|---|---| | `MigrateTables` | Before rows for any table | Create or update the table schema | | `WriteTableBatch` | When a batch of rows is ready | Insert or upsert the rows | | `DeleteStale` | After rows, in `overwrite-delete-stale` mode | Remove rows older than this sync | You'll implement these one at a time, building up the destination's capabilities as you go. ## Project Structure ```text cq-destination-sqlite/ ├── main.go ├── go.mod ├── resources/ │ └── plugin/ │ └── plugin.go # Name, Kind, Team, Version constants └── client/ ├── client.go # Client struct, constructor, Write ├── spec.go # Configuration spec ├── migrate.go # MigrateTables ├── write.go # WriteTableBatch, DeleteRecord └── deletestale.go # DeleteStale ``` ## Scaffold the Integration There is no `cq-scaffold` equivalent for destination integrations; you'll create the files manually. There aren't many of them, and this guide walks through each one. Build the minimum needed to get a gRPC server running. By the end of this section you can start your destination and CloudQuery can connect to it, even though it won't create tables or write data yet. ### Integration Constants ```go filename="resources/plugin/plugin.go" package plugin var ( Name = "sqlite" Kind = "destination" Team = "your-team" Version = "development" ) ``` `Kind` must be `"destination"`. These values appear in log output and on the CloudQuery Hub. ### Configuration Spec ```go filename="client/spec.go" package client const ( defaultBatchSize = int64(10_000) defaultBatchSizeBytes = int64(10 * 1024 * 1024) // 10 MiB ) // Spec holds the user-provided configuration for this destination. type Spec struct { // Path to the SQLite file, e.g. "./mydb.db" ConnectionString string `json:"connection_string"` // Maximum rows per write batch. Default: 10,000. BatchSize int64 `json:"batch_size,omitempty"` // Maximum bytes per write batch. Default: 10 MiB. BatchSizeBytes int64 `json:"batch_size_bytes,omitempty"` } func (s *Spec) SetDefaults() { if s.BatchSize == 0 { s.BatchSize = defaultBatchSize } if s.BatchSizeBytes == 0 { s.BatchSizeBytes = defaultBatchSizeBytes } } func (s *Spec) Validate() error { if s.ConnectionString == "" { return fmt.Errorf("connection_string is required") } return nil } ``` A user's destination YAML block looks like: ```yaml kind: destination spec: name: my-sqlite registry: grpc path: localhost:7777 spec: connection_string: ./cloudquery.db ``` ### Client The `Client` struct holds your database connection and the batchwriter instance. `Write` delegates to the batchwriter and doesn't change as you add features. ```go filename="client/client.go" package client "context" "database/sql" "encoding/json" "fmt" "github.com/apache/arrow-go/v18/arrow" "github.com/cloudquery/plugin-sdk/v4/message" "github.com/cloudquery/plugin-sdk/v4/plugin" "github.com/cloudquery/plugin-sdk/v4/schema" "github.com/cloudquery/plugin-sdk/v4/writers/batchwriter" "github.com/rs/zerolog" // Pure-Go SQLite driver (no C compiler required). // Registers the "sqlite" driver name with database/sql. _ "modernc.org/sqlite" ) type Client struct { plugin.UnimplementedSource db *sql.DB logger zerolog.Logger spec Spec writer *batchwriter.BatchWriter } // Read satisfies the plugin.Client interface. Return an error if your // destination doesn't need to support reading rows back. func (c *Client) Read(_ context.Context, _ *schema.Table, _ chan<- arrow.RecordBatch) error { return fmt.Errorf("read not supported by this destination") } // New is the constructor passed to plugin.NewPlugin. func New(_ context.Context, logger zerolog.Logger, specBytes []byte, _ plugin.NewClientOptions) (plugin.Client, error) { c := &Client{ logger: logger.With().Str("module", "sqlite-dest").Logger(), } if err := json.Unmarshal(specBytes, &c.spec); err != nil { return nil, fmt.Errorf("failed to unmarshal spec: %w", err) } c.spec.SetDefaults() if err := c.spec.Validate(); err != nil { return nil, err } var err error c.writer, err = batchwriter.New(c, batchwriter.WithLogger(c.logger), batchwriter.WithBatchSize(c.spec.BatchSize), batchwriter.WithBatchSizeBytes(c.spec.BatchSizeBytes), ) if err != nil { return nil, fmt.Errorf("failed to create batchwriter: %w", err) } // modernc.org/sqlite uses the driver name "sqlite" (vs "sqlite3" for mattn/go-sqlite3). c.db, err = sql.Open("sqlite", c.spec.ConnectionString) if err != nil { return nil, fmt.Errorf("failed to open database: %w", err) } return c, nil } func (c *Client) Close(_ context.Context) error { if c.db == nil { return fmt.Errorf("client already closed") } err := c.db.Close() c.db = nil return err } // Write is called once per sync. It passes the message channel to the // batchwriter, which routes messages to MigrateTables, WriteTableBatch, // and DeleteStale. Always call Flush; the batchwriter may hold buffered // rows that haven't been flushed yet. func (c *Client) Write(ctx context.Context, res <-chan message.WriteMessage) error { if err := c.writer.Write(ctx, res); err != nil { return fmt.Errorf("failed to write: %w", err) } if err := c.writer.Flush(ctx); err != nil { return fmt.Errorf("failed to flush: %w", err) } return nil } ``` **Why `modernc.org/sqlite`?** The official CloudQuery SQLite destination uses [`github.com/mattn/go-sqlite3`](https://github.com/mattn/go-sqlite3), which requires a C compiler. This guide uses [`modernc.org/sqlite`](https://pkg.go.dev/modernc.org/sqlite), a pure-Go port that compiles with a standard `go build` on any platform. The API is identical; only the import path and driver name differ. ### Starter Stubs Create these two files now. They give the batchwriter valid methods to call so the whole thing compiles. You'll replace them with real implementations in the next steps. ```go filename="client/migrate.go" package client "context" "github.com/cloudquery/plugin-sdk/v4/message" ) func (c *Client) MigrateTables(_ context.Context, _ message.WriteMigrateTables) error { return nil // Step 2: create tables } ``` ```go filename="client/write.go" package client "context" "fmt" "github.com/cloudquery/plugin-sdk/v4/message" ) func (c *Client) WriteTableBatch(_ context.Context, _ string, _ message.WriteInserts) error { return nil // Step 3: write rows } func (c *Client) DeleteStale(_ context.Context, _ message.WriteDeleteStales) error { return nil // Step 4: delete stale rows } func (c *Client) DeleteRecord(_ context.Context, _ message.WriteDeleteRecords) error { return fmt.Errorf("DeleteRecord is not supported by this destination") } ``` ### Entry Point ```go filename="main.go" package main "context" "log" "github.com/cloudquery//cq-destination-sqlite/client" internalPlugin "github.com/cloudquery//cq-destination-sqlite/resources/plugin" "github.com/cloudquery/plugin-sdk/v4/plugin" "github.com/cloudquery/plugin-sdk/v4/serve" ) func main() { p := plugin.NewPlugin( internalPlugin.Name, internalPlugin.Version, client.New, plugin.WithKind(internalPlugin.Kind), // "destination" plugin.WithTeam(internalPlugin.Team), ) if err := serve.Plugin(p, serve.WithDestinationV0V1Server()).Serve(context.Background()); err != nil { log.Fatalf("failed to serve plugin: %v", err) } } ``` `serve.WithDestinationV0V1Server()` adds gRPC endpoints compatible with older source integration protocol versions. Include it in every destination for broad compatibility. ### go.mod ```text filename="go.mod" module github.com//cq-destination-sqlite go 1.21 require ( github.com/apache/arrow-go/v18 v18.0.0 github.com/cloudquery/plugin-sdk/v4 v4.0.0 github.com/rs/zerolog v1.33.0 modernc.org/sqlite v1.34.0 ) ``` The versions above are placeholders. Fetch the latest before running: ```bash go get github.com/cloudquery/plugin-sdk/v4@latest go get modernc.org/sqlite@latest go mod tidy ``` ### Run It ```bash go mod tidy go build -o cq-destination-sqlite . ./cq-destination-sqlite serve ``` You should see: ``` Plugin server listening address=127.0.0.1:7777 plugin=your-team/destination/sqlite@development ``` Your destination is running. CloudQuery can connect to it. It won't create tables or write data yet. That's what the next two sections add. ## Create Tables Implement `MigrateTables` so your destination creates a table the first time rows arrive for it. `MigrateTables` is called once per table before any rows are written. At a minimum, it needs to create the table if it doesn't exist. Replace `client/migrate.go` with this: ```go filename="client/migrate.go" package client "context" "fmt" "strings" "github.com/apache/arrow-go/v18/arrow" "github.com/cloudquery/plugin-sdk/v4/message" "github.com/cloudquery/plugin-sdk/v4/schema" ) func (c *Client) MigrateTables(ctx context.Context, msgs message.WriteMigrateTables) error { for _, msg := range msgs { table := msg.Table c.logger.Info().Str("table", table.Name).Msg("Migrating table") exists, err := c.tableExists(table.Name) if err != nil { return err } if !exists { if err := c.createTable(ctx, table); err != nil { return err } } } return nil } func (c *Client) tableExists(tableName string) (bool, error) { rows, err := c.db.Query(fmt.Sprintf("PRAGMA table_info('%s')", tableName)) if err != nil { return false, err } defer rows.Close() return rows.Next(), rows.Err() } func (c *Client) createTable(ctx context.Context, table *schema.Table) error { var sb strings.Builder sb.WriteString(`CREATE TABLE IF NOT EXISTS "`) sb.WriteString(table.Name) sb.WriteString(`" (`) var primaryKeys []string for i, col := range table.Columns { sb.WriteString(`"` + col.Name + `" `) sb.WriteString(arrowTypeToSQLite(col.Type)) if col.NotNull { sb.WriteString(" NOT NULL") } if i < len(table.Columns)-1 { sb.WriteString(", ") } if col.PrimaryKey { primaryKeys = append(primaryKeys, `"`+col.Name+`"`) } } // Composite primary key: SQLite requires this syntax when there are multiple keys. if len(primaryKeys) > 0 { sb.WriteString(`, CONSTRAINT "`) sb.WriteString(table.Name) sb.WriteString(`_cqpk" PRIMARY KEY (`) sb.WriteString(strings.Join(primaryKeys, ", ")) sb.WriteString(")") } sb.WriteString(")") _, err := c.db.ExecContext(ctx, sb.String()) return err } // arrowTypeToSQLite maps an Arrow DataType to a SQLite storage class. // SQLite has four storage classes: TEXT, INTEGER, REAL, BLOB. func arrowTypeToSQLite(t arrow.DataType) string { switch t.ID() { case arrow.INT8, arrow.INT16, arrow.INT32, arrow.INT64, arrow.UINT8, arrow.UINT16, arrow.UINT32, arrow.UINT64, arrow.BOOL: return "INTEGER" case arrow.FLOAT32, arrow.FLOAT64: return "REAL" case arrow.BINARY, arrow.LARGE_BINARY: return "BLOB" case arrow.TIMESTAMP, arrow.DATE32, arrow.DATE64, arrow.TIME32, arrow.TIME64: return "TIMESTAMP" default: // UTF8, LARGE_UTF8, lists, structs, maps → TEXT return "TEXT" } } ``` ### Try It Start your destination, then run a sync against any source. Here's a config using the public xkcd source (no API key needed): ```yaml filename="config.yaml" kind: source spec: name: xkcd path: cloudquery/xkcd registry: cloudquery version: "v1.5.38" tables: ["*"] destinations: - my-sqlite --- kind: destination spec: name: my-sqlite registry: grpc path: localhost:7777 spec: connection_string: ./test.db ``` ```bash ./cq-destination-sqlite serve & cloudquery sync config.yaml sqlite3 test.db ".tables" # → xkcd_comics ``` The table exists. No rows yet: `WriteTableBatch` still returns `nil` without writing anything. That's what the next section adds. ## Write Rows Implement `WriteTableBatch` so rows actually land in the database. When the batchwriter has accumulated enough rows for a table (up to `batch_size`), it calls `WriteTableBatch` with all the accumulated `WriteInsert` messages. You get the table name and the messages. Each message contains an Apache Arrow `RecordBatch` (a columnar block of rows). Replace `client/write.go` with this: ```go filename="client/write.go" package client "context" "database/sql" "fmt" "strings" "github.com/apache/arrow-go/v18/arrow" "github.com/apache/arrow-go/v18/arrow/array" "github.com/cloudquery/plugin-sdk/v4/message" "github.com/cloudquery/plugin-sdk/v4/schema" ) func (c *Client) WriteTableBatch(ctx context.Context, name string, msgs message.WriteInserts) error { if len(msgs) == 0 { return nil } // Wrap all inserts in a single transaction (much faster than one per row). tx, err := c.db.BeginTx(ctx, nil) if err != nil { return err } defer func() { if err != nil { if rbErr := tx.Rollback(); rbErr != nil { c.logger.Error().Err(rbErr).Str("table", name).Msg("rollback failed") } } }() for _, msg := range msgs { if err = c.insertRows(ctx, tx, msg); err != nil { return err } } return tx.Commit() } func (c *Client) insertRows(ctx context.Context, tx *sql.Tx, msg *message.WriteInsert) error { table := msg.GetTable() sc := msg.Record.Schema() // Tables with primary keys use INSERT OR REPLACE (upsert). // Tables without primary keys use plain INSERT (append). var sqlStr string if len(table.PrimaryKeys()) == 0 { sqlStr = buildInsert(sc) } else { sqlStr = buildUpsert(sc) } for _, row := range recordToRows(msg.Record) { if _, err := tx.ExecContext(ctx, sqlStr, row...); err != nil { return fmt.Errorf("insert failed for table %s: %w", table.Name, err) } } return nil } func buildInsert(sc *arrow.Schema) string { return buildSQL("insert into", sc) } // buildUpsert uses INSERT OR REPLACE, SQLite's upsert syntax. // On a primary key conflict it deletes the old row and inserts the new one. func buildUpsert(sc *arrow.Schema) string { return buildSQL("insert or replace into", sc) } func buildSQL(verb string, sc *arrow.Schema) string { tableName, _ := sc.Metadata().GetValue(schema.MetadataTableName) var sb strings.Builder sb.WriteString(verb + ` "` + tableName + `" (`) fields := sc.Fields() for i, f := range fields { sb.WriteString(`"` + f.Name + `"`) if i < len(fields)-1 { sb.WriteString(", ") } } sb.WriteString(") VALUES (") for i := range fields { sb.WriteString(fmt.Sprintf("$%d", i+1)) if i < len(fields)-1 { sb.WriteString(", ") } } sb.WriteString(")") return sb.String() } func (c *Client) DeleteStale(_ context.Context, _ message.WriteDeleteStales) error { return nil // Step 4: implement stale data deletion } func (c *Client) DeleteRecord(_ context.Context, _ message.WriteDeleteRecords) error { return fmt.Errorf("DeleteRecord is not supported by this destination") } ``` ### Arrow Records to Rows CloudQuery passes row data as Apache Arrow `RecordBatch` values, a columnar in-memory format. To call `sql.ExecContext`, you need to convert from columnar Arrow format to row-oriented `[]any` slices. Add these helpers at the bottom of `client/write.go`: ```go filename="client/write.go" // recordToRows converts a columnar Arrow RecordBatch into row-oriented slices // for sql.ExecContext. func recordToRows(rec arrow.RecordBatch) [][]any { numRows := int(rec.NumRows()) numCols := int(rec.NumCols()) rows := make([][]any, numRows) for i := range rows { rows[i] = make([]any, numCols) } for colIdx, col := range rec.Columns() { for rowIdx := 0; rowIdx < numRows; rowIdx++ { if col.IsNull(rowIdx) { rows[rowIdx][colIdx] = nil continue } rows[rowIdx][colIdx] = arrowValueAt(col, rowIdx) } } return rows } // arrowValueAt extracts a single value from an Arrow array as a Go-native type // suitable for use as a SQL parameter. func arrowValueAt(col arrow.Array, i int) any { switch c := col.(type) { case *array.String: return c.Value(i) case *array.LargeString: return c.Value(i) case *array.Int8: return int64(c.Value(i)) case *array.Int16: return int64(c.Value(i)) case *array.Int32: return int64(c.Value(i)) case *array.Int64: return c.Value(i) case *array.Uint8: return int64(c.Value(i)) case *array.Uint16: return int64(c.Value(i)) case *array.Uint32: return int64(c.Value(i)) case *array.Uint64: return int64(c.Value(i)) case *array.Float32: return float64(c.Value(i)) case *array.Float64: return c.Value(i) case *array.Boolean: if c.Value(i) { return int64(1) } return int64(0) case *array.Binary: return c.Value(i) case *array.LargeBinary: return c.Value(i) default: // For complex types (timestamps, lists, maps, structs) fall back // to the string representation. Add explicit cases as needed. return col.ValueStr(i) } } ``` {/* vale off */} CloudQuery also uses custom Arrow extension types (UUID, MAC address, Inet, etc.) on top of the standard types. The `default` case above handles them via `ValueStr`. For production use, see the [official write.go](https://github.com/cloudquery/cloudquery/blob/main/plugins/destination/sqlite/client/write.go) for the complete mapping. {/* vale on */} ### Try It Rebuild and run the sync again: ```bash go build -o cq-destination-sqlite . && ./cq-destination-sqlite serve & cloudquery sync config.yaml ``` Verify rows landed: ```bash # Check total row count (should be 3,000+ comics) sqlite3 test.db "SELECT COUNT(*) FROM xkcd_comics;" # Spot-check the most recent entries sqlite3 test.db "SELECT num, title FROM xkcd_comics ORDER BY num DESC LIMIT 5;" ``` Expected output: ``` 3228 3228|Sunscreen Misconceptions 3227|Research Ethics 3226|Deepest Lake 3225|Sitting Room 3224|Lossless ``` Your destination is now fully functional for the most common case: append and overwrite write modes. ## Advanced: Schema Migration The `MigrateTables` implementation in Step 2 creates tables but doesn't handle changes to an existing schema. If a new column is added to a source, or you change a column type, the destination needs to either add the column safely or force-recreate the table. Replace the entire contents of `client/migrate.go` with this full implementation: ```go filename="client/migrate.go" package client "context" "fmt" "strings" "github.com/apache/arrow-go/v18/arrow" "github.com/cloudquery/plugin-sdk/v4/message" "github.com/cloudquery/plugin-sdk/v4/schema" ) func (c *Client) MigrateTables(ctx context.Context, msgs message.WriteMigrateTables) error { for _, msg := range msgs { table := msg.Table c.logger.Info().Str("table", table.Name).Msg("Migrating table") existing, err := c.getExistingColumns(table.Name) if err != nil { return err } if existing == nil { // Table doesn't exist: create it. if err := c.createTable(ctx, table); err != nil { return err } continue } // Table exists. Compute what changed. changes := existing.GetChanges(table) if len(changes) == 0 { continue // Nothing to do. } if msg.MigrateForce { // Force mode: drop and recreate. if err := c.recreateTable(ctx, table); err != nil { return err } continue } // Safe mode: only apply additive, non-breaking changes. if !canAutoMigrate(changes) { return fmt.Errorf("cannot safely migrate table %s: breaking changes detected. "+ "Use migrate_mode: forced to allow destructive migrations.\n%s", table.Name, schema.GetChangesSummary(map[string][]schema.TableColumnChange{table.Name: changes})) } if err := c.autoMigrate(ctx, table, changes); err != nil { return err } } return nil } // canAutoMigrate returns true if all changes can be applied without data loss. func canAutoMigrate(changes []schema.TableColumnChange) bool { for _, change := range changes { switch change.Type { case schema.TableColumnChangeTypeAdd: if change.Current.PrimaryKey || change.Current.NotNull { return false } case schema.TableColumnChangeTypeRemove: if change.Previous.PrimaryKey || change.Previous.NotNull { return false } case schema.TableColumnChangeTypeRemoveUniqueConstraint: // Safe. default: return false } } return true } func (c *Client) recreateTable(ctx context.Context, table *schema.Table) error { if _, err := c.db.ExecContext(ctx, `DROP TABLE IF EXISTS "`+table.Name+`"`); err != nil { return fmt.Errorf("failed to drop table %s: %w", table.Name, err) } return c.createTable(ctx, table) } func (c *Client) autoMigrate(ctx context.Context, table *schema.Table, changes []schema.TableColumnChange) error { for _, change := range changes { if change.Type == schema.TableColumnChangeTypeAdd { sql := fmt.Sprintf(`ALTER TABLE "%s" ADD COLUMN "%s" %s`, table.Name, change.Current.Name, arrowTypeToSQLite(change.Current.Type)) if _, err := c.db.ExecContext(ctx, sql); err != nil { return err } } } return nil } // getExistingColumns reads the live schema from SQLite using PRAGMA table_info. // Returns nil if the table does not exist. func (c *Client) getExistingColumns(tableName string) (*schema.Table, error) { rows, err := c.db.Query(fmt.Sprintf("PRAGMA table_info('%s')", tableName)) if err != nil { return nil, err } defer rows.Close() var columns []schema.Column for rows.Next() { var cid int var name, typ string var notNull bool var dflt any var pk int if err := rows.Scan(&cid, &name, &typ, ¬Null, &dflt, &pk); err != nil { return nil, err } columns = append(columns, schema.Column{ Name: name, Type: sqliteTypeToArrow(strings.ToLower(typ)), NotNull: notNull, PrimaryKey: pk != 0, }) } if err := rows.Err(); err != nil { return nil, err } if len(columns) == 0 { return nil, nil // Table does not exist. } return &schema.Table{Name: tableName, Columns: columns}, nil } // sqliteTypeToArrow maps PRAGMA type strings back to Arrow types. func sqliteTypeToArrow(typ string) arrow.DataType { switch typ { case "integer": return arrow.PrimitiveTypes.Int64 case "real": return arrow.PrimitiveTypes.Float64 case "blob": return arrow.BinaryTypes.Binary case "timestamp": return arrow.FixedWidthTypes.Timestamp_us default: return arrow.BinaryTypes.String } } // createTable and arrowTypeToSQLite remain unchanged from Step 2. ``` {/* vale off */} The official SQLite destination has fuller type handling: it normalizes Arrow types before comparison and handles CloudQuery-specific extension types. See [migrate.go](https://github.com/cloudquery/cloudquery/blob/main/plugins/destination/sqlite/client/migrate.go) for the complete implementation. {/* vale on */} ### Try It Rebuild with the updated migrate.go and run a sync: ```bash go build -o cq-destination-sqlite . && ./cq-destination-sqlite serve & cloudquery sync config.yaml ``` Inspect the table schema to confirm all columns were created correctly: ```bash sqlite3 test.db ".schema xkcd_comics" ``` You should see all source columns plus the CloudQuery system columns (`_cq_id`, `_cq_source_name`, `_cq_sync_time`) with a composite primary key constraint: ```sql CREATE TABLE "xkcd_comics" ( "_cq_id" TEXT NOT NULL, "_cq_source_name" TEXT NOT NULL, "_cq_sync_time" TIMESTAMP NOT NULL, "num" INTEGER, "title" TEXT, "year" TEXT, ... CONSTRAINT "xkcd_comics_cqpk" PRIMARY KEY ("_cq_id") ); ``` **Verify safe migration mode:** Delete the database file and sync again to create a fresh table. Then sync a second time without deleting. No error means the "no changes" path is working: ```bash rm test.db cloudquery sync config.yaml # creates the table cloudquery sync config.yaml # should complete without error (no schema changes) ``` **Verify forced migration:** Add `migrate_mode: forced` to your destination spec, then sync. This triggers a drop-and-recreate even when no changes are needed, useful to confirm the force path runs without errors: ```yaml kind: destination spec: name: my-sqlite path: localhost:7777 registry: grpc migrate_mode: forced spec: connection_string: ./test.db ``` ```bash cloudquery sync config.yaml # drops and recreates xkcd_comics sqlite3 test.db "SELECT COUNT(*) FROM xkcd_comics;" # rows are back ``` ## Advanced: Delete Stale Data When a user configures `write_mode: overwrite-delete-stale`, the CloudQuery engine sends `WriteDeleteStale` messages after rows have been written. These identify rows that were present in a previous sync but are no longer in the source. The destination should delete them. Create `client/deletestale.go` and update `client/write.go` to remove the stub: ```go filename="client/deletestale.go" package client "context" "fmt" "github.com/cloudquery/plugin-sdk/v4/message" "github.com/cloudquery/plugin-sdk/v4/schema" ) func (c *Client) DeleteStale(ctx context.Context, msgs message.WriteDeleteStales) error { for _, msg := range msgs { sql := fmt.Sprintf( `DELETE FROM "%s" WHERE "%s" = $1 AND datetime(%s) < datetime($2)`, msg.TableName, schema.CqSourceNameColumn.Name, // "_cq_source_name" schema.CqSyncTimeColumn.Name, // "_cq_sync_time" ) if _, err := c.db.ExecContext(ctx, sql, msg.SourceName, msg.SyncTime); err != nil { return fmt.Errorf("failed to delete stale rows from %s: %w", msg.TableName, err) } } return nil } ``` The `_cq_source_name` and `_cq_sync_time` columns are added automatically to every CloudQuery table by the SDK. This query deletes rows from the current source whose sync time is older than the current sync, meaning anything that no longer exists upstream. Now remove the `DeleteStale` stub from `client/write.go`. If you leave it in, Go will fail with "method redeclared" because `DeleteStale` is now defined in two files. The end of `client/write.go` should have only `DeleteRecord`: ```go filename="client/write.go" // ... (WriteTableBatch, insertRows, buildInsert, buildUpsert, buildSQL, // recordToRows, arrowValueAt (all unchanged) // DeleteRecord is the only method that stays in write.go. // DeleteStale has moved to deletestale.go. func (c *Client) DeleteRecord(_ context.Context, _ message.WriteDeleteRecords) error { return fmt.Errorf("DeleteRecord is not supported by this destination") } ``` ### Try It Update your config to use `overwrite-delete-stale` mode. This tells the engine to send `WriteDeleteStale` messages after each sync: ```yaml filename="config.yaml" kind: source spec: name: xkcd path: cloudquery/xkcd registry: cloudquery version: "v1.5.38" tables: ["*"] destinations: - my-sqlite --- kind: destination spec: name: my-sqlite registry: grpc path: localhost:7777 write_mode: overwrite-delete-stale spec: connection_string: ./test.db ``` Rebuild and sync twice: ```bash go build -o cq-destination-sqlite . && ./cq-destination-sqlite serve & # First sync: populates the table cloudquery sync config.yaml sqlite3 test.db "SELECT COUNT(*) FROM xkcd_comics;" # → 3228 (or however many exist today) # Second sync: overwrites existing rows and deletes any that are no longer in the source cloudquery sync config.yaml sqlite3 test.db "SELECT COUNT(*) FROM xkcd_comics;" # → 3228 (same count; rows were overwritten, not duplicated or deleted) ``` The count should be identical after both syncs. If rows were duplicating instead, you'd be missing `DeleteStale`. If rows were disappearing, your `datetime()` comparison in the SQL query is off. To confirm `_cq_sync_time` is being updated on each sync (which is what makes deletion work correctly): ```bash sqlite3 test.db "SELECT num, _cq_sync_time FROM xkcd_comics ORDER BY num LIMIT 3;" ``` After the second sync, `_cq_sync_time` should be a more recent timestamp than after the first sync. ## Write Modes Your destination now supports all three write modes: | `write_mode` | Messages your destination receives | |---|---| | `append` | `WriteMigrateTable`, `WriteInsert` (plain INSERT) | | `overwrite` | `WriteMigrateTable`, `WriteInsert` (INSERT OR REPLACE on tables with primary keys) | | `overwrite-delete-stale` | `WriteMigrateTable`, `WriteInsert`, `WriteDeleteStale` | Your destination does not read `write_mode` directly. The engine controls which messages it sends. The behavior is determined by how you handle `WriteInsert` (INSERT vs INSERT OR REPLACE) and whether `DeleteStale` does anything. ## Testing The SDK ships a write test suite that exercises `MigrateTables`, `WriteTableBatch`, `DeleteStale`, and schema migration scenarios, with no test cases to write yourself. ```go filename="client/client_test.go" package client "context" "encoding/json" "testing" "github.com/cloudquery/plugin-sdk/v4/plugin" ) func TestPlugin(t *testing.T) { ctx := context.Background() p := plugin.NewPlugin("sqlite", "development", New) spec := Spec{ ConnectionString: ":memory:", } specBytes, err := json.Marshal(spec) if err != nil { t.Fatal(err) } if err := p.Init(ctx, specBytes, plugin.NewClientOptions{}); err != nil { t.Fatal(err) } plugin.TestWriterSuiteRunner(t, p, plugin.WriterTestSuiteTests{ SafeMigrations: plugin.SafeMigrations{ AddColumn: true, RemoveColumn: true, }, }) } ``` ```bash go test ./client/... ``` `TestWriterSuiteRunner` creates tables, inserts rows in all three write modes, applies schema migrations, and verifies correctness. The `SafeMigrations` block tells the suite which schema changes your destination handles without data loss; set these to match your `canAutoMigrate` logic. This pattern is taken directly from the [official SQLite destination test](https://github.com/cloudquery/cloudquery/blob/main/plugins/destination/sqlite/client/client_test.go). ## Troubleshooting **"Integration server listening" never appears after `./cq-destination-sqlite serve`** The binary didn't build correctly, or the previous `go build` succeeded but the binary wasn't updated. Run `go build ./...` and check for compile errors. A common cause is a missing method: if your struct doesn't implement all of `batchwriter.Client` (`MigrateTables`, `WriteTableBatch`, `DeleteStale`, `DeleteRecord`), the build will fail with a type error. **`cloudquery sync` fails with `connection refused`** Your destination isn't running, or the port in your config doesn't match the one it's listening on. Make sure `./cq-destination-sqlite serve` is running in a separate terminal, and that the `path` in your YAML (`localhost:7777`) matches the address shown in the server output. **`failed to unmarshal spec` error on sync** The `spec` block in your YAML has a field name mismatch or indentation problem. Check that the keys match the JSON field names in your `Spec` struct exactly (e.g., `connection_string`, not `connectionString`). **`failed to open database` error on sync** The directory in your `connection_string` path doesn't exist. SQLite can create the file but not the parent directory. Create the directory first, or use a path that already exists (e.g., `./test.db` in the current directory). **Sync completes successfully but the database is empty** You're missing the `writer.Flush(ctx)` call at the end of `Write`. The batchwriter holds rows in memory until a batch fills or the flush is called explicitly. Without `Flush`, the last partial batch is never written. **`cannot safely migrate table: breaking changes detected`** A column was removed, renamed, or had its type changed between syncs. In safe migration mode, only additive changes (new optional columns) are allowed. Either: - Add `migrate_mode: forced` to your destination spec to allow destructive migrations - Delete the database file and start fresh - Restore the previous schema **Rows appear duplicated after multiple syncs** Your table has no primary keys, so the destination uses plain `INSERT` on every sync. This is correct for `append` mode. If you want deduplication, make sure your source table defines primary keys; the destination will automatically switch to `INSERT OR REPLACE`. ## Common Pitfalls **Avoid these common mistakes:** - **Always call `writer.Flush(ctx)` at the end of `Write`.** The batchwriter holds buffered rows until a batch fills or a timeout fires. If you skip `Flush`, the last batch may never be written. - **Wrap writes in a transaction.** Without a transaction, each row is its own implicit transaction, which is extremely slow for large datasets. See `WriteTableBatch` above. - **Always rollback on error.** Use `defer` with an error check as shown. Skipping rollback can leave partially-written batches in the database. - **Handle `nil` for NULL values.** Check `col.IsNull(i)` and pass `nil` as the SQL parameter, not an empty string or zero. - **`INSERT OR REPLACE` is a delete + insert, not an update.** It resets all columns, not only the conflicting ones. This is correct for CloudQuery's use case (full resource snapshots) but worth understanding if you expect partial updates. ## Next Steps - **[Publish to the Hub](/cli/integrations/creating-new-integration/publishing)**: make your destination available to others - **[Go Source Guide](/cli/integrations/creating-new-integration/go-source)**: build a source integration that produces the data your destination consumes - **[Running Locally](/cli/advanced/running-locally)**: detailed guide on local registry, gRPC, and Docker modes - **[Performance Tuning](/cli/advanced/performance-tuning)**: batch size, concurrency, and write optimization - **[PostgreSQL Destination](https://github.com/cloudquery/cloudquery/tree/main/plugins/destination/postgresql)**: a more complex example using `mixedbatchwriter` with bulk insert support ## Real-World Examples - [SQLite Destination](https://github.com/cloudquery/cloudquery/tree/main/plugins/destination/sqlite): the official reference this guide is based on (`batchwriter`, CGO-based) - [PostgreSQL Destination](https://github.com/cloudquery/cloudquery/tree/main/plugins/destination/postgresql): uses `mixedbatchwriter` for bulk operations - [BigQuery Destination](https://github.com/cloudquery/cloudquery/tree/main/plugins/destination/bigquery): streaming insert pattern - [All destination integrations](https://github.com/cloudquery/cloudquery/tree/main/plugins/destination) ## Resources - [CloudQuery Community](https://community.cloudquery.io) - [Go SDK Source Code](https://github.com/cloudquery/plugin-sdk) (`plugin-sdk/v4`) - [batchwriter interface](https://github.com/cloudquery/plugin-sdk/blob/main/writers/batchwriter/batchwriter.go): the four methods your client must implement - [WriteMessage types](https://github.com/cloudquery/plugin-sdk/blob/main/message/write_message.go): full list of message types your destination can receive - [Apache Arrow Go docs](https://pkg.go.dev/github.com/apache/arrow-go/v18/arrow): Arrow type system reference --- Source: https://www.cloudquery.io/docs/cli/integrations/creating-new-integration/go-source # Creating a New Source Integration in Go This guide walks through building a source integration in Go using the CloudQuery SDK. As a running example, we reference the [xkcd integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/xkcd) which fetches comic data from the [xkcd API](https://xkcd.com/json.html). Before starting, make sure you're familiar with CloudQuery [core concepts](/cli/integrations/creating-new-integration#core-concepts) and have completed the [Getting Started guide](/cli/getting-started). **Prerequisites:** - Go installed ([Go Tutorial](https://go.dev/doc/tutorial/getting-started), [A Tour of Go](https://go.dev/tour/welcome/1)) - CloudQuery CLI installed ## Scaffold a New Integration The `cq-scaffold` tool generates a new Go source integration with all the boilerplate. Download it from the [releases page](https://github.com/cloudquery/cloudquery/releases?q=scaffold&expanded=true), or install via Homebrew on macOS: ```bash brew install cloudquery/tap/scaffold ``` Create a new integration (replace `` and `` with your GitHub org and integration name): ```bash cq-scaffold source cd cq-source- go mod tidy ``` The scaffold tool only generates Go source integrations. For other languages, see the [Python](/cli/integrations/creating-new-integration/python-source), [JavaScript](/cli/integrations/creating-new-integration/javascript-source), or [Java](/cli/integrations/creating-new-integration/java-source) guides. ## Project Structure Here's the structure of the [xkcd integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/xkcd), which is representative of a typical Go source integration: ```text plugins/source/xkcd/ ├── main.go # Entry point ├── go.mod # SDK dependency (plugin-sdk/v4) ├── plugin/ │ └── plugin.go # Name, version, kind, team constants ├── client/ │ ├── client.go # Client struct (implements schema.ClientMeta) │ ├── spec.go # Configuration spec │ └── testing.go # Mock client constructor used in table unit tests ├── internal/xkcd/ │ └── xkcd.go # HTTP client + Comic struct └── resources/ ├── plugin/ │ ├── plugin.go # Creates plugin via plugin.NewPlugin() │ └── client.go # Configure function, Sync, Tables └── services/ ├── comic.go # Table definition + resolver └── comic_test.go # Tests ``` A CloudQuery integration has several distinct components. Here's what each part does and how they fit together: - **`main.go`**: the entry point. It creates the integration and starts serving it over gRPC. You rarely need to modify this file. - **`plugin/`**: defines constants like the integration name, version, team, and kind (`source`). These identify your integration on the CloudQuery Hub. - **`client/`**: the **Client** is a struct that stores everything your resolvers need: an authenticated API client, configuration values, a logger, etc. Every resolver receives the Client so it can make API calls. The **Spec** is a struct matching the user's YAML configuration. It defines what settings your integration accepts (API keys, endpoints, concurrency, etc.). - **`internal/xkcd/`** (or `internal//`): your raw API client code. This is where you make HTTP calls to the third-party API, handle authentication headers, parse responses, and define the response structs. Keeping this separate from the CloudQuery-specific code means you can test and reuse it independently. - **`resources/plugin/`**: the **Configure** function lives here. It's called once when a sync starts: it parses the user's spec, creates the API client, sets up the scheduler, and returns a `plugin.Client` that the SDK uses to run the sync. - **`resources/services/`**: one file per **table**. Each file defines a table (the name, columns, and how they map to your API response struct) and a **resolver** (the function that actually calls the API and sends results back to CloudQuery). The resolver is the heart of each table: it's where you make API calls, handle pagination, and stream results to the destination. ## How It All Connects Before reading the code, it helps to understand the flow of what happens when a user runs `cloudquery sync`: 1. The CLI starts your integration as a separate process (or connects to it over gRPC if you're running it locally) 2. Your `main.go` creates the integration and starts the gRPC server 3. The CLI sends the user's `spec` configuration to your `Configure` function 4. `Configure` parses the spec, validates it, creates an authenticated API client, and returns a `plugin.Client` 5. The CLI asks your integration for its list of tables, then for each table, calls the table's resolver 6. Each resolver fetches data from the API and sends results over a channel. The SDK handles writing them to the destination. This flow means your main implementation work is in two places: the **Configure function** (parsing configuration and creating the API client) and the **resolvers** (fetching data from the API). ## Entry Point The `main.go` creates and serves the integration. This is boilerplate that you rarely need to modify. It wires together the `serve` package and your integration: ```go package main "context" "log" "github.com/cloudquery/plugin-sdk/v4/serve" plugin "github.com/cloudquery//cq-source-/resources/plugin" ) func main() { p := serve.Plugin(plugin.Plugin()) if err := p.Serve(context.Background()); err != nil { log.Fatalf("failed to serve plugin: %v", err) } } ``` Note that `main.go` imports from `resources/plugin`: the package that creates the full integration with `Sync`, `Tables`, and `Close` methods. The top-level `plugin/` directory only holds name, version, and kind constants. This is a common point of confusion in the project layout. ## Integration Setup Both `resources/plugin/plugin.go` and `resources/plugin/client.go` live in the same Go package (`package plugin`). The `plugin.go` file creates the integration; `client.go` holds the SDK-facing client struct and all the methods the SDK calls at sync time. `resources/plugin/plugin.go` wires the constants from `plugin/` to the SDK: ```go // resources/plugin/plugin.go package plugin internalPlugin "github.com/cloudquery//cq-source-/plugin" "github.com/cloudquery/plugin-sdk/v4/plugin" ) func Plugin() *plugin.Plugin { return plugin.NewPlugin( internalPlugin.Name, internalPlugin.Version, Configure, plugin.WithKind(internalPlugin.Kind), plugin.WithTeam(internalPlugin.Team), ) } ``` `resources/plugin/client.go` defines the SDK-facing client struct and the three methods the SDK calls during a sync. This is distinct from `client/client.go` (which implements `schema.ClientMeta` and is used inside resolvers): ```go // resources/plugin/client.go package plugin "context" "encoding/json" "fmt" "github.com/cloudquery/plugin-sdk/v4/message" "github.com/cloudquery/plugin-sdk/v4/plugin" "github.com/cloudquery/plugin-sdk/v4/scheduler" "github.com/cloudquery/plugin-sdk/v4/schema" "github.com/cloudquery/plugin-sdk/v4/state" "github.com/cloudquery/plugin-sdk/v4/transformers" "github.com/rs/zerolog" "github.com/cloudquery//cq-source-/client" "github.com/cloudquery//cq-source-/resources/services" ) // Client implements plugin.Client for a source integration. // It embeds plugin.UnimplementedDestination to satisfy the full // plugin.Client interface without providing write methods. type Client struct { logger zerolog.Logger config client.Spec tables schema.Tables scheduler *scheduler.Scheduler services *yourapi.Client plugin.UnimplementedDestination } func (c *Client) Sync(ctx context.Context, options plugin.SyncOptions, res chan<- message.SyncMessage) error { tt, err := c.tables.FilterDfs(options.Tables, options.SkipTables, options.SkipDependentTables) if err != nil { return err } stateClient, err := state.NewConnectedClient(ctx, options.BackendOptions) if err != nil { return err } defer stateClient.Close() schedulerClient := client.New(c.logger, c.config, c.services, stateClient) if err := c.scheduler.Sync(ctx, schedulerClient, tt, res, scheduler.WithSyncDeterministicCQID(options.DeterministicCQID)); err != nil { return fmt.Errorf("failed to sync: %w", err) } return stateClient.Flush(ctx) } func (c *Client) Tables(_ context.Context, options plugin.TableOptions) (schema.Tables, error) { return c.tables.FilterDfs(options.Tables, options.SkipTables, options.SkipDependentTables) } func (*Client) Close(_ context.Context) error { return nil } ``` `getTables()` builds the table list once at startup, applies transformers, and injects the standard CloudQuery columns (`_cq_id`, `_cq_source_name`, `_cq_sync_time`): ```go func getTables() schema.Tables { tables := []*schema.Table{ services.ComicsTable(), } if err := transformers.TransformTables(tables); err != nil { panic(err) } for _, t := range tables { schema.AddCqIDs(t) } return tables } ``` ## Configuration & Authentication The SDK passes the user's `spec` block from their YAML configuration as raw JSON bytes to your `Configure` function. Define a Spec struct and unmarshal it: ```go // client/spec.go package client type Spec struct { AccessToken string `json:"access_token"` Concurrency int `json:"concurrency"` } func (s *Spec) SetDefaults() { if s.Concurrency == 0 { s.Concurrency = 100 } } func (s *Spec) Validate() error { if s.AccessToken == "" { return fmt.Errorf("access_token is required") } return nil } ``` `Configure` is the constructor the SDK calls once per sync. It has two distinct code paths: a fast-path for when the CLI only needs metadata (no live connection), and the normal path that creates a real API client and scheduler: ```go // resources/plugin/client.go (continued) func Configure(_ context.Context, logger zerolog.Logger, specBytes []byte, opts plugin.NewClientOptions) (plugin.Client, error) { if opts.NoConnection { // Called when the CLI needs table schema without connecting, // e.g. for documentation generation or --no-migrate. return &Client{ logger: logger.With().Str("module", "").Logger(), tables: getTables(), }, nil } var spec client.Spec if err := json.Unmarshal(specBytes, &spec); err != nil { return nil, fmt.Errorf("failed to unmarshal spec: %w", err) } spec.SetDefaults() if err := spec.Validate(); err != nil { return nil, err } apiClient, err := yourapi.NewClient(spec.AccessToken) if err != nil { return nil, fmt.Errorf("failed to create API client: %w", err) } return &Client{ logger: logger.With().Str("module", "").Logger(), config: spec, scheduler: scheduler.NewScheduler( scheduler.WithLogger(logger), scheduler.WithConcurrency(spec.Concurrency), ), services: apiClient, tables: getTables(), }, nil } ``` Users configure authentication in their YAML file. The CLI automatically resolves environment variable references: ```yaml spec: access_token: "${YOUR_API_TOKEN}" concurrency: 50 ``` For public APIs that don't require authentication (like xkcd), omit `access_token` from the Spec struct and remove its validation from `Validate()`. ## Define a Table A table in CloudQuery represents a collection of related data, typically one API resource type. In Go, you define a table as a function returning a `*schema.Table`. Each table needs three things: a **name** (which becomes the database table name), a **transformer** (which maps your Go struct fields to columns), and a **resolver** (the function that fetches data from the API). Rather than listing columns manually, the SDK can auto-map fields from a Go struct using `transformers.TransformWithStruct`. If an existing Go SDK already provides a struct for the API response, you can use it directly. Otherwise, define your own struct matching the API's JSON response. Here's the actual xkcd comics table: ```go package services "github.com/cloudquery/plugin-sdk/v4/schema" "github.com/cloudquery/plugin-sdk/v4/transformers" ) func ComicsTable() *schema.Table { return &schema.Table{ Name: "xkcd_comics", Resolver: fetchComics, Transform: transformers.TransformWithStruct( &xkcd.Comic{}, transformers.WithPrimaryKeys("Num"), ), } } ``` Notice that we don't list individual columns. `TransformWithStruct` inspects the `Comic` struct and creates a column for each exported field. The `WithPrimaryKeys("Num")` option marks the `Num` field as the primary key. The final table name `xkcd_comics` will appear directly as a database table when synced. The `Comic` struct defines the columns (from `internal/xkcd/xkcd.go`): ```go type Comic struct { Month string `json:"month"` Num int `json:"num"` Link string `json:"link"` Year string `json:"year"` News string `json:"news"` SafeTitle string `json:"safe_title"` Transcript string `json:"transcript"` Alt string `json:"alt"` Img string `json:"img"` Title string `json:"title"` Day string `json:"day"` } ``` Each struct field becomes a column in the destination table. The SDK maps Go types to appropriate database types (e.g. `string` → text, `int` → integer). The `json` tags determine how the struct is serialized but don't affect column names. Column names are derived from the Go field names, converted to snake_case. ## Write a Table Resolver The resolver is the heart of your integration: it's the function that actually calls the third-party API and sends results back to CloudQuery. The resolver signature has four arguments, each serving a specific purpose: ```go func fetchComics(ctx context.Context, meta schema.ClientMeta, parent *schema.Resource, res chan<- any) error ``` - **`ctx`**: a standard Go context, used for cancellation. If a user stops a sync, this context is cancelled, so your resolver should respect it in long-running loops. - **`meta`**: your `Client` struct (cast it with `meta.(*client.Client)`). This gives you access to the API client, credentials, and any shared state. - **`parent`**: for top-level tables, this is `nil`. For child tables (e.g. fetching commits for a specific repository), this contains the parent row so you can extract the parent's ID. - **`res`**: a channel where you send your results. Each item you send becomes a row in the destination table. Here's the xkcd resolver. It fetches the latest comic to determine the total count, then iterates through all comics by ID: ```go func fetchComics(ctx context.Context, meta schema.ClientMeta, parent *schema.Resource, res chan<- any) error { c := meta.(*client.Client) latest, err := c.XKCD.GetLatestComic(ctx) if err != nil { return err } res <- latest for i := 1; i < latest.Num; i++ { comic, err := c.XKCD.GetComic(ctx, i) if err != nil { return err } res <- comic } return nil } ``` A few important things to note about this code: - We send each comic to the `res` channel **as soon as we get it**. This is important. The SDK streams items to the destination immediately, so don't collect everything into a slice first. Streaming keeps memory usage low and gets data to the user's database faster. - You can send items one at a time or as a slice. The SDK handles both. - If an API call fails, **return the error**. The SDK will log it and report it to the user. Any items you've already sent to `res` before the error are still written to the destination, so partial results are preserved. - The struct you send (`Comic`) must match the struct used in `TransformWithStruct`. That's how the SDK knows which fields map to which columns. ## The Client The `Client` struct is where you store everything that resolvers need to access: API clients, credentials, configuration, and any other shared state. Every resolver receives it via the `meta` argument. The Client lives in `client/client.go` and must implement the `schema.ClientMeta` interface, which requires an `ID()` method: ```go type Client struct { Logger zerolog.Logger XKCD *xkcd.Client Backend state.Client } func (c *Client) ID() string { return "xkcd" } ``` The `ID()` method serves two purposes: it identifies the client in log messages, and the SDK uses it internally to track which multiplexed client is running. For a small integration like xkcd, a static string is fine. For multiplexed integrations (e.g. one that syncs multiple AWS accounts), you'd include the account name so each client has a unique ID. The Client is created inside the `Configure` function and passed to the SDK, which then provides it to every resolver. This is the main way your integration's initialization code communicates with its resolvers. ## Test Locally Start the integration as a gRPC server for debugging: ```shell go run main.go serve ``` Or build and run as a local binary: ```shell go build ./cq-source- serve ``` Then sync using the appropriate registry. See [Testing Locally](/cli/integrations/creating-new-integration#testing-locally) for configuration examples and [Running Locally](/cli/advanced/running-locally) for full details. ## Advanced: Column Resolvers Most of the time, `TransformWithStruct` handles column mapping automatically. But sometimes you need a column that doesn't come directly from the API response, maybe it's derived from other fields, or requires an additional API call. In these cases, you can add extra columns with their own resolver functions. For example, imagine we want to add an `is_good` boolean column to the xkcd comics table that doesn't exist in the API response. We add it to the `Columns` field alongside the auto-generated columns from `Transform`: ```go func ComicsTable() *schema.Table { return &schema.Table{ Name: "xkcd_comics", Resolver: fetchComics, Transform: transformers.TransformWithStruct(&xkcd.Comic{}), Columns: []schema.Column{ { Name: "is_good", Type: arrow.FixedWidthTypes.Boolean, Resolver: resolveComicIsGood, }, }, } } func resolveComicIsGood(ctx context.Context, meta schema.ClientMeta, resource *schema.Resource, c schema.Column) error { comic := resource.Item.(xkcd.Comic) return resource.Set(c.Name, strings.Contains(comic.Title, "xkcd")) } ``` The column resolver receives the current row via `resource.Item`. You cast it to your struct type, compute the value, and set it with `resource.Set()`. As big fans of meta-jokes, we define only comics with `"XKCD"` in the title to be good. These custom columns appear alongside the auto-generated columns from `TransformWithStruct`. ## Advanced: Multiplexing For our xkcd integration, multiplexing isn't necessary. There's only one xkcd API with no accounts or organizations. But many real-world integrations need to fetch data for multiple entities. For example, a GitHub integration that syncs repositories for multiple organizations needs to make separate API calls per org. Without multiplexing, these would run sequentially. With multiplexing, they run in parallel. A multiplexer is a function that takes the base client and returns a slice of clients, one per entity. The SDK calls your table resolver once for each client in the slice: ```go func AccountMultiplex(meta schema.ClientMeta) []schema.ClientMeta { client := meta.(*Client) l := make([]schema.ClientMeta, 0, len(client.accounts)) for _, acc := range client.accounts { l = append(l, client.WithAccount(acc)) } return l } ``` Then set `Multiplex: client.AccountMultiplex` on tables that need it. Make sure the client's `ID()` method returns a unique value per multiplexed entity: ```go func (c *Client) ID() string { return fmt.Sprintf("myplugin:%s", c.Account) } ``` Inside the resolver, you can then access the current account via `client.Account` to make the right API calls. ## Advanced: Incremental Tables By default, every sync fetches all data from scratch. For small APIs this is fine, but for APIs with millions of records that rarely change, re-fetching everything is wasteful. Incremental tables solve this by remembering where the last sync left off (using a **cursor**) and only fetching new data on subsequent syncs. To make a table incremental, you need to mark it as such and designate one column as the incremental key (the cursor): ```go func Items() *schema.Table { return &schema.Table{ Name: "hackernews_items", Resolver: fetchItems, IsIncremental: true, Transform: transformers.TransformWithStruct(&hackernews.Item{}), Columns: []schema.Column{ { Name: "id", Type: arrow.PrimitiveTypes.Int64, PrimaryKey: true, IncrementalKey: true, }, }, } } ``` In the resolver, use the state backend to persist the cursor: ```go func fetchItems(ctx context.Context, meta schema.ClientMeta, _ *schema.Resource, res chan<- any) error { c := meta.(*client.Client) tableName := Items().Name // Load cursor from last sync value, err := c.Backend.GetKey(ctx, tableName) // ... fetch data starting from cursor ... // Save cursor after processing err = c.Backend.SetKey(ctx, tableName, strconv.Itoa(newCursor)) err = c.Backend.Flush(ctx) // Must flush to persist return nil } ``` See [Managing Incremental Tables](/cli/advanced/managing-incremental-tables) for the full guide, and the [Hacker News integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/hackernews) for the complete working example. ## Troubleshooting **`go run main.go serve` fails with a compile error** The most common cause is an import path mismatch. Make sure `main.go` imports from `resources/plugin` (the package with the `Plugin()` function), not the top-level `plugin/` directory (which only holds constants). See the [Entry Point](#entry-point) section for the correct import path. **`cloudquery sync` fails with `connection refused`** Your integration isn't running, or the port in your config doesn't match. Make sure `go run main.go serve` is running in a separate terminal, and that the `path` in your YAML (`localhost:7777`) matches the address shown in the server output. **`failed to validate spec` error on sync** Your Spec's `Validate()` method returned an error. Check that all required fields are present in your YAML config and that environment variable references like `${MY_API_TOKEN}` are set in the shell where you're running `cloudquery sync`. **Resolver is never called / zero rows synced** The table is probably filtered out. Check that the table name listed in your YAML `tables:` field matches the name returned by your `ComicsTable()` function exactly. If you're using `tables: ["*"]`, make sure the table is included in the list returned by `getTables()` in `resources/plugin/client.go`. **Resolver runs but sends no rows to the `res` channel** Add a log statement inside the resolver to verify it's executing and that your API call is returning data. Check that you're actually sending to `res`. A common mistake is building a slice and forgetting to send its elements. **Sync works locally but re-fetches everything on the second run** You have an incremental table but forgot to call `c.Backend.Flush(ctx)` after saving the cursor. Without `Flush`, the cursor is never persisted and each sync starts from scratch. See [Advanced: Incremental Tables](#advanced-incremental-tables). ## Common Pitfalls **Avoid these common mistakes when building Go integrations:** - **Don't batch results in memory.** Send items to the `res` channel as soon as they're available. Don't collect all pages into a slice and send them at the end. This wastes memory and delays writes to the destination. - **Fetch concurrently when the API allows it.** A sequential loop is the simplest starting point, but for large datasets use `golang.org/x/sync/errgroup` with a concurrency limit so you don't overwhelm the API. See the [xkcd integration](https://github.com/cloudquery/cloudquery/blob/main/plugins/source/xkcd/resources/services/comic.go) for a working example. - **Always call `Backend.Flush(ctx)` for incremental tables.** If you skip this, your cursor won't persist and the next sync will re-fetch everything. - **Make `ID()` unique per multiplexed client.** If two multiplexed clients return the same `ID()`, the SDK won't parallelize them correctly. - **Return errors from resolvers.** Don't silently swallow API errors. Return them so the SDK can log them and surface them to the user. - **Respect context cancellation.** Check `ctx.Done()` in long-running loops so the user can cancel a sync cleanly. ## Publishing Visit [Publishing an Integration to the Hub](/cli/integrations/creating-new-integration/publishing) for release instructions. ## Real-World Examples - [xkcd](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/xkcd): starter integration referenced in this tutorial - [Hacker News](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/hackernews): incremental tables with state backend - [Kubernetes](https://github.com/cloudquery/cloudquery/tree/plugins-source-k8s-v6.2.4/plugins/source/k8s): large-scale integration with many tables and mock tests - [PostgreSQL Destination](https://github.com/cloudquery/cloudquery/tree/main/plugins/destination/postgresql): "unmanaged" destination that handles batching itself - [BigQuery Destination](https://github.com/cloudquery/cloudquery/tree/main/plugins/destination/bigquery): "managed" destination with per-table batching - [All integrations](https://github.com/cloudquery/cloudquery/tree/main/plugins) ## Next Steps Once your integration is working locally: 1. **[Publish to the Hub](/cli/integrations/creating-new-integration/publishing)**: make your integration available to others 2. **Add tests**: see `comic_test.go` in the [xkcd integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/xkcd) for a testing pattern 3. **Add incremental tables**: use the [state backend](/cli/advanced/managing-incremental-tables) for large datasets that don't change much between syncs 4. **Add multiplexing**: parallelize fetching if your integration supports multiple accounts or regions 5. **Build a destination**: see the [Go Destination guide](/cli/integrations/creating-new-integration/go-destination) to write an integration that receives and stores data ## Resources - [CloudQuery Community](https://community.cloudquery.io) - [Go SDK Source Code](https://github.com/cloudquery/plugin-sdk) (`plugin-sdk/v4`) - [How to Write a CloudQuery Source Integration](https://www.youtube.com/watch?v=3Ka_Ob8E6P8) (Video. May reference older SDK patterns; use this guide for current code.) --- Source: https://www.cloudquery.io/docs/cli/integrations/creating-new-integration # Creating a New Integration CloudQuery integrations are modular: a source integration fetches data from any third-party API, and a destination integration writes it to any supported target. You can mix and match any source with any destination. The CloudQuery protocol is language-agnostic, built on [gRPC](https://grpc.io/docs/languages/) and [Apache Arrow](https://arrow.apache.org/). For faster development, we provide SDKs that handle the protocol details so you can focus on your API logic. The code and SDKs use the term **plugin** (e.g. `plugin-sdk`, `cq-source-`), while CloudQuery documentation uses **integration**. These terms refer to the same thing. ## Choose Your Language | | Go | Python | JavaScript | Java | | --- | --- | --- | --- | --- | | Source SDK | Yes ([Guide](/cli/integrations/creating-new-integration/go-source)) | Yes ([Guide](/cli/integrations/creating-new-integration/python-source)) | Yes ([Guide](/cli/integrations/creating-new-integration/javascript-source)) | Yes ([Guide](/cli/integrations/creating-new-integration/java-source)) | | Destination SDK | Yes ([Guide](/cli/integrations/creating-new-integration/go-destination)) | Yes ([Example](https://github.com/cloudquery/cloudquery/tree/main/plugins/destination/sqlite-python)) | No | No | | Scaffold / template | [`cq-scaffold` CLI](https://github.com/cloudquery/cloudquery/releases?q=scaffold&expanded=true) | [Template repo](https://github.com/cloudquery/python-plugin-template/) | Clone [Airtable](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/airtable) | Clone [Bitbucket](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/bitbucket) | **Go** is the most mature option with a dedicated scaffold tool, step-by-step guides for both source and destination integrations, and the widest range of examples. **Python**, **JavaScript**, and **Java** SDKs are all actively maintained and work well for source integrations. ## Development Workflow Regardless of language, building a CloudQuery integration follows the same steps: 1. **Scaffold or clone**: Use the scaffold tool (Go) or clone a template/reference integration (other languages) 2. **Define tables**: Declare the tables and columns your integration exposes 3. **Implement resolvers**: Write functions that fetch data from the API and send it to CloudQuery 4. **Test locally**: Run your integration as a gRPC server or local binary and sync with `cloudquery sync` 5. **Publish**: Release your integration to the [CloudQuery Hub](/cli/integrations/creating-new-integration/publishing) ## Core Concepts These concepts apply to all CloudQuery integrations regardless of language. ### Syncs A sync is the process triggered by `cloudquery sync`. It fetches data from a source API and writes it to a destination (database, data lake, stream, etc.). When you build a source integration, you only implement the part that talks to the third-party API. The SDK handles delivering data to the destination. ### Tables A **table** is CloudQuery's unit for a collection of related data. In most databases it maps directly to a database table; in other destinations it could be a file, stream, or other medium. A table is defined by: - A **name**: follows the convention `__`, e.g. `xkcd_comics` - **Columns**: usually derived automatically from a struct or class via the SDK's transformer - A **resolver**: the function that fetches data for this table Tables can be organized into **services** (logical groupings that mirror the underlying API structure). For small integrations with only a few tables, you can skip services and put tables directly in a `resources` directory. ### Resolvers Resolvers are functions that populate table data. There are two types: - **Table resolvers** fetch data from the API and send results to a channel (Go), yield them as a generator (Python), or write them to a stream (JavaScript/Java). For top-level tables, the resolver is called once per multiplexer client. For child tables, it's called once per parent row. - **Column resolvers** (optional) handle custom column mappings. In most cases, the SDK auto-maps struct/class fields to columns, so you won't need these. ### Multiplexers Multiplexers parallelize data fetching. If your integration supports multiple accounts, organizations, or regions, a multiplexer calls the table resolver once per entity, in parallel. Many integrations don't need multiplexers. ### Incremental Tables Instead of fetching all data on every sync, incremental tables use a **cursor** stored in a state **backend** to resume from where the last sync ended. This is much more efficient for large datasets but adds complexity. Consider incremental tables when: - The API supports filtering by timestamp or cursor - Full syncs are too slow or expensive - Data volume is large and mostly unchanged between syncs Learn more in [Managing Incremental Tables](/cli/advanced/managing-incremental-tables). ## Configuration & Authentication When a user runs `cloudquery sync`, their YAML configuration is passed to your integration. The `spec` field under your source configuration contains custom settings like API keys, endpoints, and options: ```yaml filename="config.yaml" kind: source spec: name: "my-integration" registry: "grpc" path: "localhost:7777" tables: ["*"] destinations: - "sqlite" spec: # These fields are defined by YOUR integration access_token: "${MY_API_TOKEN}" base_url: "https://api.example.com" concurrency: 100 ``` Your integration receives the inner `spec` block as raw JSON. You parse it into a typed configuration object (a Go struct, Python dataclass, TypeScript object, or Java class), validate it, and use it to create an authenticated API client. The typical flow: 1. SDK passes `spec` as JSON bytes/string to your initialization function 2. You deserialize it into a typed Spec struct/class 3. You validate required fields (e.g. `access_token` must not be empty) 4. You create an API client using the spec values 5. You store the client on your `Client` struct for resolvers to use See each language guide for the specific parsing patterns. Never log or expose API tokens or secrets. Use environment variable references like `${MY_API_TOKEN}` in configuration files. The CloudQuery CLI resolves these automatically. ## Common Patterns ### Pagination Most real-world APIs require pagination. Handle this in your table resolver by looping until all pages are fetched: - **Cursor-based**: Store the cursor from each response, pass it in the next request. Send each page's results immediately. Don't accumulate them in memory. - **Offset-based**: Increment the offset by the page size on each iteration. - **Link-based**: Follow `next` URLs from the response headers or body. The key principle: **stream results as you get them**. Send each page's items to the channel/stream/generator immediately rather than collecting all pages first. This keeps memory usage low and gets data to the destination faster. ### Parent-Child Tables Many APIs are hierarchical: organizations contain repositories, repositories contain commits, etc. CloudQuery supports this via **table relations**: 1. Define a parent table (e.g. `workspaces`) and a child table (e.g. `repositories`) 2. Link them via the parent's `relations` field 3. The child resolver receives the parent row and can extract the parent ID to make its API call This is a common pattern. See the [Bitbucket integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/bitbucket) (Java) for a clean example of Workspaces → Repositories, or the [Kubernetes integration](https://github.com/cloudquery/cloudquery/tree/plugins-source-k8s-v6.2.4/plugins/source/k8s) (Go) for a complex hierarchy. ### Error Handling When an API call fails in a resolver: - **Return the error**: the SDK will log it and surface it to the user. Don't silently swallow errors. - **Don't retry internally**: the SDK and the user's infrastructure handle retries at a higher level. - **Partial results are OK**: if you've already sent some items to the channel/stream before encountering an error, those items are still written to the destination. Return the error after sending what you can. - **Rate limiting**: if the API returns a rate limit response (HTTP 429), most SDK HTTP clients handle backoff automatically. If you're making raw HTTP calls, consider adding exponential backoff for 429 and 5xx responses. ## Testing Locally There are two ways to test an integration during development. Both are covered in detail in [Running Locally](/cli/advanced/running-locally). ### gRPC Server Mode Run your integration with the `serve` command, then point `cloudquery sync` at it using `registry: grpc`: ```yaml filename="config.yaml" kind: source spec: name: "my-integration" registry: "grpc" path: "localhost:7777" tables: ["*"] destinations: - "sqlite" --- kind: destination spec: name: sqlite path: cloudquery/sqlite registry: cloudquery version: "v2.14.5" spec: connection_string: ./db.sql ``` This mode is ideal for debugging. You can attach a debugger to the running process. Errors appear in the integration's console, not in `cloudquery.log`. ### Local Binary / Docker Mode Build your integration as a binary or Docker image, then use `registry: local` or `registry: docker` in your configuration. This is closest to how users will run your integration in production. See each language guide for specific build commands. ## Publishing When your integration is ready, publish it to the [CloudQuery Hub](/cli/integrations/creating-new-integration/publishing) so others can use it. The publishing guide covers all supported languages. ## Resources - [CloudQuery Community](https://community.cloudquery.io) - [How to Write a CloudQuery Source Integration](https://www.youtube.com/watch?v=3Ka_Ob8E6P8) (Video. Go.) - [How to Write a CloudQuery Source Integration in Python](https://youtu.be/TSbGHz5Z09M) (Video. Python.) - [All source and destination integrations](https://github.com/cloudquery/cloudquery/tree/main/plugins) --- Source: https://www.cloudquery.io/docs/cli/integrations/creating-new-integration/java-source # Creating a New Source Integration in Java This guide walks through building a source integration in Java using the CloudQuery SDK. We reference the [Bitbucket integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/bitbucket) as an example throughout. Before starting, make sure you're familiar with CloudQuery [core concepts](/cli/integrations/creating-new-integration#core-concepts) and have completed the [Getting Started guide](/cli/getting-started). **Prerequisites:** - Java 20+ and [Gradle 8.4+](https://gradle.org/install/) installed ([Java tutorials](https://www.w3schools.com/java/)) - CloudQuery CLI installed - A GitHub personal access token with `read:packages` scope for SDK authentication (see [GitHub Packages docs](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-gradle-registry#authenticating-to-github-packages)) The Java SDK is distributed via GitHub Packages (not Maven Central), so you need a GitHub PAT to download the dependency. ## Get Started Clone the [Bitbucket integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/bitbucket) as a starting point. Set up GitHub Packages authentication: ```shell export GITHUB_ACTOR= export GITHUB_TOKEN= ``` Build the project: ```shell gradle build ``` ## How It All Connects When a user runs `cloudquery sync`, here's what happens with your Java integration: 1. The CLI starts your integration (or connects to it over gRPC if you're running it locally) 2. Your `MainClass.java` creates the plugin and starts the gRPC server via `PluginServe` 3. The CLI calls your plugin's `newClient` method with the user's `spec` configuration 4. `newClient` parses the spec (using Jackson), validates it, creates an authenticated API client, and transforms tables 5. The CLI calls `sync`, which runs each table's resolver through the scheduler 6. Each resolver is a lambda that calls `stream.write()` for each record. The SDK handles writing them to the destination. Your main implementation work is in three places: the **`newClient` method** (parsing configuration and creating the API client), the **table definitions** (defining columns and relations), and the **resolvers** (lambda functions that fetch data from the API). ## Project Structure Here's the structure based on the [Bitbucket integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/bitbucket): ```text plugins/source/bitbucket/ ├── settings.gradle ├── gradlew / gradlew.bat ├── Dockerfile ├── gradle/ └── app/ ├── build.gradle # SDK dependency (io.cloudquery:plugin-sdk-java) └── src/main/java/bitbucket/ ├── MainClass.java # Entry point ├── BitbucketPlugin.java # Plugin definition ├── client/ │ ├── BitbucketClient.java # API client (implements ClientMeta) │ └── configuration/ │ └── Spec.java # Configuration spec (@Data + Jackson) └── resources/ ├── Workspaces.java # Table definition └── Repositories.java # Child table definition ``` Here's what each component does: - **`MainClass.java`**: the entry point. Creates the plugin and starts the gRPC server via `PluginServe.builder()`. This is boilerplate you rarely modify. - **`BitbucketPlugin.java`**: the **Plugin class** extends `io.cloudquery.plugin.Plugin` and implements three key methods: `newClient` (parses the spec, creates the API client), `tables` (returns the list of tables), and `sync` (runs resolvers through the scheduler). - **`client/BitbucketClient.java`**: the **Client** implements `ClientMeta` and wraps your authenticated API client. Every resolver receives this client so it can make API calls. It also provides an `id()` method used in logs and for multiplexing. - **`client/configuration/Spec.java`**: the **Spec** is a Lombok `@Data` class deserialized from the user's YAML configuration via Jackson. It defines what settings your integration accepts (credentials, endpoints, concurrency). - **`resources/`**: one file per **table**. Each file defines a table using the builder pattern (name, columns, transform, relations) and a **resolver** lambda that fetches data from the API. The Bitbucket example shows parent-child tables: `Workspaces.java` defines the parent, `Repositories.java` defines the child. ## Entry Point The `MainClass.java` creates and serves the plugin. This is boilerplate that you rarely need to change: ```java public class MainClass { public static void main(String[] args) { BitbucketPlugin plugin = new BitbucketPlugin(); PluginServe pluginServe = PluginServe.builder() .args(args) .plugin(plugin) .build(); int exitCode = pluginServe.Serve(); System.exit(exitCode); } } ``` ## Plugin Class The Plugin class is the central piece of your integration. It extends `io.cloudquery.plugin.Plugin` and implements three key methods: `newClient` (called with the user's spec to set up the API client), `tables` (returns available tables), and `sync` (runs the resolvers through a scheduler): ```java public class BitbucketPlugin extends Plugin { public BitbucketPlugin() { super("bitbucket", PLUGIN_VERSION); setTeam("my-team"); setKind(PluginKind.Source); } @Override public ClientMeta newClient(String spec, boolean noConnection) { // Parse spec, create API client, transform tables Spec parsedSpec = objectMapper.readValue(spec, Spec.class); return new BitbucketClient(parsedSpec); } @Override public List tables() { return List.of(Workspaces.getTable()); } @Override public void sync(List
tables, SyncStream syncStream) { Scheduler.builder() .client(client) .tables(tables) .syncStream(syncStream) .concurrency(spec.getConcurrency()) .build() .sync(); } } ``` ## Define a Table In Java, tables use the builder pattern. Similar to Go's `TransformWithStruct`, the Java SDK provides `TransformWithClass` which auto-maps fields from a Java model class (like a POJO or Lombok `@Data` class) to table columns. This means you don't need to define columns manually. The SDK inspects your model class and creates appropriate columns for each field. ```java public class Workspaces { public static Table getTable() { return Table.builder() .name("bitbucket_workspaces") .description("Bitbucket workspaces") .transform(TransformWithClass.builder(Workspace.class) .pkField("uuid") .build()) .relations(List.of(Repositories.getTable())) .resolver(resolveWorkspaces()) .build(); } } ``` Key details: - Use `Table.builder()` to construct tables - `TransformWithClass.builder(ModelClass.class)` auto-maps fields to columns - Set primary keys via `.pkField("fieldName")` - Define parent-child relations via `.relations(List.of(ChildTable.getTable()))` ## Write a Table Resolver Resolvers are where the actual API fetching happens. In Java, `TableResolver` is a functional interface, so you can implement it as a lambda. The lambda receives three arguments: - **`clientMeta`**: your Client class (cast it to access API methods). This is the same object returned by `newClient` in your plugin. - **`parent`**: for top-level tables, this is `null`. For child tables, it contains the parent row so you can extract the parent's data (e.g. a workspace name to list its repositories). - **`stream`**: call `stream.write(item)` to send each record to the destination. You can also use `stream::write` as a method reference with `forEach`. Here's a top-level resolver that fetches workspaces, and a child resolver that fetches repositories for each workspace: ```java // Top-level resolver: called once, fetches all workspaces private static TableResolver resolveWorkspaces() { return (clientMeta, parent, stream) -> { BitbucketClient client = (BitbucketClient) clientMeta; List workspaces = client.listWorkspaces(); workspaces.forEach(stream::write); }; } // Child resolver: called once per parent workspace row private static TableResolver resolveRepositories() { return (clientMeta, parent, stream) -> { BitbucketClient client = (BitbucketClient) clientMeta; // Extract the workspace name from the parent row String workspaceName = ((Workspace) parent.getItem()).getName(); client.listRepositoriesForWorkspace(workspaceName) .forEach(stream::write); }; } ``` The child resolver demonstrates a key pattern: `parent.getItem()` returns the model object from the parent row. You cast it to the parent's type and extract whatever you need to make the child API call. The SDK automatically calls the child resolver once for each row in the parent table, so you don't need to loop over parents yourself. ## Client The client implements `io.cloudquery.schema.ClientMeta`: ```java public class BitbucketClient implements ClientMeta { private final Spec spec; @Override public String id() { return "bitbucket"; } public List listWorkspaces() { // HTTP calls using Unirest or your preferred HTTP client } } ``` ## Configuration & Authentication The SDK passes the user's `spec` block as a JSON string to your plugin's `newClient` method. Use Lombok `@Data` with Jackson for deserialization: ```java @Data public class Spec { @JsonProperty("username") private String username; @JsonProperty("password") private String password; @JsonProperty("concurrency") private int concurrency = 1000; } ``` Then in `newClient`, parse the spec and create your authenticated API client: ```java @Override public ClientMeta newClient(String spec, boolean noConnection) throws Exception { ObjectMapper mapper = new ObjectMapper(); Spec parsedSpec = mapper.readValue(spec, Spec.class); // Validate required fields if (parsedSpec.getUsername() == null || parsedSpec.getUsername().isEmpty()) { throw new IllegalArgumentException("username is required"); } // Create authenticated API client return new BitbucketClient(parsedSpec); } ``` Users configure authentication in their YAML file. The CLI automatically resolves environment variable references: ```yaml spec: username: "${BITBUCKET_USERNAME}" password: "${BITBUCKET_APP_PASSWORD}" concurrency: 500 ``` ## Common Pitfalls **Avoid these common mistakes when building Java integrations:** - **Set up GitHub Packages auth before building.** The SDK is distributed via GitHub Packages, not Maven Central. Without `GITHUB_ACTOR` and `GITHUB_TOKEN` set, `gradle build` will fail to resolve the dependency. - **Stream results immediately.** Call `stream.write()` or `stream::write` for each item as you get it. Don't accumulate all results in a list first. - **Handle pagination in resolvers.** Most APIs return paginated results. Loop until all pages are fetched, writing each page's items to the stream immediately. - **Make `id()` unique per multiplexed client.** If you're multiplexing over accounts or workspaces, include the entity name in the `id()` return value. - **Pass Docker build args for CI.** When building Docker images, remember to pass `--build-arg GITHUB_ACTOR` and `--build-arg GITHUB_TOKEN` or the build will fail. ## Test Locally Start the integration as a gRPC server: ```shell gradle run --args serve ``` You can also build and run as a Docker container. See the [Bitbucket Dockerfile](https://github.com/cloudquery/cloudquery/blob/9a630b01890bea7b30e45007f0c00169ed066d23/plugins/source/bitbucket/Dockerfile). Note that Docker builds require `GITHUB_ACTOR` and `GITHUB_TOKEN` as build args: ```shell docker build -t my-integration:latest \ --build-arg GITHUB_ACTOR= \ --build-arg GITHUB_TOKEN= . ``` See [Testing Locally](/cli/integrations/creating-new-integration#testing-locally) for configuration examples and [Running Locally](/cli/advanced/running-locally) for full details. ## Publishing Visit [Publishing an Integration to the Hub](/cli/integrations/creating-new-integration/publishing) for release instructions. Java integrations can also be distributed as Docker images. ## Real-World Examples - [Bitbucket](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/bitbucket): parent-child tables, REST API with pagination, Arrow type mapping ## Next Steps Once your integration is working locally: 1. **[Publish to the Hub](/cli/integrations/creating-new-integration/publishing)**: make your integration available to others 2. **Add parent-child tables**: use `.relations()` on the parent table builder to link hierarchical resources (see the Workspaces → Repositories pattern in the [Bitbucket integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/bitbucket)) 3. **Build a Docker image**: see the [Bitbucket Dockerfile](https://github.com/cloudquery/cloudquery/blob/9a630b01890bea7b30e45007f0c00169ed066d23/plugins/source/bitbucket/Dockerfile) for a production-ready example 4. **Add pagination**: most APIs require it; loop until all pages are fetched, streaming each page's items immediately ## Resources - [CloudQuery Community](https://community.cloudquery.io) - [Java SDK Source Code](https://github.com/cloudquery/plugin-sdk-java) --- Source: https://www.cloudquery.io/docs/cli/integrations/creating-new-integration/javascript-source # Creating a New Source Integration in JavaScript This guide walks through building a source integration in JavaScript/TypeScript using the CloudQuery SDK. We reference the [Airtable integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/airtable) as an example throughout. Before starting, make sure you're familiar with CloudQuery [core concepts](/cli/integrations/creating-new-integration#core-concepts) and have completed the [Getting Started guide](/cli/getting-started). **Prerequisites:** - Node.js 20+ installed ([Node.js tutorial](https://nodejs.org/en/learn/getting-started/introduction-to-nodejs)) - CloudQuery CLI installed - Familiarity with [TypeScript](https://www.typescriptlang.org/docs/handbook/typescript-in-5-minutes.html) is helpful (the SDK emits TypeScript type definitions) ## Get Started Clone the [Airtable integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/airtable) as a starting point, then install dependencies: ```shell npm ci ``` Or install the SDK directly if starting from scratch: ```shell npm i @cloudquery/plugin-sdk-javascript ``` Use the Apache Arrow version bundled with the CloudQuery SDK. Do not install it separately via `npm`. See [example in the Airtable integration][airtable-arrow]. ## How It All Connects When a user runs `cloudquery sync`, here's what happens with your JavaScript integration: 1. The CLI starts your integration (or connects to it over gRPC if you're running it locally) 2. Your `main.ts` creates the plugin and starts the gRPC server 3. The CLI calls your `newClient` function with the user's `spec` configuration 4. `newClient` parses the spec, validates it, creates an authenticated API client, and discovers available tables 5. The CLI calls `sync`, which runs each table's resolver through the scheduler 6. Each resolver is an async function that calls `stream.write()` for each record. The SDK handles writing them to the destination. Your main implementation work is in three places: the **`newClient` function** (parsing configuration and creating the API client), the **table definitions** (defining columns and how they map to API data), and the **resolvers** (fetching data from the API). ## Project Structure Here's the structure based on the [Airtable integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/airtable): ```text plugins/source/airtable/ ├── package.json # SDK dependency (@cloudquery/plugin-sdk-javascript) ├── tsconfig.json ├── Dockerfile └── src/ ├── main.ts # Entry point ├── serve.ts # Creates serve command ├── plugin.ts # Plugin definition (newPlugin, newClient, sync) ├── tables.ts # Table definitions, column mapping, resolvers ├── spec.ts # Spec parsing + JSON schema validation └── airtable.ts # API types and enums ``` Here's what each component does: - **`main.ts`** / **`serve.ts`**: the entry point. Creates the plugin and starts the gRPC server. This is boilerplate you rarely modify. - **`plugin.ts`**: the core wiring. The `newClient` function is called when a sync starts: it parses the spec, creates the API client, discovers tables, and returns an object with `tables()`, `sync()`, and `close()` methods. This is the equivalent of Go's `Configure` function. - **`tables.ts`**: defines your tables and resolvers. Each table specifies its name, columns (with Apache Arrow types), and a resolver function that fetches data from the API. - **`spec.ts`**: defines and validates the user's configuration using JSON Schema (via Ajv). This catches invalid configuration before your code runs. - **`airtable.ts`** (or `.ts`): API-specific types, enums, and client code. Keeping this separate from CloudQuery-specific code means you can test it independently. ## Entry Point The `main.ts` is minimal boilerplate. It creates and runs the serve command: ```typescript // src/main.ts createMyServeCommand().parse(); ``` ```typescript // src/serve.ts export const createMyServeCommand = () => createServeCommand(newMyPlugin()); ``` ## Plugin Setup The plugin is created using `newPlugin()` with a `newClient` function. Unlike Go (where you define a `Configure` function) or Python (where you extend a Plugin class), the JavaScript SDK uses a functional approach: `newClient` is an async function that receives the spec, sets up your API client, and returns an object with `tables`, `sync`, and `close` methods. The `newPlugin()` call registers your integration's name, version, and team, and passes `newClient` as the initialization callback: ```typescript export const newMyPlugin = () => { const pluginClient = { // ...internal state }; const newClient = async (logger, spec, options) => { // Parse spec, create API client, discover tables return { plugin: { ...newUnimplementedDestination(), tables: () => pluginClient.allTables, sync: (options) => sync({ tables, client, stream: options.stream, /* ... */ }), close: async () => { /* cleanup */ }, }, }; }; return newPlugin('my-integration', version, newClient, { kind: 'source', team: 'my-team', jsonSchema: JSON_SCHEMA, }); }; ``` ## Define a Table In JavaScript/TypeScript, tables are created using factory functions rather than classes. Each table needs a name, a list of columns (with their Arrow types and resolvers), and a table resolver function. Unlike Go where the SDK auto-maps struct fields, in JavaScript you explicitly define each column and how it maps to the API response data using `pathResolver`: ```typescript const myTable = createTable({ name: 'my_integration_items', description: 'Items from the API', columns: [ createColumn({ name: 'id', type: new Utf8(), primaryKey: true, resolver: pathResolver('id') }), createColumn({ name: 'name', type: new Utf8(), resolver: pathResolver('name') }), createColumn({ name: 'created_at', type: new Timestamp(), resolver: pathResolver('created_at') }), createColumn({ name: 'active', type: new Bool(), resolver: pathResolver('active') }), createColumn({ name: 'metadata', type: new JSONType(), resolver: pathResolver('metadata') }), ], resolver: myTableResolver, }); ``` Key details: - Column types are **Apache Arrow types**: `Utf8`, `Timestamp`, `Float64`, `Bool`, `Int64`, `Uint64` - For JSON columns, use `JSONType()` from `@cloudquery/plugin-sdk-javascript/types/json` - Use `pathResolver('field_name')` for direct field-to-column mappings - Custom column resolvers have the signature `(client, resource, column) => Promise` ## Write a Table Resolver The resolver is an async function that fetches data from the API and writes each record to a stream. The three arguments serve the same purpose as in other SDKs, adapted for JavaScript's async patterns: ```typescript const myTableResolver = async (clientMeta, parent, stream) => { const client = clientMeta; // your API client state from newClient const items = await client.listItems(); for (const item of items) { stream.write(item); // write each record as an object } }; ``` - **`clientMeta`**: the client state you set up in `newClient`. This is where you'd access your authenticated API client, configuration values, etc. - **`parent`**: `null` for top-level tables. For child tables (e.g. fetching comments for a specific post), this contains the parent row so you can extract the parent ID. - **`stream`**: call `stream.write(record)` with plain objects whose keys match column names. Write each item as soon as you get it from the API. Don't collect everything into an array first. This keeps memory usage low and gets data to the destination faster. For paginated APIs, loop through pages inside the resolver and call `stream.write()` for each item on each page. The SDK handles batching and delivery to the destination. ## Configuration & Authentication The SDK passes the user's `spec` block as a JSON string to your `newClient` function. Use [Ajv](https://ajv.js.org/) for JSON Schema validation: ```typescript // src/spec.ts export const JSON_SCHEMA = { type: 'object', properties: { access_token: { type: 'string' }, endpoint_url: { type: 'string', default: 'https://api.example.com' }, concurrency: { type: 'number', default: 10000 }, }, required: ['access_token'], }; export const parseSpec = (spec: string) => { const parsed = JSON.parse(spec); const ajv = new Ajv({ useDefaults: true }); const validate = ajv.compile(JSON_SCHEMA); if (!validate(parsed)) { throw new Error(`Invalid spec: ${JSON.stringify(validate.errors)}`); } return parsed; }; ``` Then in your `newClient` function, parse the spec and create your API client: ```typescript const newClient = async (logger, spec, options) => { const parsedSpec = parseSpec(spec); const apiClient = new MyApiClient(parsedSpec.access_token, parsedSpec.endpoint_url); // ... }; ``` Users configure authentication in their YAML file. The CLI automatically resolves environment variable references: ```yaml spec: access_token: "${MY_API_TOKEN}" endpoint_url: "https://api.example.com" ``` The `jsonSchema` option passed to `newPlugin()` enables the CLI to validate user configuration before your code runs. ## Common Pitfalls **Avoid these common mistakes when building JavaScript integrations:** - **Don't install Apache Arrow separately.** Use the version bundled with `@cloudquery/plugin-sdk-javascript`. A separate install will cause version conflicts. - **Stream results immediately.** Call `stream.write()` for each item or page as you get it. Don't accumulate everything in an array first. - **Use `pathResolver` for direct mappings.** Don't write custom column resolvers when a field path will do. - **Handle async errors properly.** Make sure your resolver's `async` function either `await`s or catches all promises. Unhandled rejections will crash the integration. - **Pass `jsonSchema` to `newPlugin()`.** This lets the CLI validate the user's spec before your code runs, giving better error messages. ## Test Locally Start the integration as a gRPC server: ```shell npm run dev # Or after building: node dist/main.js serve ``` You can also build and run as a Docker container. See the [Airtable Dockerfile](https://github.com/cloudquery/cloudquery/blob/cb4a46ed9bdd205f19f3d92636a40914aedc6b24/plugins/source/airtable/Dockerfile). The Dockerfile exposes port **7777** and uses the entry command `node dist/main.js serve --address [::]:7777`. See [Testing Locally](/cli/integrations/creating-new-integration#testing-locally) for configuration examples and [Running Locally](/cli/advanced/running-locally) for full details. ## Publishing Visit [Publishing an Integration to the Hub](/cli/integrations/creating-new-integration/publishing) for release instructions. JavaScript integrations can also be distributed as Docker images. ## Real-World Examples - [Airtable](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/airtable): dynamic table generation from API schema with Arrow type mapping ## Next Steps Once your integration is working locally: 1. **[Publish to the Hub](/cli/integrations/creating-new-integration/publishing)**: make your integration available to others 2. **Add tests**: see `serve.test.ts` in the [Airtable integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/airtable) for a testing pattern 3. **Add JSON Schema validation**: pass `jsonSchema` to `newPlugin()` so the CLI validates user configuration before your code runs 4. **Build a Docker image**: see the [Airtable Dockerfile](https://github.com/cloudquery/cloudquery/blob/cb4a46ed9bdd205f19f3d92636a40914aedc6b24/plugins/source/airtable/Dockerfile) for a production-ready example ## Resources - [CloudQuery Community](https://community.cloudquery.io) - [JavaScript SDK Source Code](https://github.com/cloudquery/plugin-sdk-javascript) - [CloudQuery SDK on npm](https://www.npmjs.com/package/@cloudquery/plugin-sdk-javascript) [airtable-arrow]: https://github.com/cloudquery/cloudquery/blob/cb4a46ed9bdd205f19f3d92636a40914aedc6b24/plugins/source/airtable/src/tables.ts#L12 --- Source: https://www.cloudquery.io/docs/cli/integrations/creating-new-integration/publishing # Publishing an Integration to the Hub With the announcement of [CloudQuery Hub](https://www.cloudquery.io/blog/announcing-cloudquery-new-hub), we are excited to see the community contribute integrations to the Hub. This guide covers publishing both **source** and **destination** integrations; the steps are the same for both. ## Prerequisites - You have created a [CloudQuery Platform](https://cloud.cloudquery.io/) account and completed the onboarding process to create a team - You have the [CloudQuery CLI](/cli/getting-started) installed (version >= `v5.0.0`) - The integration you'd like to publish is written in one of the following: - Go with SDK version >= `v4.17.1` (source or destination) - Python with SDK version >= `v0.1.12` (source or destination) - JavaScript with SDK version >= `v0.1.0` (source only) - The integration is initialized with its name, team, and kind (`"source"` or `"destination"`). See example [here](https://github.com/cloudquery/cloudquery/blob/b7ef6f6ed8948272a429f35614fa28559397227a/plugins/source/test/resources/plugin/plugin.go#L15) - You are authenticated to [CloudQuery Platform](https://cloud.cloudquery.io/) using the `cloudquery login` command ## Create an Integration Definition on Hub Before publishing an integration, you need to create an integration definition on the CloudQuery Platform site. Log in to [CloudQuery Platform](https://cloud.cloudquery.io/). If you have not created a team yet, you will be asked to create a new one. ![CloudQuery Hub create team dialog for publishing custom integrations](/images/docs/publishing-plugins/create-team.png) The display name will be visible on Hub next to your integration. The team name is going to be used in configurations to reference your integration using `/`. To create a new integration definition, create a new integration from the Integrations tab. Fill in the necessary details and upload the integration image. When creating the definition, select the correct **Kind** (`source` or `destination`); this must match the `Kind` constant in your integration's `resources/plugin/plugin.go` file. ### Staging Releases Release your integrations for internal testing first before making it public. Set the **Visibility** to `Private` and **Release Stage** to `Preview`. This enables testing of premium integrations without being charged and with your team only (you can invite other users to your team to test the integration). When you want to release the integration in public, set the Visibility to `Public` and eventually, change the **Release Stage** to `GA` (Generally Available) to begin charging for the usage. ## Publishing an Integration ## Documentation Format - The only documentation format supported at the moment is markdown, and the `cloudquery publish` command will only upload markdown files with the `.md` extension - You can have multiple markdown files as documentation. The command concatenates files in alphabetical order, and if one of the files is named `overview.md` it will show up first - The Hub title-cases the markdown filename. For example `overview.md` will be displayed as `Overview` - The Hub does not render HTML tags in markdown files - The Hub does not support relative assets (e.g. `./assets/logo.png`). We recommend using absolute URLs for assets e.g. `https://raw.githubusercontent.com///main/assets/logo.png` in case you have the assets on GitHub ## Next Steps - [Go Source Guide](/cli/integrations/creating-new-integration/go-source): Build a source integration in Go - [Go Destination Guide](/cli/integrations/creating-new-integration/go-destination): Build a destination integration in Go - [Publishing an Addon](/cli/advanced/publishing-an-addon-to-the-hub): Publish addons alongside your integration - [Integration Architecture](/cli/core-concepts/integrations): Understand how integrations work --- Source: https://www.cloudquery.io/docs/cli/integrations/creating-new-integration/python-source # Creating a New Source Integration in Python This guide walks through building a source integration in Python using the CloudQuery SDK. We reference the [Python Integration Template](https://github.com/cloudquery/python-plugin-template/) and the [Typeform integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/typeform) as examples. Before starting, make sure you're familiar with CloudQuery [core concepts](/cli/integrations/creating-new-integration#core-concepts) and have completed the [Getting Started guide](/cli/getting-started). **Prerequisites:** - Python 3.11+ installed ([Python tutorial](https://docs.python.org/3/tutorial/)) - [uv](https://docs.astral.sh/uv/) package manager (recommended) or pip - CloudQuery CLI installed ## Get Started Clone the [Python Integration Template](https://github.com/cloudquery/python-plugin-template/) as a starting point: ```shell git clone https://github.com/cloudquery/python-plugin-template.git cq-source- cd cq-source- ``` Install dependencies using uv (recommended): ```shell uv sync ``` Or using pip in a virtual environment: ```shell pip3 install -r requirements.txt ``` If starting from scratch, install the SDK directly: ```shell uv add cloudquery-plugin-sdk # or: pip3 install cloudquery-plugin-sdk ``` ## How It All Connects When a user runs `cloudquery sync`, here's what happens with your Python integration: 1. The CLI starts your integration (or connects to it over gRPC if you're running it locally) 2. Your `main.py` creates the plugin and starts the gRPC server 3. The CLI sends the user's `spec` configuration to your plugin's `init` method 4. `init` parses the spec, validates it, creates an authenticated API client, and sets up a scheduler 5. The CLI calls `sync`, which runs each table's resolver through the scheduler 6. Each resolver is a generator that `yield`s results. The SDK handles writing them to the destination. Your main implementation work is in three places: the **`init` method** (parsing configuration and creating the API client), the **Table classes** (defining what data your integration exposes), and the **Resolvers** (fetching data from the API). ## Project Structure Here's the typical structure based on the [template](https://github.com/cloudquery/python-plugin-template/) and [Typeform integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/typeform): ```text cq-source-/ ├── main.py # Entry point ├── pyproject.toml # Dependencies (SDK >= 0.1.47) ├── Dockerfile ├── plugin/ │ ├── __init__.py # Exports the Plugin class │ ├── plugin.py # Plugin class definition │ ├── client/ │ │ └── client.py # Spec dataclass + Client wrapper │ └── tables/ │ ├── __init__.py # Exports Table classes │ └── items.py # Table + Resolver per file └── / └── client.py # Raw API client (HTTP calls, auth) ``` Here's what each component does: - **`main.py`**: the entry point. Creates your plugin and starts the gRPC server. You rarely need to modify this. - **`plugin/plugin.py`**: the **Plugin class** is the central coordinator. It implements `init` (parsing configuration), `get_tables` (listing available tables), and `sync` (running resolvers through the scheduler). This is where your integration's setup logic lives. - **`plugin/client/`**: the **Spec** dataclass defines what settings your integration accepts (API keys, endpoints, concurrency). The **Client** wrapper stores the authenticated API client and any shared state that resolvers need. - **`plugin/tables/`**: one file per **table**. Each file contains a Table class (defining the table name and columns) and a Resolver class (the generator that fetches data from the API). The resolver is the heart of each table. - **`/client.py`**: your raw API client code. This is where you make HTTP calls, handle auth headers, parse responses, and manage pagination. Keeping this separate from CloudQuery-specific code means you can test it independently. ## Entry Point The `main.py` creates and serves the plugin. This is boilerplate that you rarely need to change: ```python from cloudquery.sdk import serve from plugin import ExamplePlugin def main(): p = ExamplePlugin() serve.PluginCommand(p).run(sys.argv[1:]) if __name__ == "__main__": main() ``` ## Plugin Class The Plugin class is the central piece of your integration. It extends `plugin.Plugin` and implements four key methods that the SDK calls at different stages of a sync: - **`set_logger`**: called first, gives you a logger for debugging - **`init`**: called with the user's spec JSON. This is where you parse configuration, validate it, create your API client, and set up the scheduler - **`get_tables`**: returns the list of tables your integration supports (filtered by the user's configuration) - **`sync`**: runs the actual sync by feeding tables and resolvers to the scheduler ```python from cloudquery.sdk import plugin PLUGIN_NAME = "my-integration" PLUGIN_VERSION = "0.1.0" TEAM_NAME = "my-team" PLUGIN_KIND = "source" class MyPlugin(plugin.Plugin): def __init__(self) -> None: super().__init__( PLUGIN_NAME, PLUGIN_VERSION, plugin.plugin.Options(team=TEAM_NAME, kind=PLUGIN_KIND), ) def set_logger(self, logger): self._logger = logger def init(self, spec, no_connection=False): # Parse JSON spec, create API client and scheduler ... def get_tables(self, options): # Return filtered list of tables ... def sync(self, options): # Build resolver list, run scheduler, yield SyncMessages ... ``` ## Define a Table Unlike Go (where tables are functions returning a struct), in Python a table is a **class** that extends `Table`. You define the table name, columns, and a resolver property. Columns use [PyArrow types](https://arrow.apache.org/docs/python/api/datatypes.html) for the underlying type system. This ensures consistent types across all CloudQuery SDKs regardless of language. ```python from cloudquery.sdk.schema import Column, Table class Items(Table): def __init__(self) -> None: super().__init__( name="example_items", title="Example Items", columns=[ Column("id", pa.uint64(), primary_key=True), Column("name", pa.string()), Column("created_at", pa.timestamp(unit="s")), Column("active", pa.bool_()), ], ) @property def resolver(self): return ItemResolver(table=self) ``` Key details: - Column types are **PyArrow types**: `pa.string()`, `pa.uint64()`, `pa.timestamp()`, `pa.bool_()`, `pa.date64()`, `pa.float64()` - For JSON columns, use `JSONType()` from `cloudquery.sdk.types` - Set `primary_key=True` on primary key columns - For parent-child relations, pass `relations=[ChildTable()]` - For incremental tables, set `is_incremental=True` ## Write a Resolver The resolver is where the actual API fetching happens. In Python, resolvers work differently from Go. Instead of sending items to a channel, you write a **generator** that `yield`s results one at a time. This is the Python-native way to stream data lazily without loading everything into memory at once. A resolver is a class extending `TableResolver` that implements the `resolve` method: ```python from typing import Any, Generator from cloudquery.sdk.scheduler import TableResolver from cloudquery.sdk.schema.resource import Resource class ItemResolver(TableResolver): def __init__(self, table) -> None: super().__init__(table=table) def resolve(self, client, parent_resource: Resource) -> Generator[Any, None, None]: for item in client.api_client.list_items(): yield item # yield a dict matching the table columns ``` The `resolve` method receives two arguments: - **`client`**: your Client wrapper, which gives access to the API client and any shared state you set up in the plugin's `init` method. - **`parent_resource`**: for top-level tables this is `None`. For child tables (e.g. fetching issues for a specific project), this contains the parent row so you can access parent fields via `parent_resource.item["field_name"]`. Each `yield` produces one row in the destination table. The yielded value should be a dict whose keys match the column names defined in your Table class. The SDK handles mapping the dict values to the correct columns and types. **Important:** `resolve` must be a generator (using `yield`), not a regular function that returns a list. The SDK relies on the generator pattern to stream results incrementally. If you collect all items into a list and return them, the SDK won't receive them correctly. For parent-child table relationships, expose child resolvers via a `child_resolvers` property on your resolver class: `[table.resolver for table in self._table.relations]`. ## Configuration & Authentication The SDK passes the user's `spec` block as a JSON string to your plugin's `init` method. Define a `@dataclass` for the spec and parse it: ```python from dataclasses import dataclass, field @dataclass class Spec: access_token: str = field(default="") base_url: str = field(default="https://api.example.com") concurrency: int = field(default=100) def validate(self): if not self.access_token: raise ValueError("access_token is required") ``` Then in your plugin's `init` method, deserialize and validate: ```python def init(self, spec, no_connection=False): parsed = json.loads(spec) self._spec = Spec(**parsed) self._spec.validate() # Create authenticated API client self._api_client = MyApiClient( access_token=self._spec.access_token, base_url=self._spec.base_url, ) # Create scheduler self._scheduler = Scheduler(concurrency=self._spec.concurrency) ``` Users configure authentication in their YAML file. The CLI automatically resolves environment variable references: ```yaml spec: access_token: "${MY_API_TOKEN}" base_url: "https://api.example.com" ``` ## Common Pitfalls **Avoid these common mistakes when building Python integrations:** - **`resolve` must be a generator.** Use `yield` to produce results. Don't return a list. The SDK expects a generator for streaming. - **Yield results as you go.** Don't collect all pages into a list then yield them at the end. Yield each item or page immediately to keep memory usage low. - **Use `@dataclass` for the Spec, not Pydantic.** The SDK examples and template use standard dataclasses. - **Return errors, don't swallow them.** If an API call fails, let the exception propagate so the SDK can log it and surface it to the user. - **Use the bundled PyArrow.** Don't install a separate version of `pyarrow`. Use the one that comes with `cloudquery-plugin-sdk` for compatibility. ## Test Locally Start the integration as a gRPC server: ```shell uv run main serve # or: python main.py serve ``` Then sync using `registry: grpc`. You can also build and run as a Docker container. See the [Typeform Dockerfile](https://github.com/cloudquery/cloudquery/blob/main/plugins/source/typeform/Dockerfile). See [Testing Locally](/cli/integrations/creating-new-integration#testing-locally) for configuration examples and [Running Locally](/cli/advanced/running-locally) for full details. ## Publishing Visit [Publishing an Integration to the Hub](/cli/integrations/creating-new-integration/publishing) for release instructions. Python integrations can also be distributed as Docker images. ## Real-World Examples - [Python Integration Template](https://github.com/cloudquery/python-plugin-template/): starter template for new integrations - [xkcd Example](https://github.com/cloudquery/python-plugin-template/tree/XKCD-Example): a starter integration built from the template - [Typeform](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/typeform): single-endpoint REST API integration - [Square](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/square): OpenAPI-generated tables using `cloudquery.sdk.transformers.openapi` ## Next Steps Once your integration is working locally: 1. **[Publish to the Hub](/cli/integrations/creating-new-integration/publishing)**: make your integration available to others 2. **Add tests**: see the test patterns in [Typeform](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/typeform) for reference 3. **Add incremental tables**: set `is_incremental=True` on tables and use the state client for cursor management 4. **Try OpenAPI generation**: if your API has an OpenAPI spec, see the [Square integration](https://github.com/cloudquery/cloudquery/tree/main/plugins/source/square) for auto-generating tables ## Resources - [CloudQuery Community](https://community.cloudquery.io) - [How to Write a Source Integration in Python](https://youtu.be/TSbGHz5Z09M) (Video. May reference older SDK patterns; use this guide for current code.) - [Python SDK Source Code](https://github.com/cloudquery/plugin-sdk-python) - [CloudQuery SDK on PyPI](https://pypi.org/project/cloudquery-plugin-sdk/) --- Source: https://www.cloudquery.io/docs/cli/integrations/destinations # Destination Integrations Destination integrations are responsible for writing data from source integrations to various storage systems. They handle schema migration, data persistence, and optimization for different storage backends. ## Find Your Integration's Docs Full documentation for each destination integration (configuration reference, write modes, and connection details) lives on the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/destination). Jump directly to any integration below. ### Data Warehouses & Lakes | Integration | Hub Docs | | ----------- | ------------------------------------------------------------------------------------------- | | PostgreSQL | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/postgresql/latest/docs) | | BigQuery | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/bigquery/latest/docs) | | Snowflake | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/snowflake/latest/docs) | | Databricks | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/databricks/latest/docs) | | ClickHouse | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/clickhouse/latest/docs) | | DuckDB | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/duckdb/latest/docs) | ### Storage & Files | Integration | Hub Docs | | ------------------- | --------------------------------------------------------------------------------------- | | File (S3/GCS/Local) | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/file/latest/docs) | | GCS | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/gcs/latest/docs) | | Azure Blob | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/azblob/latest/docs) | ### Other Databases | Integration | Hub Docs | | ------------- | ---------------------------------------------------------------------------------------------- | | MySQL | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/mysql/latest/docs) | | MongoDB | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/mongodb/latest/docs) | | Elasticsearch | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/elasticsearch/latest/docs) | | SQLite | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/sqlite/latest/docs) | [Browse all destination integrations →](https://www.cloudquery.io/hub/plugins/destination) ## Popular Destination Categories ### Relational Databases - **PostgreSQL**: Full-featured relational database with JSON support - **MySQL**: Popular open-source database with wide compatibility - **SQLite**: Lightweight, file-based database for development and testing - **SQL Server**: Microsoft's enterprise database platform ### Data Warehouses - **BigQuery**: Google's serverless data warehouse - **Snowflake**: Cloud-based data platform for analytics - **ClickHouse**: High-performance analytical database - **Redshift**: Amazon's cloud data warehouse ### NoSQL Databases - **MongoDB**: Document-based NoSQL database - **Elasticsearch**: Search and analytics engine - **Neo4j**: Graph database for relationship data ### Message Queues & Streaming - **Kafka**: Distributed streaming platform - **RabbitMQ**: Message broker for applications - **Apache Pulsar**: Cloud-native messaging and streaming ### File Storage - **S3**: Amazon's object storage service - **GCS**: Google Cloud Storage - **Azure Blob**: Microsoft's object storage - **Local Files**: CSV, JSON, Parquet formats ## Destination Configuration Reference ### Basic Configuration Destination integrations are configured in your CloudQuery configuration file. Each destination requires: - **Name**: Unique identifier for the destination - **Path**: Plugin path (e.g., `cloudquery/postgresql`) - **Version**: Plugin version to use - **Write Mode**: How to handle data updates - **Connection Details**: Database connection information Example configuration: ```yaml kind: destination spec: name: postgresql path: cloudquery/postgresql registry: cloudquery version: "VERSION_DESTINATION_POSTGRESQL" spec: connection_string: "postgresql://user:pass@localhost:5432/dbname" ``` ### Complete Destination Spec Reference Available options for the destination integration `spec` object: #### name (`string`, required) Name of the integration. If you have multiple destination integrations, this must be unique. The name field may be used to uniquely identify a particular destination configuration. For example, if you have two configs for the PostgreSQL plugin for syncing different databases, one may be named `db-1` and the other `db-2`. In this case, the `path` option below must be used to specify the download path for the plugin. #### registry (`string`, optional, default: `cloudquery`, available: `github`, `cloudquery`, `local`, `grpc`, `docker`) - `cloudquery`: CloudQuery will look for and download the plugin from the official CloudQuery registry, and then execute it. - `github`: **Deprecated**. CloudQuery will look for and download the plugin from GitHub, and then execute it. - `local`: CloudQuery will execute the plugin from a local path. - `grpc`: mostly useful in debug mode when plugin is already running in a different terminal, CloudQuery will connect to the gRPC plugin server directly without spawning the process. - `docker`: CloudQuery will run the plugin in a Docker container. This is most useful for plugins written in Python, as they do not support the `local`, `github` and `cloudquery` registries. #### docker_registry_auth_token (`string`, optional, default: `""`, introduced in CLI `v5.7.0`) Authentication token for private Docker container registries. Required if the destination plugin is hosted in a private Docker registry. Only relevant when `registry` is set to `docker`. The token is a base64-encoded JSON string: ```shell echo -n "{\"username\":\"\",\"password\":\"\"}" | base64 ``` Details about specific private container registries: **AWS ECR:** The username is `AWS` and you can get the password by running `aws ecr get-login-password --region `. ```shell echo -n "{\"username\":\"AWS\",\"password\":\"$(aws ecr get-login-password --region )\"}" | base64 ``` **GitHub Container Registry:** The username is your GitHub username and you use a personal access token as the password. More information can be found [in the GitHub docs](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic). ```shell export CR_PAT=YOUR_TOKEN echo -n "{\"username\":\"USERNAME\",\"password\":\"$CR_PAT\"}" | base64 ``` #### path (`string`, required) Configures how to retrieve the plugin. The contents depend on the value of `registry` (`cloudquery` by default). - For plugins hosted on the CloudQuery registry, `path` should be of the form `"/"`. For official plugins, should be `cloudquery/`. - For plugins hosted on GitHub, `path` should be of the form `"/"`. - For plugins that are located in the local filesystem, `path` should be a filesystem path to the plugin binary. - To connect to a running plugin via `grpc` (mostly useful for debugging), `path` should be the host-port of the plugin (e.g. `localhost:7777`). #### version (`string`, required) `version` must be a valid [SemVer](https://semver.org/), e.g. `vMajor.Minor.Patch`. You can find all official plugin versions under [our GitHub releases page](https://github.com/cloudquery/cloudquery/releases), and for community integrations you can find it in the relevant community repository. Required for integrations using the `cloudquery` or `github` registries. #### write_mode (`string`, optional, default: `overwrite-delete-stale`. Available: `overwrite-delete-stale`, `overwrite`, `append`) Specifies the update method to use when inserting rows. The exact semantics depend on the destination plugin, and not all destinations support every option, so check the destination plugin documentation for details. - `overwrite-delete-stale`: `sync`s overwrite existing rows with the same primary key, and delete rows that are no longer present in the cloud. - `overwrite`: Same as `overwrite-delete-stale`, but doesn't delete stale rows from previous `sync`s. - `append`: Rows are never overwritten or deleted, only appended. > Switching from `overwrite-delete-stale` or `overwrite` to `append`, or from `append` to `overwrite-delete-stale` or `overwrite` is not supported without dropping all tables specified in the configuration. > To drop tables automatically, use the `migrate_mode: forced` option. #### migrate_mode (`string`, optional, default: `safe`. Available: `safe`, `forced`) Specifies the migration mode to use when source tables are changed. In `safe` mode (the default), CloudQuery will not run migrations that would result in data loss, and will print an error instead. In `forced` mode, CloudQuery will run migrations that may result in data loss and the migration should succeed without errors, unless a table has user created dependent objects (e.g. views). Not all destination plugins support `migrate_mode: forced`, refer to the specific destination plugin page to see if it is supported. Read more about how CloudQuery handles migrations [here](/cli/managing-cloudquery/migrations). #### pk_mode (`string`, optional, default: `default`, Available: `default`, `cq-id-only` introduced in CLI `v2.5.2`) Specifies the Primary Keys that the destination will configure when using the `overwrite` or `overwrite-delete-stale` mode. - `default`: The default primary keys are used. - `cq-id-only`: The `_cq_id` field is used as the only primary key for each table. This is useful when you don't want breaking changes to primary keys to impact your schema. It is highly recommended that if you are using this feature you should also use the [`deterministic_cq_id` feature in the source](/cli/integrations/sources#deterministic_cq_id). If you are using `overwrite` mode and a source updates a primary key, this will result in a new row being inserted. If you are using `overwrite-delete-stale` mode, a new row will be inserted and the old row will be deleted as a stale resource. Note: using this parameter might result in changes to query performance as CloudQuery will not be creating indexes for the default primary key columns. Supported by destination plugins released on 2023-03-21 and later #### sync_group_id Supported only for `write_mode: append` and `write_mode: overwrite` modes at the moment. A value for an additional column named `_cq_sync_group_id` that will be added to each table. In `overwrite` mode the column will be added as an additional primary key. This is useful when splitting a sync into [multiple parallel jobs](/cli/managing-cloudquery/running-in-parallel). Using the same `sync_group_id` allows identifying separate syncs jobs as belonging to the same group. The value supports the following placeholders: `{{SYNC_ID}}, {{YEAR}}, {{MONTH}}, {{DAY}}, {{HOUR}}, {{MINUTE}}` which are set at sync time. Common use cases include: 1. Setting `sync_group_id: "{{YEAR}}-{{MONTH}}-{{DAY}}"` to group syncs by day, providing a historical view of the data partitioned by day. 2. Setting `sync_group_id: "{{SYNC_ID}}"` to enable joining data from different tables that were all part of the same sync job. The value of `SYNC_ID` can be controlled using the `--invocation-id` flag passed to the `cloudquery sync` command. #### send_sync_summary (`bool`, optional) When set to `true`, CloudQuery will send a summary of the sync to the destination plugin. The summary includes the number of resources synced, number of errors and details about the plugins (both source and destination). This information will be available in the destination as a separate table named `cloudquery_sync_summaries`. #### spec (`object`, optional) Integration-specific configurations. Visit [destination integrations](https://www.cloudquery.io/hub/plugins/destination) documentation for more information. The following options are available for most destination plugins **under the nested plugin spec**: ##### batch_size (`int`, optional) The number of resources to insert in a single batch. Only applies to plugins that utilize batching. This setting works in conjunction with `batch_size_bytes`, and batches are written whenever either `batch_size` or `batch_size_bytes` is reached. Every plugin has its own default value for `batch_size`. ##### batch_size_bytes (`int`, optional) The max number of bytes to use for a single batch. Only applies to plugins that utilize batching. This setting works in conjunction with `batch_size`, and batches are written whenever either `batch_size` or `batch_size_bytes` is reached. Every plugin has its own default value for `batch_size_bytes`. Note that the size in bytes is calculated based on the size of data in memory, not the serialized data, and it is best to choose a `batch_size_bytes` significantly lower than any hard limits. ## Write Modes Different write modes control how data is updated: - **`overwrite-delete-stale`**: Replace existing data and remove stale records - **`overwrite`**: Replace existing data but keep stale records - **`append`**: Only add new data, never update or delete ## Performance Optimization ### Batch Settings - **batch_size**: Number of records per batch - **batch_size_bytes**: Maximum bytes per batch - **batch_timeout**: Maximum time to wait before flushing ### Schema Optimization - **Primary Keys**: Configure primary key strategy - **Indexes**: Create indexes for better query performance - **Partitioning**: Use table partitioning for large datasets ## Migration Handling CloudQuery automatically handles schema migrations when source schemas change: - **Safe Mode**: Prevents data loss, requires manual intervention - **Forced Mode**: Automatically applies migrations that may cause data loss ## Creating Custom Destinations Need a destination that doesn't exist? Learn how to [create your own destination integration](/cli/integrations/creating-new-integration). ## Next Steps - [Source Integrations](/cli/integrations/sources) - Configure data sources for your syncs - [Transformer Integrations](/cli/integrations/transformers) - Transform data between sources and destinations - [Schema Migrations](/cli/managing-cloudquery/migrations) - Understand how CloudQuery handles schema changes - [Performance Tuning](/cli/advanced/performance-tuning) - Optimize write performance and batch settings - [Dashboards & Visualizations](/cli/core-concepts/dashboards) - Visualize synced data with BI tools --- Source: https://www.cloudquery.io/docs/cli/integrations/hub-directory # Integration Directory Browse all available integrations on the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source). Each link below goes to the full documentation for that integration, including configuration reference, supported tables, and version history. ## Source Integrations Source integrations extract data from cloud providers, SaaS tools, security platforms, and databases. [Browse all 70+ source integrations →](https://www.cloudquery.io/hub/plugins/source) ### Cloud Infrastructure | Integration | Description | Hub Docs | | ------------------------ | ------------------------------------------------------ | ---------------------------------------------------------------------------------------- | | AWS | EC2, S3, IAM, Lambda, RDS, and 300+ other AWS services | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs) | | AWS Cost & Usage Reports | AWS billing and cost data from CUR | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/awscur/latest/docs) | | GCP | Compute, Storage, IAM, BigQuery, GKE, and more | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs) | | Azure | VMs, Storage, Active Directory, AKS, and more | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/azure/latest/docs) | | Kubernetes | Cluster resources, workloads, RBAC, and networking | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) | | Oracle Cloud | Oracle Cloud Infrastructure resources | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/oracle/latest/docs) | | DigitalOcean | Droplets, spaces, networking, and more | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/digitalocean/latest/docs) | | AliCloud | Alibaba Cloud resources | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/alicloud/latest/docs) | ### Security & Identity | Integration | Description | Hub Docs | | ---------------- | ------------------------------------------- | -------------------------------------------------------------------------------------------- | | Okta | Identity and access visibility | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/okta/latest/docs) | | Entra ID | Microsoft Azure Active Directory / Entra ID | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/entraid/latest/docs) | | CrowdStrike | Endpoint security data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/crowdstrike/latest/docs) | | SentinelOne | Endpoint detection and response data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/sentinelone/latest/docs) | | Wiz | Cloud security posture data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/wiz/latest/docs) | | Palo Alto Cortex | Palo Alto Networks Cortex data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/palo-alto-cortex/latest/docs) | | Snyk | Vulnerability and dependency data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/snyk/latest/docs) | | Cloudflare | DNS, firewall, and network data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/cloudflare/latest/docs) | ### Engineering & DevOps | Integration | Description | Hub Docs | | ----------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | GitHub | Repositories, teams, members, actions, and security advisories | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/github/latest/docs) | | GitLab | Projects, merge requests, pipelines, and users | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/gitlab/latest/docs) | | Backstage | Software catalog and developer portal data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/backstage/latest/docs) | | Datadog | Monitoring data, dashboards, and alerts | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/datadog/latest/docs) | | PagerDuty | Incident, on-call schedule, and service data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/pagerduty/latest/docs) | ### Cloud FinOps | Integration | Description | Hub Docs | | ----------- | --------------------------------------- | -------------------------------------------------------------------------------------- | | AWS Pricing | AWS service pricing catalog | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/awspricing/latest/docs) | | Stripe | Payment, billing, and subscription data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/stripe/latest/docs) | ### Databases | Integration | Description | Hub Docs | | ----------- | -------------------------------------- | -------------------------------------------------------------------------------------- | | PostgreSQL | Extract data from PostgreSQL databases | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/postgresql/latest/docs) | | MySQL | Extract data from MySQL databases | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/mysql/latest/docs) | --- ## Destination Integrations Destination integrations write synced data to databases, data warehouses, and storage systems. [Browse all destination integrations →](https://www.cloudquery.io/hub/plugins/destination) ### Data Warehouses & Lakes | Integration | Description | Hub Docs | | ----------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | PostgreSQL | SQL database for joins, queries, and compliance reporting | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/postgresql/latest/docs) | | BigQuery | Google Cloud analytics and AI | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/bigquery/latest/docs) | | Snowflake | Enterprise cloud data warehouse | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/snowflake/latest/docs) | | Databricks | Unified analytics and AI platform | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/databricks/latest/docs) | | ClickHouse | Column-oriented analytics database | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/clickhouse/latest/docs) | | DuckDB | In-process analytical database | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/duckdb/latest/docs) | ### Storage & Files | Integration | Description | Hub Docs | | ------------------- | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | File (S3/GCS/Local) | Write to CSV, JSON, or Parquet files in object stores or local disk | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/file/latest/docs) | | GCS | Google Cloud Storage | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/gcs/latest/docs) | | Azure Blob | Azure Blob Storage | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/azblob/latest/docs) | ### Other Databases | Integration | Description | Hub Docs | | ------------- | ----------------------------- | ---------------------------------------------------------------------------------------------- | | MySQL | Write data into MySQL | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/mysql/latest/docs) | | MongoDB | Write data into MongoDB | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/mongodb/latest/docs) | | Elasticsearch | Write data into Elasticsearch | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/elasticsearch/latest/docs) | | SQLite | Lightweight local database | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/sqlite/latest/docs) | --- ## Transformer Integrations Pre-load transformer integrations modify records as they flow from source to destination, before the data is written. See [Transformer Integrations](/cli/integrations/transformers) for configuration details. | Integration | Description | Hub Docs | | -------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | Basic | Rename tables, add or remove columns, obfuscate sensitive data, filter rows, add timestamps | [Docs](https://www.cloudquery.io/hub/plugins/transformer/cloudquery/basic/latest/docs) | | JSON Flattener | Flatten single-level JSON fields into individually typed destination columns | [Docs](https://www.cloudquery.io/hub/plugins/transformer/cloudquery/jsonflattener/latest/docs) | --- ## Transformations (Post-Load dbt Packages) Post-load transformation addons are dbt packages that normalize and enrich your synced data after it lands in the destination. [Browse all transformations →](https://www.cloudquery.io/hub/addons/transformation) | Transformation | Description | Hub Docs | | --------------------- | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | AWS Asset Inventory | Normalize AWS resources into a unified asset inventory schema | [Docs](https://www.cloudquery.io/hub/addons/transformation/cloudquery/aws-asset-inventory/latest/docs) | | Azure Asset Inventory | Normalize Azure resources into a unified asset inventory schema | [Docs](https://www.cloudquery.io/hub/addons/transformation/cloudquery/azure-asset-inventory/latest/docs) | | GCP Asset Inventory | Normalize GCP resources into a unified asset inventory schema | [Docs](https://www.cloudquery.io/hub/addons/transformation/cloudquery/gcp-asset-inventory/latest/docs) | | AWS Cost | Process and model AWS cost and usage data for analysis | [Docs](https://www.cloudquery.io/hub/addons/transformation/cloudquery/aws-cost/latest/docs) | | AWS Data Resilience | Evaluate AWS backup coverage and data resilience posture | [Docs](https://www.cloudquery.io/hub/addons/transformation/cloudquery/aws-data-resilience/latest/docs) | --- ## Next Steps - [Source Integrations Reference](/cli/integrations/sources): Full source spec documentation - [Destination Integrations Reference](/cli/integrations/destinations): Full destination spec documentation - [Transformer Integrations Reference](/cli/integrations/transformers): How to use pre-load transformers - [Creating New Integrations](/cli/integrations/creating-new-integration): Build and publish your own integration --- Source: https://www.cloudquery.io/docs/cli/integrations # Integrations CloudQuery's modular architecture allows you to connect 70+ data sources to any destination of your choice. This section covers all available integrations and how to work with them. Learn more about [integration architecture](/cli/core-concepts/integrations) and [how syncs work](/cli/core-concepts/syncs). ## Source Integrations Source integrations extract data from cloud providers, databases, SaaS applications, and other APIs. They handle authentication, data extraction, and schema definition. [Browse Source Integrations →](/cli/integrations/sources) ## Destination Integrations Destination integrations write data to databases, data warehouses, message queues, and storage systems. They handle schema migration and data persistence. [Browse Destination Integrations →](/cli/integrations/destinations) ## Creating Custom Integrations Learn how to build your own source or destination integrations using CloudQuery's SDKs. After development, you can [publish integrations to the hub](/cli/integrations/creating-new-integration/publishing). [Creating New Integration →](/cli/integrations/creating-new-integration) ## Integration Architecture CloudQuery integrations communicate over [gRPC](https://github.com/cloudquery/plugin-pb) and are language-agnostic. They can be implemented in Go, Python, Java, JavaScript, and other languages that support gRPC and Apache Arrow. CloudQuery's modular architecture lets you mix any source with any destination. Integrations can be written in any language, are built on Apache Arrow for fast data processing, and can be either official or community-maintained. ## Getting Started 1. **Choose a Source**: Browse our [source integrations](/cli/integrations/sources) to find the data you need 2. **Select a Destination**: Pick from our [destination integrations](/cli/integrations/destinations) for your data warehouse 3. **Configure**: Use our [configuration guide](/cli/core-concepts/configuration) to set up your sync 4. **Run**: Execute your first sync with `cloudquery sync` ## Next Steps - [Transformer Integrations](/cli/integrations/transformers) - Process data between sources and destinations - [Performance Tuning](/cli/advanced/performance-tuning) - Optimize sync speed and resource usage - [Deployment Options](/cli/managing-cloudquery/deployments/overview) - Run syncs in production environments - [Publishing Integrations](/cli/integrations/creating-new-integration/publishing) - Share your custom integrations on the hub --- Source: https://www.cloudquery.io/docs/cli/integrations/sources # Source Integrations Source integrations are responsible for extracting and transforming data from various APIs, cloud providers, SaaS applications, and databases. They define the schema (tables) and handle authentication with the supported services. ## Find Your Integration's Docs Full documentation for each source integration (configuration reference, supported tables, and authentication) lives on the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source). Jump directly to any integration below. ### Cloud Infrastructure | Integration | Hub Docs | | ------------------------ | ---------------------------------------------------------------------------------------- | | AWS | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs) | | AWS Cost & Usage Reports | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/awscur/latest/docs) | | GCP | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs) | | Azure | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/azure/latest/docs) | | Kubernetes | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) | | Oracle Cloud | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/oracle/latest/docs) | | DigitalOcean | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/digitalocean/latest/docs) | | AliCloud | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/alicloud/latest/docs) | ### Security & Identity | Integration | Hub Docs | | ---------------- | -------------------------------------------------------------------------------------------- | | Okta | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/okta/latest/docs) | | Entra ID | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/entraid/latest/docs) | | CrowdStrike | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/crowdstrike/latest/docs) | | SentinelOne | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/sentinelone/latest/docs) | | Wiz | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/wiz/latest/docs) | | Palo Alto Cortex | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/palo-alto-cortex/latest/docs) | | Snyk | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/snyk/latest/docs) | | Cloudflare | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/cloudflare/latest/docs) | ### Engineering & DevOps | Integration | Hub Docs | | ----------- | ------------------------------------------------------------------------------------- | | GitHub | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/github/latest/docs) | | GitLab | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/gitlab/latest/docs) | | Backstage | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/backstage/latest/docs) | | Datadog | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/datadog/latest/docs) | | PagerDuty | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/pagerduty/latest/docs) | ### Cloud FinOps | Integration | Hub Docs | | ----------- | -------------------------------------------------------------------------------------- | | AWS Pricing | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/awspricing/latest/docs) | | Stripe | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/stripe/latest/docs) | ### Databases | Integration | Hub Docs | | ----------- | -------------------------------------------------------------------------------------- | | PostgreSQL | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/postgresql/latest/docs) | | MySQL | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/mysql/latest/docs) | [Browse all 70+ source integrations →](https://www.cloudquery.io/hub/plugins/source) ## Popular Source Categories ### Cloud Providers - **AWS**: EC2, S3, RDS, Lambda, and 200+ other services - **Google Cloud**: Compute Engine, Cloud Storage, BigQuery, and more - **Azure**: Virtual Machines, Storage Accounts, SQL Database, and more - **DigitalOcean**: Droplets, Spaces, Load Balancers, and more ### SaaS Applications - **GitHub**: Repositories, issues, pull requests, and user data - **GitLab**: Projects, merge requests, pipelines, and more - **Slack**: Channels, messages, users, and workspace data - **Salesforce**: Accounts, contacts, opportunities, and custom objects ### Databases & APIs - **PostgreSQL**: Database schemas, tables, and metadata - **MongoDB**: Collections, documents, and database information - **REST APIs**: Generic REST API integration for any HTTP service ## Source Configuration Reference ### Basic Configuration Source integrations are configured in your CloudQuery configuration file. Each source requires: - **Name**: Unique identifier for the source - **Path**: Plugin path (e.g., `cloudquery/aws`) - **Version**: Plugin version to use - **Tables**: Which tables to sync (supports wildcards) - **Destinations**: Where to send the data Example configuration: ```yaml kind: source spec: name: aws path: cloudquery/aws registry: cloudquery version: "VERSION_SOURCE_AWS" tables: ["aws_s3_buckets", "aws_ec2_instances"] destinations: ["postgresql"] spec: # AWS-specific configuration regions: ["us-east-1", "us-west-2"] ``` ### Complete Source Spec Reference Following are all available options for the top level source integration `spec` object. For individual integration configuration, see the relevant integration page on [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source) (e.g. [AWS integration configuration](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs)). #### name (`string`, required) Name of the integration. If you have multiple source integrations, this must be unique. The name field may be used to uniquely identify a particular source configuration. For example, if you have two configs for the AWS plugin for syncing different accounts, one may be named `aws-account-1` and the other `aws-account-2`. In this case, the `path` option below must be used to specify the download path for the plugin. #### registry (`string`, optional, default: `cloudquery`, available: `github`, `cloudquery`, `local`, `grpc`, `docker`) - `cloudquery`: CloudQuery will look for and download the plugin from the official CloudQuery registry, and then execute it. - `github`: **Deprecated**. CloudQuery will look for and download the plugin from GitHub, and then execute it. - `local`: CloudQuery will execute the plugin from a local path. - `grpc`: mostly useful in debug mode when plugin is already running in a different terminal, CloudQuery will connect to the gRPC plugin server directly without spawning the process. - `docker`: CloudQuery will run the plugin in a Docker container. This is most useful for plugins written in Python, as they do not support the `local`, `github` and `cloudquery` registries. #### docker_registry_auth_token (`string`, optional, default: `""`, introduced in CLI `v5.7.0`) Authentication token for private Docker container registries. This is required if the plugin is hosted in a private Docker container registry. The token should be a valid Docker registry token that can be used to pull the plugin image. This option is only relevant when `registry` is set to `docker`. The token is a base64 encoded string. Here is an example of how to generate the token: ```shell echo -n "{\"username\":\"\",\"password\":\"\"}" | base64` ``` Details about specific private container registries: **AWS ECR:** The username is `AWS` and you can get the password by running `aws ecr get-login-password --region `. Replace `` with the region where the ECR is located. Generating the token for AWS ECR would look like this: ```shell echo -n "{\"username\":\"AWS\",\"password\":\"$(aws ecr get-login-password --region )\"}" | base64 ``` **GitHub Container Registry:** The username is `USERNAME` and you use a personal access token as the password. More information can be found [here](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic) Generating the token for GitHub Container Registry would look like this: ```shell export CR_PAT=YOUR_TOKEN echo -n "{\"username\":\"USERNAME\",\"password\":\"$CR_PAT\"}" | base64 ``` #### path (`string`, required) Configures how to retrieve the plugin. The contents depend on the value of `registry` (`cloudquery` by default). - For plugins hosted on the CloudQuery registry, `path` should be of the form `"/"`. For official plugins, should be `cloudquery/`. - For plugins hosted on GitHub, `path` should be of the form `"/"`. - For plugins that are located in the local filesystem, `path` should be a filesystem path to the plugin binary. - To connect to a running plugin via `grpc` (mostly useful for debugging), `path` should be the host-port of the plugin (e.g. `localhost:7777`). - For plugins distributed via Docker, `path` should be the name of the Docker image (optionally including a tag, the same as you would use for `docker run`, e.g. `ghcr.io/cloudquery/cq-source-typeform:v1.0.0`). #### version (`string`, required) `version` must be a valid [SemVer](https://semver.org/), e.g. `vMajor.Minor.Patch`. You can find all official plugin versions under [cloudquery/cloudquery/releases](https://github.com/cloudquery/cloudquery/releases), and for community integrations you can find it in the relevant community repository. Required for integrations using the `cloudquery` or `github` registries. #### tables (`[]string`, required) > This option was changed to required in versions >= `v3.0.0` of the CloudQuery CLI. In previous versions it was optional and defaulted to `["*"]` (sync all tables). Tables to sync from the source plugin. It accepts wildcards. For example, to match all tables use `["*"]` and to match all EC2-related tables use `aws_ec2_*`. Syncing all tables can be slow on some integrations (e.g. AWS, GCP, Azure). Prior to CLI version `v6.0.0`, matched tables would also sync all their descendant tables, unless these were skipped in `skip_tables`. You must now explicitly list descendant tables in `tables` to sync them or set `skip_dependent_tables` to `false`. #### skip_tables (`[]string`, optional, default: `[]`) Specify which tables to skip when syncing the source plugin. It accepts wildcards. This config is useful when using wildcards in `tables`, or when you wish to skip dependent tables. If a table with dependencies is skipped, all its dependent tables will also be skipped. #### skip_dependent_tables (`bool`, optional, default: `true`, introduced in CLI `v2.3.7`) If set to `false`, dependent tables will be included in the sync when their parents are matched, even if not explicitly included by the `tables` configuration. Prior to CLI version `v6.0.0`, this option defaulted to `false`. We've changed the default to `true` to avoid new tables implicitly being synced when added to plugins. #### destinations (`[]string`, required) Specify the names of the destinations to sync the data of the source plugin to. #### deterministic_cq_id (`bool`, optional, default: `false`, introduced in CLI `v2.4.1`) A flag that indicates whether the value of `_cq_id` should be a UUID that is a hash of the primary keys or a random UUID. If a resource has no primary keys defined the value will always be a random UUID. Supported by source plugins released on 2023-03-08 and later #### backend_options (`object`, optional) Configures the state backend for [incremental table](/cli/advanced/managing-incremental-tables) syncs. Contains two required sub-fields: - `table_name` (`string`, required): The name of the table used to store key-value pairs for incremental sync progress. - `connection` (`string`, required): Connection reference for the destination integration that stores the state. Typically uses the `@@plugins..connection` variable syntax to reference another integration's connection. Example: ```yaml backend_options: table_name: cq_state_aws connection: "@@plugins.postgresql.connection" ``` See [Managing Incremental Tables](/cli/advanced/managing-incremental-tables) for more details. #### otel_endpoint (preview) (`string`, optional, introduced in CLI `v3.10.0`) OpenTelemetry [OTLP/HTTP](https://opentelemetry.io/docs/specs/otel/protocol/exporter/) endpoint for sending sync traces. Also supports Jaeger endpoints. See [Monitoring](/cli/managing-cloudquery/monitoring/overview) for setup instructions, Jaeger examples, and Datadog/Grafana integration guides. #### otel_endpoint_insecure (preview) (`bool`, optional, default: `false`, introduced in CLI `v3.10.0`) If set to `true`, the exporter will connect via `http` instead of `https` (skip TLS verification). Only for development/local use. #### spec (`object`, optional) Integration-specific configurations. Visit [source integrations](https://www.cloudquery.io/hub/plugins/source) documentation for more information. ### Deprecated Options #### concurrency This option was deprecated in CLI `v3.6.0` in favor of plugin level concurrency, as each integration has its own concurrency requirements. See more in each integration's documentation. #### scheduler This option was deprecated in CLI `v3.6.0` in favor of plugin level scheduler, as each integration has its own scheduler requirements. See more in each integration's documentation. #### backend This option was deprecated in CLI `v3.6.0` in favor of `backend_options`. See [Managing Incremental Tables](/cli/advanced/managing-incremental-tables) for more information. #### backend_spec This option was deprecated in CLI `v3.6.0` in favor of `backend_options`. See [Managing Incremental Tables](/cli/advanced/managing-incremental-tables) for more information. ## Authentication Each source integration handles authentication differently: - **Cloud Providers**: Use IAM roles, service accounts, or API keys - **SaaS Applications**: OAuth tokens, API keys, or personal access tokens - **Databases**: Connection strings with credentials Refer to each integration's documentation for specific authentication requirements. ## Performance Considerations - **Table Selection**: Use specific table names instead of wildcards when possible - **Rate Limiting**: Some APIs have rate limits that affect sync performance - **Incremental Syncs**: Many sources support incremental syncing for better performance - **Concurrency**: Adjust concurrency settings based on API limits ## Creating Custom Sources Need a source that doesn't exist? Learn how to [create your own source integration](/cli/integrations/creating-new-integration). ## Next Steps - [Destination Integrations](/cli/integrations/destinations) - Configure where your synced data is stored - [Transformer Integrations](/cli/integrations/transformers) - Transform data between sources and destinations - [Managing Incremental Tables](/cli/advanced/managing-incremental-tables) - Set up and manage incremental syncs - [Performance Tuning](/cli/advanced/performance-tuning) - Optimize source sync speed and resource usage - [Monitoring](/cli/managing-cloudquery/monitoring/overview) - Set up observability for your syncs --- Source: https://www.cloudquery.io/docs/cli/integrations/transformers # Transformer Integrations Transformer integrations are pre-load integrations that modify data as it flows from a source to a destination during a sync. They let you rename tables, remove sensitive columns, filter rows, flatten JSON, and more - without writing custom code or touching your source or destination configuration. For an overview of how transformations fit into the broader CloudQuery pipeline (including post-load dbt transformations), see [Transformations](/cli/core-concepts/transformations). ## How Transformer Integrations Work Transformers sit between a source and a destination in the sync pipeline: 1. **Data Interception**: The transformer receives records after they leave the source integration. 2. **Transformation**: It applies configured transformations to each record (schema changes, column modifications, row filtering, etc.). 3. **Data Delivery**: The transformed records are sent to the destination integration for writing. You wire a transformer into a destination by adding its name to the `transformers` list in the destination spec. The CLI chains them automatically. ## Available Transformer Integrations There are two official transformer integrations: | Transformer | Description | | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | [Basic](https://www.cloudquery.io/hub/plugins/transformer/cloudquery/basic/latest/docs) | Rename tables, add or remove columns, obfuscate sensitive data, normalize casing, drop rows by value, and add timestamps or literal columns. | | [JSON Flattener](https://www.cloudquery.io/hub/plugins/transformer/cloudquery/jsonflattener/latest/docs) | Flatten single-level JSON object fields into individually typed destination columns while preserving the original JSON column. | You can also browse all transformations (including post-load dbt packages) in the [CloudQuery Hub](https://www.cloudquery.io/hub/addons/transformation). ## Configuration A transformer integration requires a `kind: transformer` spec in your configuration file, and the destination must reference it by name. Here's a complete example that syncs AWS data through a Basic transformer into PostgreSQL: ```yaml copy kind: source spec: name: "aws" path: "cloudquery/aws" registry: "cloudquery" version: "VERSION_SOURCE_AWS" destinations: ["postgresql"] tables: ["aws_s3_buckets", "aws_ec2_instances"] --- kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" registry: "cloudquery" version: "VERSION_DESTINATION_POSTGRESQL" write_mode: "overwrite-delete-stale" migrate_mode: forced transformers: - "basic" spec: connection_string: "postgresql://user:password@localhost:5432/db_name" --- kind: transformer spec: name: "basic" path: "cloudquery/basic" registry: "cloudquery" version: "VERSION_TRANSFORMER_BASIC" spec: transformations: - kind: obfuscate_sensitive_columns - kind: change_table_names tables: ["*"] new_table_name_template: "cq_sync_{{.OldName}}" ``` The destination references `"basic"` by name. The CLI matches this to the transformer spec with `name: "basic"` and routes records through it before writing. If you plan to modify the schema from a previous sync (e.g. renaming tables or removing columns), set `migrate_mode: forced` on the destination so the schema is recreated. ## Transformer Spec Reference Available options for the transformer integration `spec` object: ### name (`string`, required) Name of the integration. Must be unique if you have multiple transformer integrations. The name is how destinations reference this transformer. For example, if you have two configs for the Basic integration transforming data differently for two destinations, you could name them `basic-1` and `basic-2`. In this case, the `path` option below must be used to specify the download path for the integration. ### registry (`string`, optional, default: `cloudquery`, available: `cloudquery`, `local`, `grpc`, `docker`) - `cloudquery`: Download the integration from the official CloudQuery registry and execute it. - `local`: Execute the integration from a local filesystem path. - `grpc`: Connect to an already-running integration via gRPC (useful for debugging). - `docker`: Run the integration as a Docker container. ### path (`string`, required) Configures how to retrieve the integration. The format depends on the `registry` value. - For `cloudquery` registry: `"cloudquery/"` (e.g. `cloudquery/basic`). - For `local` registry: a filesystem path to the integration binary. - For `grpc` registry: the host and port of the running integration (e.g. `localhost:7777`). ### version (`string`, required) Must be a valid [SemVer](https://semver.org/), e.g. `vMajor.Minor.Patch`. You can find official integration versions on the [GitHub releases page](https://github.com/cloudquery/cloudquery/releases) or on each integration's hub page. ### spec (`object`, optional) Integration-specific configuration. See the documentation for each transformer: - [Basic transformer configuration](https://www.cloudquery.io/hub/plugins/transformer/cloudquery/basic/latest/docs) - [JSON Flattener configuration](https://www.cloudquery.io/hub/plugins/transformer/cloudquery/jsonflattener/latest/docs) ## Common Use Cases ### Renaming Tables Add a prefix to all synced tables to avoid naming conflicts: ```yaml copy transformations: - kind: change_table_names tables: ["*"] new_table_name_template: "cq_{{.OldName}}" ``` ### Obfuscating Sensitive Data Automatically redact all columns marked sensitive by the source, plus specific columns by name: ```yaml copy transformations: - kind: obfuscate_sensitive_columns - kind: obfuscate_columns tables: ["aws_iam_users"] columns: ["password_last_used"] ``` ### Dropping Rows Filter out rows where a column matches a specific value: ```yaml copy transformations: - kind: drop_rows tables: ["aws_ec2_instances"] columns: ["region"] value: "us-west-1" ``` ### Removing Columns Strip columns you don't need in the destination: ```yaml copy transformations: - kind: remove_columns tables: ["aws_s3_buckets"] columns: ["policy"] ``` Transformations are applied sequentially. If you rename tables, subsequent transformations must use the new table names. ## Creating Custom Transformers Need a transformation that doesn't exist? Learn how to [create your own transformer integration](/cli/integrations/creating-new-integration). ## Next Steps - [Source Integrations](/cli/integrations/sources) - Configure data extraction from cloud providers and APIs - [Destination Integrations](/cli/integrations/destinations) - Configure where transformed data is stored - [Configuration Guide](/cli/core-concepts/configuration) - Learn how to combine sources, transformers, and destinations - [Transformations (Policies)](/cli/core-concepts/transformations) - Use dbt and SQL transformations for analytics --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/amazon-ecs # Running CloudQuery on Amazon ECS Deploy CloudQuery on AWS ECS using Fargate. This uses the AWS CLI and AWS CloudFormation to create the required resources. ## Prerequisites Before starting the deployment process, you need to have the following prerequisites: - AWS CLI installed and configured more information can be found [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) - Basic understanding of [AWS ECS and Fargate](https://docs.aws.amazon.com/AmazonECS/latest/userguide/what-is-fargate.html) - IDs of a private subnet and a security group to be used by the deployed ECS cluster. - A name and a region of an S3 bucket that you will sync the data to ## Step 1: Generate a CloudQuery Configuration File Create a new file named `cloudquery.yml` with the following content: ```yaml kind: source spec: # Source spec section name: aws path: "cloudquery/aws" registry: "cloudquery" version: "VERSION_SOURCE_AWS" tables: ["aws_s3_buckets"] destinations: ["s3"] --- kind: destination spec: name: "s3" path: "cloudquery/s3" registry: "cloudquery" version: "VERSION_DESTINATION_S3" write_mode: "append" spec: bucket: region: path: "{{TABLE}}/{{UUID}}.parquet" format: "parquet" athena: true ``` This will create a configuration file that will instruct CloudQuery to collect data from AWS and store it in an S3 bucket. You will need to replace the following placeholders: - `REPLACE_WITH_S3_DESTINATION_BUCKET`: use the name of the S3 bucket you want to use to store the data - `REPLACE_WITH_AWS_REGION`: specify the region where the bucket is You can also modify the configuration file to collect only the data you need. For more information on how to create a configuration file, [visit our docs](/cli/integrations/sources). To inject the configuration file into the prebuilt container, base64 encode the content of the `cloudquery.yml` file. Assuming you are running on a Linux or MacOS machine you can do this conversion by running the following command: ```bash cat cloudquery.yml | base64 ``` ## Step 2: Create a Secret to store a CloudQuery API Key Downloading integrations requires users to be authenticated, normally this means running `cloudquery login` but that is not doable in ECS. The recommended way to handle this is to use an API key. More information on generating an API Key can be found [here](/cli/managing-cloudquery/deployments/generate-api-key). Once you have a CloudQuery API Key you are going to create a Secret in AWS Secrets Manager to store the API Key. To create a secret, use the following command: ```bash aws secretsmanager create-secret \ --name CQ-APIKey \ --description "API Key to authenticate with CloudQuery hub" \ --secret-string "" ``` ## Step 3: Create a CloudFormation template This template will create the required resources for the deployment of CloudQuery on AWS ECS. Create a new file named `cloudquery-ecs-resources.yaml` with the following content: ```yaml AWSTemplateFormatVersion: "2010-09-09" Description: "Deploy CloudQuery on AWS ECS with Fargate" Parameters: CQVersion: Description: Please enter the version of CloudQuery you want to deploy. This should be in the format X, X.Y, X.Y.Z, or `latest` Type: String ScheduleExpression: Description: Please enter the Eventbridge Schedule Expression. This can either be a Rate or a cron expression. This will define how often CloudQuery will run the sync. Type: String Default: "rate(24 hours)" PrivateSubnetIds: Description: Please enter a comma separated list of Subnet Ids where you want to run CloudQuery and the Database. Type: String SecurityGroupIds: Description: Please enter a comma separated list of Security Group IDs that you want to attach to the Fargate node. Type: String DestinationS3Bucket: Description: Please enter the name of the S3 bucket where you want to store the CloudQuery results Type: String CQConfiguration: Description: Please enter the CloudQuery configuration file encoded in base64 Type: String CQApiKey: Description: ARN of the secret containing the CloudQuery API Key Type: String AWSMarketplace: Description: If you are using the AWS Marketplace version of CloudQuery, set this to true Type: String AllowedValues: [true, false] Default: false Resources: #### ECS Cluster: ECSCluster: Type: "AWS::ECS::Cluster" ScheduledWorkerTask: Type: "AWS::ECS::TaskDefinition" Properties: RequiresCompatibilities: - FARGATE NetworkMode: awsvpc Cpu: 1024 Memory: 2GB ExecutionRoleArn: !GetAtt ExecutionRole.Arn TaskRoleArn: !GetAtt ExecutionRole.Arn ContainerDefinitions: - Essential: "true" Command: - "echo $CQ_CONFIG| base64 -d > ./file.yml;/app/cloudquery sync ./file.yml --log-console --log-format json" Image: !Sub ghcr.io/cloudquery/cloudquery:${CQVersion} Name: ScheduledWorker EntryPoint: - "/bin/sh" - "-c" Environment: - Name: CQ_INSTALL_SRC Value: CLOUDFORMATION - Name: CQ_CONFIG Value: !Ref CQConfiguration - Name: CQ_AWS_MARKETPLACE_CONTAINER Value: !Ref AWSMarketplace Secrets: - ValueFrom: !Ref CQApiKey Name: CLOUDQUERY_API_KEY LogConfiguration: LogDriver: awslogs Options: awslogs-group: !Ref LogGroup awslogs-region: !Ref AWS::Region awslogs-stream-prefix: !Ref AWS::StackName LogGroup: DeletionPolicy: Retain UpdateReplacePolicy: Retain Type: AWS::Logs::LogGroup Properties: RetentionInDays: 14 # Scheduler Configurations Schedule: Type: AWS::Scheduler::Schedule Properties: FlexibleTimeWindow: Mode: "OFF" ScheduleExpression: !Ref ScheduleExpression State: ENABLED Target: Arn: !GetAtt ECSCluster.Arn EcsParameters: NetworkConfiguration: AwsvpcConfiguration: AssignPublicIp: DISABLED SecurityGroups: !Split [",", !Ref SecurityGroupIds] Subnets: !Split [",", !Ref PrivateSubnetIds] LaunchType: FARGATE PlatformVersion: 1.4.0 TaskCount: 1 TaskDefinitionArn: !Ref ScheduledWorkerTask RoleArn: !GetAtt SchedulerExecutionRole.Arn SchedulerExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Action: - sts:AssumeRole Effect: Allow Principal: Service: - scheduler.amazonaws.com Path: "/" Policies: - PolicyName: SchedulerExecutionRole PolicyDocument: Version: "2012-10-17" Statement: - Action: - ecs:RunTask Effect: Allow Resource: !Ref ScheduledWorkerTask Condition: ArnEquals: ecs:cluster: !GetAtt ECSCluster.Arn - Effect: Allow Action: - iam:PassRole Resource: !GetAtt ExecutionRole.Arn ####### IAM role for Fargate execution##### ExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Statement: - Effect: Allow Principal: Service: ecs-tasks.amazonaws.com Action: "sts:AssumeRole" Policies: - PolicyName: AccessAPIKeySecret PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - secretsmanager:GetSecretValue Resource: !Sub ${CQApiKey} - PolicyName: WriteDataToS3Destination PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - s3:PutObject Resource: !Sub arn:${AWS::Partition}:s3:::${DestinationS3Bucket}/* - PolicyName: DenyData PolicyDocument: Version: 2012-10-17 Statement: - Effect: Deny NotResource: - !Sub arn:${AWS::Partition}:s3:::${DestinationS3Bucket}/* Action: - s3:GetObject - Effect: Deny Resource: "*" Action: - cloudformation:GetTemplate - dynamodb:GetItem - dynamodb:BatchGetItem - dynamodb:Query - dynamodb:Scan - ec2:GetConsoleOutput - ec2:GetConsoleScreenshot - kinesis:Get* - lambda:GetFunction - logs:GetLogEvents - sdb:Select* - sqs:ReceiveMessage ManagedPolicyArns: - "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy" - "arn:aws:iam::aws:policy/ReadOnlyAccess" - "arn:aws:iam::aws:policy/AWSMarketplaceMeteringFullAccess" Outputs: ClusterId: Value: !Ref ECSCluster TaskArn: Value: !Ref ScheduledWorkerTask ``` ## Step 4: Deploy the CloudFormation Template You can deploy the CloudFormation template using the `aws cloudformation deploy` command. This command will create the required resources for the deployment of CloudQuery on AWS ECS with Fargate. If you are using the AWS Marketplace version of CloudQuery, you can set the `AWSMarketplace` parameter to `true`. ```bash aws cloudformation deploy --template-file cloudquery-ecs-resources.yaml --stack-name --parameter-overrides CQApiKey= CQVersion=latest PrivateSubnetIds=, SecurityGroupIds= DestinationS3Bucket= CQConfiguration= --capabilities CAPABILITY_IAM ``` Replace the following values in the command above: - `STACK_NAME`: the name of the CloudFormation stack that you will deploy. - `SECRET_ARN`: The ARN of the secret created in Step 2 - `SUBNET_ID_1`, `SUBNET_ID_2`: IDs of the private subnets for the ECS cluster. You can specify any number of subnets that you want. If you use more than one, separate the subnet IDs by a comma. - `SecurityGroup_ID_1` - IDs of the security groups for the task. You can specify any number of security groups that you want. If you use more than one, separate the security group IDs by a comma. - `DESTINATION_BUCKET` - The name of the bucket to sync to. - `BASE64_ENCODED_CONFIG` - The base64 encoded configuration file from step 1. ## Step 5: Run a CloudQuery sync To get the values for Cluster Name and Task ARN you can use the following command: ```bash aws cloudformation describe-stacks --stack-name --query "Stacks[].Outputs" ``` Now that the task definition is registered, it's time to run the CloudQuery task on ECS using the `aws ecs run-task` command: ```bash aws ecs run-task \ --cluster \ --task-definition \ --launch-type FARGATE \ --network-configuration 'awsvpcConfiguration={subnets=[,],securityGroups=[,],assignPublicIp=ENABLED}' ``` Replace the following placeholders: - `` with the name of the ECS cluster you created in Step 4 - `` with the ARN of the task definition you registered in Step 5 - `` and `` with the IDs of the subnets in which you want to run the task. You can specify any number of subnets that you want - `` and `` with the IDs of the security groups for the task. You can specify any number of security groups that you want If you deploy in a private subnet you will need to set the `assignPublicIp` to `DISABLED` ## Conclusion You now have a working CloudQuery deployment that runs on a regular basis and stores the results in an S3 bucket. Use this as a starting point for further customization. To learn more about performance you can check out the [performance tuning guide](/cli/advanced/performance-tuning). If you have any questions or comments, Reach out to us on [GitHub](https://github.com/cloudquery/cloudquery) or our [Community](https://community.cloudquery.io)! --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/apache-airflow # Orchestrating CloudQuery Syncs with Apache Airflow and Kubernetes Apache Airflow is a popular open source workflow management tool. It can be used to schedule CloudQuery syncs, optionally retry them and send notifications when syncs fail. In this guide, we will show you how to get started with Airflow and CloudQuery. We will use the [KubernetesOperator](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html), which allows us to run tasks in Kubernetes pods. ## Prerequisites ### Generating a CloudQuery API key Downloading integrations requires users to be authenticated, normally this means running `cloudquery login` but that is not doable in an Apache Airflow environment. The recommended way to handle this is to use an API key. More information on generating an API Key can be found [here](/cli/managing-cloudquery/deployments/generate-api-key). ### Apache Airflow Installation This guide assumes that you have a working Airflow installation and an available Kubernetes cluster, and experience with operating both of these. If you don't, you should consider some simpler orchestration options to get started, such as [GitHub Actions](/cli/managing-cloudquery/deployments/github-actions), [Kestra](/cli/managing-cloudquery/deployments/kestra), or even a simple cron-based deployment. If you decide to proceed with Airflow, you can install it locally on Kubernetes using [Minikube](https://minikube.sigs.k8s.io/) and the [Airflow Helm chart](https://airflow.apache.org/docs/helm-chart/). You will need to have the ability to set up DAGs. When deployed to Kubernetes, this is done with [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) and Persistent Volume Claims. For example, to map a local directory at `/data/airflow/dags` (inside the Minikube container, if not running on bare metal), you can use the following configurations and commands to create the Persistent Volume and Persistent Volume Claim: ```bash copy filename="pv-volume.yaml" apiVersion: v1 kind: PersistentVolume metadata: name: task-pv-volume labels: type: local spec: storageClassName: manual capacity: storage: 1Gi accessModes: - ReadOnlyMany - ReadWriteMany hostPath: path: "/data/airflow/dags" ``` ```bash copy kubectl apply --namespace airflow -f pv-volume.yaml ``` ```yaml copy filename="pv-claim.yaml" apiVersion: v1 kind: PersistentVolumeClaim metadata: name: task-pv-claim spec: storageClassName: manual accessModes: - ReadOnlyMany resources: requests: storage: 100Mi ``` ```bash copy kubectl apply --namespace airflow -f pv-claim.yaml ``` Finally, install the Airflow Helm chart with the persistent volume claim enabled: ```bash copy helm upgrade --install airflow apache-airflow/airflow --namespace airflow --create-namespace --set dags.persistence.enabled=true \ --set dags.persistence.existingClaim=task-pv-claim \ --set dags.gitSync.enabled=false ``` You will then need to expose the Airflow web server to access the Airflow UI at `http://localhost:8080`: ```bash copy kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow ``` ## Setting up the KubernetesOperator The following Python code creates a DAG that runs a CloudQuery sync every day. It uses the KubernetesOperator to run the sync in a Kubernetes pod. ```python copy filename="cloudquery.py" from datetime import datetime, timedelta from airflow import DAG from airflow.operators.python import PythonOperator from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator from kubernetes.client import models as k8s # Change these to match your requirements. Change start_date to today's date, but keep it static default_args = { 'owner': '', # TODO: Set to your name 'depends_on_past': False, 'start_date': datetime(2023, 4, 26), # TODO: Change to today's date 'retries': 0, } with DAG( 'cloudquery_sync', default_args=default_args, schedule_interval=timedelta(days=1), ) as dag: cloudquery_operator = KubernetesPodOperator( task_id='cloudquery_sync', name='cloudquery-sync', namespace='airflow', image='ghcr.io/cloudquery/cloudquery:latest', cmds=['/app/cloudquery', 'sync', '/mnt/config.yaml', '--log-console', '--log-level', 'info'], # We're passing the CloudQuery API key as an environment variable for brevity, but it's better to use a k8s secret # See https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html env_vars={ "CLOUDQUERY_API_KEY": "", }, arguments=[], volume_mounts=[ k8s.V1VolumeMount( name="cloudquery-config", mount_path="/mnt/config.yaml", sub_path="config.yaml", read_only=True ) ], volumes=[ k8s.V1Volume( name="cloudquery-config", config_map=k8s.V1ConfigMapVolumeSource( name="cloudquery-config", items=[k8s.V1KeyToPath(key="config.yaml", path="config.yaml")] ) )], get_logs=True, ) ``` This relies on a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) for the CloudQuery config. To create this, first create a file called `config.yaml` with the contents of your CloudQuery config file. Here we will use an example config file that syncs the `aws_ec2_instance` table from the `aws` integration to a Postgres database, but refer to our [Getting Started guide](/cli/getting-started) and the specific documentation for each integration to see how to configure it for the data you wish to sync. ```yaml copy filename="config.yaml" kind: source spec: # Source spec section name: aws path: cloudquery/aws version: "VERSION_SOURCE_AWS" tables: ["aws_s3_buckets"] destinations: ["postgresql"] spec: # AWS-specific configuration goes here --- kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" version: "VERSION_DESTINATION_POSTGRESQL" write_mode: "overwrite-delete-stale" spec: ## integration-specific configuration for PostgreSQL. ## See all available options here: https://github.com/cloudquery/cloudquery/tree/main/plugins/destination/postgresql#postgresql-spec ## Required. Connection string to your PostgreSQL instance ## In production it is highly recommended to use environment variable expansion ## connection_string: ${PG_CONNECTION_STRING} connection_string: "postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable" ``` Now, create the ConfigMap: ```bash copy kubectl create configmap cloudquery-config --from-file=config.yaml --namespace airflow ``` If everything is working, you should now be able to go to the Airflow UI and see the DAG. You can trigger it manually to test it out. ![CloudQuery job in Airflow UI](/images/docs/deployment/airflow.png) ## Next steps We covered the basics of how to use the KubernetesOperator to run CloudQuery syncs using Airflow. This is a complex terrain and we haven't covered all the possible options here, so please refer to the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/index.html) for more information. For using environment variables and secrets, see the [documentation on the KubernetesPodOperator](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html). The friendly [CloudQuery Community](https://community.cloudquery.io) is also always happy to help! --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/choosing-a-deployment # Choosing a Deployment Method This guide helps you pick the right deployment approach based on your team's needs, scale, and operational maturity. ## Decision matrix | | Local CLI | Docker | Kubernetes | CloudQuery Platform | |---|---|---|---|---| | **Best for** | Development, testing, one-off syncs | Small-to-medium production workloads | Large-scale, multi-source production | Teams wanting managed infrastructure | | **Setup complexity** | Low: install binary and run | Low: single container | Medium: requires K8s cluster and Helm | Low: sign up and connect | | **Scaling** | Single machine | Single host, limited scaling | Horizontal scaling across nodes | Managed by Platform | | **Scheduling** | Manual or cron | cron or Docker-native scheduling | CronJob resource | Built-in UI scheduling | | **Maintenance** | Minimal | Container updates | Cluster ops, Helm chart updates | Managed by CloudQuery | | **Monitoring** | Manual/logs | Container logs | K8s observability stack | Built-in monitoring | | **Data residency** | Full control | Full control | Full control | Depends on deployment (SaaS or self-hosted) | ## When to use each option ### Local CLI Run CloudQuery directly on your machine. Best for: - Getting started and learning CloudQuery - Development and testing new configurations - One-off data extractions or ad hoc queries - CI/CD pipelines (e.g., [GitHub Actions](/cli/managing-cloudquery/deployments/github-actions)) [Getting started guide →](/cli/getting-started) ### Docker Run CloudQuery in a container. Best for: - Small production deployments on a single host - Teams already using Docker for other services - Environments where installing binaries directly isn't preferred [Docker deployment guide →](/cli/managing-cloudquery/deployments/docker) ### Kubernetes Deploy CloudQuery on a Kubernetes cluster using Helm charts. Best for: - Production workloads syncing from many sources - Teams that already operate Kubernetes clusters - Environments needing horizontal scaling and high availability - Organizations with existing K8s observability and alerting Options include: - [Kubernetes CronJob](/cli/managing-cloudquery/deployments/kubernetes-cronjob): schedule syncs as K8s CronJobs - [Amazon ECS](/cli/managing-cloudquery/deployments/amazon-ecs): run on AWS ECS with Fargate or EC2 - [Google Cloud Run](/cli/managing-cloudquery/deployments/google-cloud-run): serverless containers on GCP ### CloudQuery Platform Use the managed service. Best for: - Teams that want fast time-to-value without managing infrastructure - Organizations that prefer a web UI for configuration and monitoring - Smaller teams without dedicated infrastructure resources - Teams that want built-in asset inventory, SQL console, and alerts [Platform quickstart →](/platform/quickstart/creating-a-new-account) ## Orchestration options If you're running the CLI in production, you'll typically pair it with an orchestrator for scheduling: | Orchestrator | Complexity | Best for | |---|---|---| | **cron** | Low | Simple, single-machine setups | | **GitHub Actions** | Low | CI/CD-integrated syncs, teams already on GitHub | | **Apache Airflow** | Medium | Teams with existing Airflow infrastructure | | **Kestra** | Medium | Event-driven workflows | | **Kubernetes CronJob** | Medium | Teams already running K8s | See the individual deployment guides for step-by-step instructions with each orchestrator. ## Next Steps - [Docker Deployment](/cli/managing-cloudquery/deployments/docker) - Deploy with Docker containers - [GitHub Actions](/cli/managing-cloudquery/deployments/github-actions) - Run syncs in CI/CD pipelines - [Kubernetes CronJob](/cli/managing-cloudquery/deployments/kubernetes-cronjob) - Schedule syncs on Kubernetes - [Performance Tuning](/cli/advanced/performance-tuning) - Optimize sync performance in production --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/containers-readonly-filesystem # Containers with Read-Only Root Filesystem By default, CloudQuery requires a read-write file system to create a socket file used by the integrations to communicate with the CLI. It is however not possible to create a read-write socket on a read-only file system. For this to work, an integration needs to be started as a gRPC server locally. ## Creating a container that works with a read-only file system To create a container that works with a read-only file system, you will need the following: - a CloudQuery configuration file that is used to install the required integrations - a CloudQuery configuration file that contains the specs needed for the sync - a start script used as an entry point for the container that will start the integrations and run the sync > Downloading integrations requires users to be authenticated, normally this means running `cloudquery login` but that is not doable in a CI environment or inside of a docker build process. The recommended way to handle this is to use an API key. More information on generating an API Key can be found [here](/cli/managing-cloudquery/deployments/generate-api-key) In the example below, we will use the [xkcd](https://github.com/hermanschaaf/cq-source-xkcd) source integration and [PostgreSQL](https://www.cloudquery.io/hub/plugins/destination/cloudquery/postgresql/v6.1.3/docs) destination integration. ### Create a configuration file for the installation This configuration file should reference the integrations that you want to install with the actual source where they should be downloaded from: ```yaml copy filename="install.yaml" kind: source spec: name: xkcd path: hermanschaaf/xkcd version: v2.0.0 registry: github destinations: ["postgresql"] tables: ["*"] --- kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" version: "VERSION_DESTINATION_POSTGRESQL" ``` ### Create a configuration file for the sync This configuration file should reference the integrations running as gRPC servers (fill in the `connection_string`). We will run xkcd on port 7778 and PostgreSQL on port 7779: ```yaml copy filename="sync.yaml" kind: source spec: name: xkcd registry: grpc path: localhost:7778 destinations: ["postgresql"] tables: ["*"] --- kind: destination spec: name: "postgresql" registry: grpc path: localhost:7779 spec: connection_string: "[REDACTED]" ``` ### Create a start script to start the integrations This shell script will start the integrations and the sync with the provided parameters when we run the container. To make it easier to maintain, we will use the `find` command to get the path to the actual integration binary. This way, we don't have to worry about the version of the integration. ```shell copy filename="start.sh" #!/bin/sh # get path to the xkcd integration: xkcd_plugin_path=$(find .cq -regex ".*/xkcd/.*/plugin") # get path to the postgresql integration: postgresql_plugin_path=$(find .cq -regex ".*/postgresql/.*/plugin") # start the integrations and run the sync with the provided parameters: sh -c "${xkcd_plugin_path} serve --address localhost:7778" & \ sh -c "${postgresql_plugin_path} serve --address localhost:7779" & \ /app/cloudquery $@ ``` ### Connecting it all together We will use the `ghcr.io/cloudquery/cloudquery:latest` container as a base and add all the files in it. Then we will override the entry point with our start script: ```docker FROM ghcr.io/cloudquery/cloudquery:latest as build WORKDIR /app # this is the install.yaml file we created above COPY ./install.yaml /app/install.yaml ARG CLOUDQUERY_API_KEY # install the integrations RUN /app/cloudquery plugin install install.yaml FROM ghcr.io/cloudquery/cloudquery:latest WORKDIR /app # Copy the .cq directory which contains the integrations COPY --from=build /app/.cq /app/.cq # add the sync config file COPY ./sync.yaml /app/sync.yaml # add the start script COPY --chmod=0755 ./start.sh /app/start.sh # override the start script ENTRYPOINT [ "/app/start.sh"] ``` ## Run the Container First, build the container as you would normally do: ```bash docker build --build-arg CLOUDQUERY_API_KEY= ./ -t my-cq-container:latest ``` Run the container with `--read-only` option and a command to pass to our start script. You will also need to add the `--no-log-file` option to skip logging to a file. Here is an example: ```bash copy docker run --read-only --add-host=host.docker.internal:host-gateway my-cq-container:latest --no-log-file sync sync.yaml ``` ## Related Documentation - [Running integrations locally](/cli/advanced/running-locally) - [Deployment - Docker](/cli/managing-cloudquery/deployments/docker) - [Deployment - Docker Offline Installation](/cli/managing-cloudquery/deployments/docker-offline-installation) ## Next Steps - [Docker Deployment](/cli/managing-cloudquery/deployments/docker) - General Docker deployment guide - [Docker Offline Installation](/cli/managing-cloudquery/deployments/docker-offline-installation) - Install images in air-gapped environments - [Security](/cli/managing-cloudquery/security) - CloudQuery security best practices --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/docker-offline-installation # Docker - Offline Installation You can run CloudQuery in a container with integrations pre-installed. This is useful for isolated deployments where you don't want to download integrations from the internet. Downloading integrations requires users to be authenticated, normally this means running `cloudquery login` but that is not doable in a CI environment or in a Docker build process. The recommended way to handle this is to use an API key. More information on generating an API Key can be found [here](/cli/managing-cloudquery/deployments/generate-api-key) To download the integrations based on your configuration file, use the `cloudquery plugin install` command. Below is an example `Dockerfile` based on the [CloudQuery container](/cli/managing-cloudquery/deployments/docker). It uses a `build.spec.yaml` with the minimum configuration required to download the integrations. ```yaml # build.spec.yaml kind: source spec: name: aws path: cloudquery/aws registry: cloudquery version: "VERSION_SOURCE_AWS" tables: ["aws_ec2_instances"] destinations: ["postgresql"] --- kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" registry: "cloudquery" version: "VERSION_DESTINATION_POSTGRESQL" spec: ``` ```docker # Dockerfile FROM ghcr.io/cloudquery/cloudquery:latest AS build WORKDIR /app COPY ./build.spec.yaml /app/build.spec.yaml ARG CLOUDQUERY_API_KEY RUN /app/cloudquery plugin install build.spec.yaml FROM ghcr.io/cloudquery/cloudquery:latest WORKDIR /app # Copy the .cq directory which contains the integrations COPY --from=build /app/.cq /app/.cq ``` Build this container as you would normally do: ```bash docker build --build-arg CLOUDQUERY_API_KEY= ./ -t my-cq-container:latest ``` ### Run the Container Run the container as you would run the default CloudQuery container. Here is an example: ```bash copy docker run \ # you can mount a different config file that uses the same integrations as in the build.spec -v :/config.yml \ # set any env variable with -e = my-cq-container:latest \ sync /config.yml ``` ### Enforcing locally cached integrations When using `registry: cloudquery`, CloudQuery CLI tries to download integrations from CloudQuery Hub, or use cached integrations if available. To enforce using locally cached plugins, you can set the `registry` to `local` and specify local plugin paths. For example: ```yaml # local.spec.yaml kind: source spec: name: aws path: .cq/plugins/source/cloudquery/aws/VERSION_SOURCE_AWS/plugin registry: local version: "VERSION_SOURCE_AWS" tables: ["aws_ec2_instances"] destinations: ["postgresql"] --- kind: destination spec: name: "postgresql" path: .cq/plugins/destination/cloudquery/postgresql/VERSION_DESTINATION_POSTGRESQL/plugin registry: local version: "VERSION_DESTINATION_POSTGRESQL" ``` To run the sync, you can use the following command: ```bash docker run \ -v :/local.spec.yaml \ my-cq-container:latest \ sync /local.spec.yaml ``` > You can configure the default plugins download path via the `--cq-dir` [CLI argument](/cli/cli-reference/cloudquery_plugin_install#options-inherited-from-parent-commands). ## Troubleshooting If you encounter the following error when running the `cloudquery plugin install` command: ``` tls: failed to verify certificate: x509: certificate signed by unknown authority ``` You probably need to install certificates in your container image. To identify which certificates are needed, you can run the following command: ```bash openssl s_client -showcerts -connect api.cloudquery.io:443 ``` To extract the certificates to files, you can use the following command. This will create a file for each certificate in the current directory. ```bash openssl s_client -showcerts -connect api.cloudquery.io:443 fname}' ``` Then update your `Dockerfile` to copy over the certificates to your container and install them. The full Dockerfile should look like this: ```docker # Dockerfile FROM ghcr.io/cloudquery/cloudquery:latest AS build RUN apk add --no-cache ca-certificates WORKDIR /app COPY ./build.spec.yaml /app/build.spec.yaml COPY ./cert*.pem /usr/local/share/ca-certificates/ ARG CLOUDQUERY_API_KEY ENV SSL_CERT_DIR=/usr/local/share/ca-certificates/ RUN /app/cloudquery plugin install build.spec.yaml FROM ghcr.io/cloudquery/cloudquery:latest WORKDIR /app # Copy the .cq directory which contains the integrations COPY --from=build /app/.cq /app/.cq ``` ### Related docs Read more about the `plugin install` command in the [CLI Documentation](/cli/cli-reference/cloudquery_plugin_install). ## Next Steps - [Docker Deployment](/cli/managing-cloudquery/deployments/docker) - General Docker deployment guide - [Containers with Read-Only Filesystem](/cli/managing-cloudquery/deployments/containers-readonly-filesystem) - Run in restricted environments - [Using an Offline License](/cli/advanced/using-an-offline-license) - Use paid integrations in air-gapped environments --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/docker # Docker It is possible to use CloudQuery in an isolated container. You can pull the relevant image via `docker pull ghcr.io/cloudquery/cloudquery:latest`. ## Configuration ### Create a Config File on the Host Machine CloudQuery uses YAML files as the primary means of configuration. A simple example `config.yml` file would look like: ```yaml copy kind: source spec: name: aws path: cloudquery/aws registry: cloudquery version: "VERSION_SOURCE_AWS" tables: ["aws_ec2_*"] destinations: ["postgresql"] --- kind: destination spec: name: postgresql path: cloudquery/postgresql registry: cloudquery version: "VERSION_DESTINATION_POSTGRESQL" spec: connection_string: "postgres://postgres:pass@host.docker.internal:5432/postgres?sslmode=disable" ``` ### Run the Container Downloading integrations requires users to be authenticated, normally this means running `cloudquery login` but that is not doable in a CI environment or in a Docker build process. The recommended way to handle this is to use an API key. More information on generating an API Key can be found [here](/cli/managing-cloudquery/deployments/generate-api-key) For the CloudQuery docker container to use this configuration file you will need to mount the volume to the container like so: ```bash copy docker run --pull=always \ -v :/config.yml \ # set any env variable with -e = -e CLOUDQUERY_API_KEY= ghcr.io/cloudquery/cloudquery:latest \ sync /config.yml ``` As with running any `cloudquery` command on your CLI you can override the configuration with the [optional flags](/cli/cli-reference/cloudquery) with the docker container. You will also need to make sure you load any ENV variables for source and destination integrations, such as your `AWS_*` keys etc. If you split the configuration between multiple files, you can mount the directory containing them, instead of just the `config.yml` file. If you are running Docker on an ARM Apple device and you see a segmentation fault when running the container like so `qemu: uncaught target signal 11 (Segmentation fault) - core dumped`; make sure you are running the latest Docker for Mac release. ## Caching Due to the way `cloudquery` is [designed](/cli/core-concepts/architecture) it downloads all the components to interact with source and destination integrations. This means that with a docker container it runs the download step each time, as the local cache is lost between executions. To avoid this we recommend mounting a volume to cache the data and configuring `cloudquery` to use this via the `--cq-dir` optional flag. An example of this would be: ```bash copy docker run --pull=always \ -v :/config.yml \ -v :/cache/.cq \ # set any env variable with -e = ghcr.io/cloudquery/cloudquery:latest \ sync /config.yml --cq-dir /cache/.cq ``` Depending on your operating system, the built components maybe different between your local system and the container. To avoid compatibility issues, use a separate cache directory for the container than a local instance of `cloudquery`. ## Next Steps - [Docker Offline Installation](/cli/managing-cloudquery/deployments/docker-offline-installation) - Install Docker images without internet access - [Containers with Read-Only Filesystem](/cli/managing-cloudquery/deployments/containers-readonly-filesystem) - Run in restricted container environments - [Monitoring](/cli/managing-cloudquery/monitoring/overview) - Set up observability for containerized syncs - [Performance Tuning](/cli/advanced/performance-tuning) - Optimize sync performance --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/generate-api-key # Generate a CloudQuery API Key In automated environments, an API Key can be used with CloudQuery Hub and premium integrations instead of `cloudquery login`. Create an API key by following these steps: 1. Go to [CloudQuery Platform](https://cloud.cloudquery.io) and log in or register for a new account. 2. If you are not already in a team, you will be prompted to create a new team. 3. Once you are logged in, click `Team Settings` in the left sidebar. 4. Go to the `API Keys` tab. ![API Keys](/images/docs/deployment/generate-api-key/01-api-keys.png) 5. Click `Generate new key` ![Generate new key](/images/docs/deployment/generate-api-key/02-generate-new-key.png) 6. Choose a key name for you to identify this key by, select the scope of the key and an expiration date. Then click `Generate new key`. ![Generate key](/images/docs/deployment/generate-api-key/03-generate-key.png) If you intend to use the key to only run syncs we recommend selecting `Syncs operations only` as the scope. If you intend to use the key to interact with the CloudQuery REST API, select `Read and write` as the scope. 7. Copy the key and store it in a safe place. You will not be able to see it again after this step. ![Copy key](/images/docs/deployment/generate-api-key/04-save-key.png) To use the API key with the CloudQuery CLI, set the `CLOUDQUERY_API_KEY` environment variable to the value of the key. ```bash export CLOUDQUERY_API_KEY= ``` ## Next Steps - [Deployment Options](/cli/managing-cloudquery/deployments/overview) - Choose a deployment strategy for your syncs - [Docker Deployment](/cli/managing-cloudquery/deployments/docker) - Use your API key in Docker containers - [GitHub Actions](/cli/managing-cloudquery/deployments/github-actions) - Use your API key in CI/CD pipelines --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/github-actions # Deploy-less Orchestration with GitHub Actions Load AWS resources into a PostgreSQL database by running CloudQuery as a [GitHub Action](https://github.com/features/actions). ## Prerequisites ### Generating a CloudQuery API key Downloading integrations requires users to be authenticated, normally this means running `cloudquery login` but that is not doable in a CI environment like GitHub Actions. The recommended way to handle this is to use an API key. More information on generating an API Key can be found [here](/cli/managing-cloudquery/deployments/generate-api-key). ### AWS Authentication Since we'll be running CloudQuery in the context of a GitHub Action runner, we'll need to add AWS authentication. To set up authentication with AWS from GitHub Actions you can follow the [Configuring OpenID Connect in Amazon Web Services blog](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) from GitHub. > The role that you create will be used by CloudQuery to download your cloud configuration, so you should also grant it permissions to read from your cloud (e.g. ReadOnlyAccess permission policy) ## Creating the CloudQuery configuration file Under the root of your repository, create a new `cloudquery.yml` file with the following content: ```yaml copy kind: source spec: name: 'aws' path: cloudquery/aws registry: cloudquery version: "VERSION_SOURCE_AWS" tables: ['*'] destinations: ['postgresql'] --- kind: destination spec: name: 'postgresql' path: cloudquery/postgresql registry: cloudquery version: "VERSION_DESTINATION_POSTGRESQL" spec: connection_string: ${CQ_DSN} # The CQ_DSN environment variable will be set by GitHub Action workflow ``` > For more configuration options, [visit our docs](/cli/integrations/sources) ## Creating the GitHub Action workflow First we'll need [to create a GitHub secret](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) with the name `CQ_DSN` and the value of the connection string to your PostgreSQL database. Create a workflow file under `.github/workflows/cloudquery.yml` with the following content, and fill in `` and `` according to the role you created in the prerequisites. ```yaml copy name: CloudQuery on: schedule: - cron: '0 3 * * *' # Run daily at 03:00 (3am) jobs: cloudquery: permissions: id-token: write contents: read runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 # Checkout the code so we have access to the config file - name: Configure AWS credentials # Setup AWS credentials (example) uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: # based on the role you created in the prerequisites aws-region: # based on the region you created the role in - uses: cloudquery/setup-cloudquery@v3 name: Setup CloudQuery with: version: "vVERSION_CLI" - name: Sync with CloudQuery run: cloudquery sync cloudquery.yml --log-console env: CLOUDQUERY_API_KEY: ${{ secrets.CLOUDQUERY_API_KEY }} # See https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/generate-api-key CQ_DSN: ${{ secrets.CQ_DSN }} # Connection string to a PostgreSQL database ``` Once committed to the default branch of the repository, the above workflow will run daily at 3 a.m. and will sync the AWS source integration with the PostgreSQL destination integration. > **Warning** > GitHub automatically disables workflows on public repositories [if no repository activity has occurred for 60 days](https://docs.github.com/en/actions/using-workflows/disabling-and-enabling-a-workflow). This may impact your sync if the repository does not receive regular commits. ## Running CloudQuery in parallel to speed up sync time By default, CloudQuery extracts all supported resources, which can take a bit of time, depending on the number of resources you have in your AWS account. With the [GitHub Actions matrix configuration](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs), you can split the sync process into multiple jobs and run them in parallel. To do so, create the following workflow file under `.github/workflows/cloudquery-parallel.yml`: ```yaml copy name: CloudQuery Parallel on: schedule: - cron: '0 3 * * *' # Run daily at 03:00 (3am) jobs: cloudquery: permissions: id-token: write contents: read runs-on: ubuntu-latest strategy: matrix: shard: [1/4, 2/4, 3/4, 4/4] # Split the sync into 4 parts steps: - uses: actions/checkout@v4 # Checkout the code so we have access to the config file - name: Configure AWS credentials # Setup AWS credentials (example) uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: # based on the role you created in the prerequisites aws-region: # based on the region you created the role in - uses: cloudquery/setup-cloudquery@v3 name: Setup CloudQuery with: version: "vVERSION_CLI" - name: Sync with CloudQuery run: cloudquery sync cloudquery.yml --log-console --shard ${{ matrix.shard }} env: CLOUDQUERY_API_KEY: ${{ secrets.CLOUDQUERY_API_KEY }} # See https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/generate-api-key CQ_DSN: ${{ secrets.CQ_DSN }} # Connection string to a PostgreSQL database ``` Once committed to the default branch of the repository, the above workflow will run daily at 3 a.m and will sync the AWS source integration with the PostgreSQL destination integration, in parallel. ## Next Steps - [Generate API Key](/cli/managing-cloudquery/deployments/generate-api-key) - Create an API key for CI/CD authentication - [Running Syncs in Parallel](/cli/managing-cloudquery/running-in-parallel) - Split syncs across parallel jobs - [Monitoring](/cli/managing-cloudquery/monitoring/overview) - Track sync results from CI/CD --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/google-cloud-run # Scheduling CloudQuery Syncs with Google Cloud Run and Cloud Scheduler [Google Cloud Run](https://cloud.google.com/run) is a managed compute platform that runs stateless containers invoked via web requests or Pub/Sub events. It provides a serverless way to run CloudQuery syncs on Google Cloud without managing infrastructure. [Cloud Run quotas](https://cloud.google.com/run/quotas) impose a maximum timeout of 60 minutes per request (configurable up to 60 minutes for HTTP-triggered services). If your sync takes longer than this, consider deploying to a [Virtual Machine](/cli/managing-cloudquery/deployments/google-cloud-vm) or using [Cloud Composer](https://cloud.google.com/composer/docs/concepts/overview) instead. ## Prerequisites - A Google Cloud project with billing enabled - [Google Cloud CLI (`gcloud`)](https://cloud.google.com/sdk/docs/install) installed and configured - A CloudQuery API key ([generate one here](/cli/managing-cloudquery/deployments/generate-api-key)) - A destination database accessible from Cloud Run (e.g., Cloud SQL, BigQuery) ## How it works Cloud Run containers must accept incoming HTTP connections on a configurable port (8080 by default). To run CloudQuery (a CLI tool, not a web server) you need a wrapper that: 1. Listens for incoming HTTP requests 2. Runs `cloudquery sync` when a request is received 3. Returns the result Cloud Scheduler triggers this endpoint on a cron schedule to run syncs automatically. ## Step 1: Create the configuration file Create a `config.yml` with your source and destination configuration: ```yaml copy kind: source spec: name: gcp path: cloudquery/gcp registry: cloudquery version: "VERSION_SOURCE_GCP" tables: ["gcp_compute_*"] destinations: ["postgresql"] --- kind: destination spec: name: postgresql path: cloudquery/postgresql registry: cloudquery version: "VERSION_DESTINATION_POSTGRESQL" spec: connection_string: "${PG_CONNECTION_STRING}" ``` ## Step 2: Create the Dockerfile Create a `Dockerfile` that wraps CloudQuery with a web server. The [cloudquery/cloudrun-example](https://github.com/cloudquery/cloudrun-example) repository provides a complete example. The key pattern is: ```dockerfile copy FROM ghcr.io/cloudquery/cloudquery:latest COPY config.yml /config.yml COPY server.sh /server.sh RUN chmod +x /server.sh ENTRYPOINT ["/server.sh"] ``` The `server.sh` script should start a lightweight HTTP server that triggers `cloudquery sync /config.yml` on incoming requests and returns the exit code as the HTTP response. See the [cloudquery/cloudrun-example](https://github.com/cloudquery/cloudrun-example) repository for a complete, working implementation of the wrapper server. ## Step 3: Build and push the container Create an Artifact Registry repository if you don't have one: ```bash copy gcloud artifacts repositories create \ --repository-format docker \ --location ``` Build and push the container image: ```bash copy # Configure Docker authentication for Artifact Registry gcloud auth configure-docker -docker.pkg.dev # Build the container docker build -t -docker.pkg.dev///cloudquery-sync . # Push to Artifact Registry docker push -docker.pkg.dev///cloudquery-sync ``` ## Step 4: Deploy to Cloud Run ```bash copy gcloud run deploy cloudquery-sync \ --image -docker.pkg.dev///cloudquery-sync \ --region \ --no-allow-unauthenticated \ --timeout 3600 \ --memory 2Gi \ --set-env-vars "CLOUDQUERY_API_KEY=,PG_CONNECTION_STRING=" ``` Key flags: - `--no-allow-unauthenticated`: Restricts access to authenticated callers only (Cloud Scheduler will use a service account) - `--timeout 3600`: Sets the maximum request timeout to 60 minutes - `--memory 2Gi`: Allocate enough memory for the sync process. Adjust based on the number of tables being synced. ## Step 5: Schedule with Cloud Scheduler Create a Cloud Scheduler job that triggers the Cloud Run service on a cron schedule: ```bash copy gcloud scheduler jobs create http cloudquery-daily-sync \ --location \ --schedule "0 3 * * *" \ --uri "$(gcloud run services describe cloudquery-sync --region --format 'value(status.url)')" \ --http-method GET \ --oidc-service-account-email @.iam.gserviceaccount.com \ --oidc-token-audience "$(gcloud run services describe cloudquery-sync --region --format 'value(status.url)')" ``` This schedules a sync every day at 3 a.m.. The `--oidc-service-account-email` flag ensures the request is authenticated using the specified service account. ## Authentication For GCP source integrations, Cloud Run services automatically receive a service account identity. Grant the service account the required read-only permissions for the resources you want to sync: ```bash copy # Example: grant Viewer role for GCP source integration gcloud projects add-iam-policy-binding \ --member "serviceAccount:@.iam.gserviceaccount.com" \ --role "roles/viewer" ``` For non-GCP sources (AWS, Azure, etc.), pass the required credentials as environment variables via `--set-env-vars` or use [Secret Manager](https://cloud.google.com/run/docs/configuring/secrets). ## Caveats - **Timeout limit**: Cloud Run HTTP-triggered services have a maximum timeout of 60 minutes. Syncs that take longer will be terminated. Use [table filtering](/cli/advanced/performance-tuning#use-wildcard-matching) to reduce sync time, or switch to a [VM-based deployment](/cli/managing-cloudquery/deployments/google-cloud-vm) for large estates. - **Stateless execution**: Each invocation starts fresh. Integration binaries are downloaded on every run unless you bake them into the Docker image. See the [Docker caching guide](/cli/managing-cloudquery/deployments/docker#caching) for strategies to reduce startup time. - **Cold starts**: Cloud Run may scale to zero between invocations, adding startup latency. For scheduled syncs this is usually acceptable. ## Next Steps - [Google Cloud VM Deployment](/cli/managing-cloudquery/deployments/google-cloud-vm) - Alternative GCP deployment with persistent VMs - [Performance Tuning](/cli/advanced/performance-tuning) - Optimize sync speed within Cloud Run timeout limits - [Monitoring](/cli/managing-cloudquery/monitoring/overview) - Set up observability for Cloud Run syncs --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/google-cloud-vm # Deploying CloudQuery to a Google Cloud Virtual Machine (VM) Install CloudQuery on a Google Cloud Virtual Machine (VM) and schedule regular syncs with cron. Running CloudQuery on a VM like this is one of the simplest ways to get started with CloudQuery on the Google Cloud Platform ecosystem. ## Prerequisites - A CloudQuery API key. More information on generating an API Key can be found [here](/cli/managing-cloudquery/deployments/generate-api-key). - A Google Cloud account. You can [sign up here](https://cloud.google.com/). - A Google Cloud Project with billing enabled - The following APIs enabled on the project, at a minimum: - Compute Engine API - Cloud Logging API - Cloud SQL Admin API - Cloud Resource Manager API You can enable these APIs by going to the [APIs & Services](https://console.cloud.google.com/apis/dashboard) section of the Google Cloud Console and clicking on **Enable APIs and Services**. ## Step 1: Create a Google Cloud Virtual Machine 1. Go to the [Google Cloud Console](https://console.cloud.google.com/) and select your project. 2. Click on the hamburger menu in the top left corner and select **Compute Engine**. 3. Click on **Create Instance**. 4. Give your instance a name and select a region and zone. 5. Select a machine type. We recommend using a machine with at least 2 vCPUs and 8GB of memory. The required size will vary depending on the size of your dataset you are syncing and how you configure the concurrency of the sync. 6. Under **Identity and API access**, select **Allow full access to all Cloud APIs**. This will allow the GCP source integration to access the APIs it needs to list all GCP projects in your account. Note that this permission is **distinct from** the permissions we will later grant to the service account for this VM. You can also restrict this to only the APIs you need, but this is not covered in this tutorial. 7. Create the instance ## Step 2: Install CloudQuery 1. SSH into your instance. You can do this by clicking on the SSH button next to your instance in the Google Cloud Console. 2. Download CloudQuery by running the following command from the [Getting Started guide for Linux](/cli/getting-started/quickstart/linux): ```bash copy curl -L https://github.com/cloudquery/cloudquery/releases/download/cli-vVERSION_CLI/cloudquery_linux_amd64 -o cloudquery chmod a+x cloudquery ``` ## Step 3: Create a Cloud SQL Instance 1. Click on the hamburger menu in the top left corner and select **SQL**. 2. Click on **Create Instance**. 3. Select **PostgreSQL**. (Or another database if you prefer--CloudQuery supports many destinations. You will need to modify your destination configuration accordingly later.) 4. Give your instance a name and select a region and zone. 5. Select a machine type. We recommend using a machine with at least 2 vCPUs and 8GB of memory. The required size will vary depending on the size of your dataset you are syncing and how you configure the concurrency of the sync. 6. Under **Networking**, select **Private IP**. If you later want to use Cloud SQL console or access the database from a service outside of your GCP VPC network (like Grafana or Superset), then you can also add a public IP. For a sync to complete, we only need a Private IP. 7. Create the instance ## Step 4: Add a database and user 1. Click on your newly created Cloud SQL instance from the Cloud SQL page. 2. Click on **Databases**. 3. Click on **Create Database**. 4. Give your database a name, like `cloudquery`. 5. Click on **Users**. 6. Click on **Create User Account**. 7. Give your user a name and password. The user name can again be `cloudquery`, with a password of your choice. 8. Click on **Create**. It is also possible to use Cloud IAM for authentication, instead of a password, but this is not covered in this tutorial. In short, you will need to run Cloud SQL Proxy on your VM and configure CloudQuery to connect to the proxy. ## Step 5: Configure CloudQuery 1. Back in the SSH session connected to the VM instance, create a new file called `config.yaml` in the same directory as the CloudQuery binary. 2. Add the following contents to the file, replacing the values with your own: ```yaml copy filename="config.yaml" kind: source spec: # Source spec section name: "gcp" path: "cloudquery/gcp" registry: "cloudquery" version: "VERSION_SOURCE_GCP" tables: ["gcp_storage_buckets"] # Add more tables here if you want to sync more data destinations: ["postgresql"] spec: # GCP Spec section # project_ids: ["my-project"] --- kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" registry: "cloudquery" version: "VERSION_DESTINATION_POSTGRESQL" spec: connection_string: "postgresql://cloudquery:@:5432/cloudquery" ``` If you are using a database other than PostgreSQL, you will need to modify the `destination` section of the configuration file accordingly. See the [destinations](https://www.cloudquery.io/hub/plugins/destination) section of the docs for more information. Similarly, if you are using a different source, follow the [documentation for your source integration](https://www.cloudquery.io/hub/plugins/source) to get the correct configuration. ## Step 6: Add IAM roles 1. Click on the hamburger menu in the top left corner and select **IAM & Admin**. 2. Click on **IAM**. 3. Find the service principal that is running your VM instance. You can find this information on the VM details page: it will typically be in the format `123456789012-compute@developer.gserviceaccount.com` 4. Click **Edit principal** (the pencil icon) next to the service account. 5. Add the following roles to the service account that is running your VM instance. - **Viewer**: This will allow the GCP integration to read GCP resources from your projects - **Browser**: This is necessary to allow the GCP integration to list all projects in your account These permissions assume you are using the GCP source integration, it may be too broad for your use case or for other integrations. Following the principle of least privilege, you should make the scope as small as possible for your needs. ## Step 7: Run a Sync 1. Back in the SSH session connected to the VM instance, run the following command to start the sync: ```bash copy CLOUDQUERY_API_KEY= ./cloudquery sync config.yaml ``` 2. You should see a successful sync! If not, check the CLI output and `cloudquery.log` for errors. 3. You can now fine-tune your configuration file: you may want to try `tables: ["*"]` to sync all tables at least once and get a complete asset inventory. Be aware, however, that this can take a long time on large account. We also don't recommend using `*` in production, as when you perform version upgrades, this will cause new tables to automatically get synced and may cause unexpected issues. In general, we recommend only syncing the tables you need. For tips on performance-tuning, see the [performance tuning](/cli/advanced/performance-tuning) guide. ## Step 8: Query the data We can now query the data in our database. We will use the Cloud SQL interface on the Google Cloud console, but you can also use any other tool that supports PostgreSQL. 1. Click on the hamburger menu in the top left corner and select **SQL**. 2. Click on your database. 3. Click on **Open Cloud Shell**. 4. Edit the command if necessary, then press enter to execute it in the cloud shell. 5. Run any SQL query you wish. For example, the following query lists all the storage buckets in your GCP account: ```sql copy SELECT * FROM gcp_storage_buckets; ``` ## Step 9: Set up a cron schedule ### Option 1: Basic cron schedule Now that we have a working sync, we can set up a cron schedule to run it automatically. 1. SSH into your instance. You can do this by clicking on the SSH button next to your instance in the Google Cloud Console. 2. Run the following command to open the crontab editor: ```bash copy crontab -e ``` 3. Add the following line to the crontab file, replacing the path with the path to your CloudQuery binary and configuration file: ```bash copy 0 1 * * * /path/to/cloudquery sync /path/to/config.yaml ``` 4. Save the file and exit the editor. The sync will now run every day at 1 a.m. ### Option 2: Run on reboot Alternatively, we can use the `@reboot` cron directive to run the sync every time the VM instance is rebooted. This is useful if you want to keep the VM instance stopped most of the day, but have it come up and sync once a day. To do this, update the line in the crontab file: ```bash copy @reboot /path/to/cloudquery sync /path/to/config.yaml ``` We can now use the **Instance Schedule** feature on GCP to have the VM come up at a certain time every day: 1. Click on the hamburger menu in the top left corner and select **Compute Engine**. 2. Click on **VM Instances**. 3. Click on the **Instance Schedules** tab. 4. Click on the **Create Instance Schedule** button. 5. Configure the start time and end time so that the VM instance is up for long enough to run the sync to completion. 6. Click on **Submit**. 7. Click on the newly created schedule. 8. Click on **Add Instances to Schedule**. 9. Select your VM instance and click on **Add**. For this to work, you may first need to add the `Compute Instance Admin (v1)` role to the compute-system managed role (do this from the IAM page). ## Summary You now have CloudQuery running on a GCP VM with scheduled syncs. We have also seen how to set up a cron schedule to run the sync automatically. If you have any questions, check out the video above or join our [Community](https://community.cloudquery.io) to chat with us! ## Next Steps - [Google Cloud Run](/cli/managing-cloudquery/deployments/google-cloud-run) - Alternative serverless deployment on GCP - [Monitoring](/cli/managing-cloudquery/monitoring/overview) - Set up observability for your syncs - [Performance Tuning](/cli/advanced/performance-tuning) - Optimize sync speed and resource usage --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/kestra # Orchestrating CloudQuery Syncs with Kestra This tutorial shows how to run CloudQuery as a [Kestra](https://kestra.io/) flow, using the HackerNews API as a source and CSV file as a destination. Kestra is an open-source event-driven orchestrator, which has a [dedicated CloudQuery plugin](https://kestra.io/plugins/plugin-cloudquery). You can use Kestra to: - schedule [CloudQuery syncs](https://kestra.io/plugins/tasks/io.kestra.plugin.cloudquery.Sync) in a simple, declarative way using a built-in code editor - run your syncs based on events from other systems e.g. a new file in your S3 bucket, new message in a queue, or a new webhook request - monitor your syncs from a friendly UI and get notified when they fail. ## Prerequisites - [Docker installed](https://docs.docker.com/get-docker/) - [A CloudQuery API key](/cli/managing-cloudquery/deployments/generate-api-key) ## Step 1: Install Kestra Kestra needs CloudQuery API key stored as a secret. Unless you use the Enterprise edition, the API key needs to be Base64 encoded and passed as an environment variable. To encode the CloudQuery API key stored in the `CLOUDQUERY_API_KEY` environment variable, run ```shell export SECRET_CLOUDQUERY_API_KEY=$(echo $CLOUDQUERY_API_KEY | base64) ``` Then start Kestra container and pass in the new environment variable: ```bash docker run --pull=always --rm -it -p 8080:8080 --user=root -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp kestra/kestra:latest server local ``` To learn more about Kestra startup configuration and options, see the [Kestra Getting started guide](https://kestra.io/docs/getting-started). Once the container is running, open http://localhost:8080 in your browser. ## Step 2: Create a Kestra flow Inside the Kestra UI, go to the `Flows` page and click on `Create`. You can now paste the following content (make sure to update the connection string and set the CloudQuery API key): ```yaml tasks: - id: wdir type: io.kestra.plugin.core.flow.WorkingDirectory tasks: - id: config_files type: io.kestra.plugin.core.storage.LocalFiles inputs: config.yml: | kind: source spec: name: hackernews path: cloudquery/hackernews registry: cloudquery version: v3.7.23 tables: ["*"] destinations: - file spec: item_concurrency: 10 start_time: "{{ now() | dateAdd(-2, 'HOURS') | date("yyyy-MM-dd'T'HH:mm:ssX", timeZone="UTC") }}" --- kind: destination spec: name: "file" path: "cloudquery/file" registry: "cloudquery" version: "v5.4.22" write_mode: "append" spec: path: "/output/{{'{{TABLE}}/{{UUID}}.{{FORMAT}}'}}" format: "csv" # options: parquet, json, csv - id: run_sync type: io.kestra.plugin.cloudquery.CloudQueryCLI env: # Visit https://kestra.io/docs/concepts/secret to learn how to manage secrets in Kestra CLOUDQUERY_API_KEY: "{{ secret('CLOUDQUERY_API_KEY') }}" commands: - cloudquery sync config.yml --log-console --log-level=debug outputFiles: - "**/*.csv" ``` The flow will extract HackerNews Items published in the last two hours and load them into a CSV file. You may notice that the input to the `config_files` task is virtually identical to your CloudQuery configuration file. Kestra flows are built using declarative YAML syntax, similar to CloudQuery configuration files. Once you configured the flow in the editor, click on `Save`. ![Kestra Flow Editor Screenshot](/images/docs/deployment/kestra-flow.png) ## Step 3: Run the flow To manually start a Kestra flow, click on the `Execute` button in the top right corner. Then, confirm by clicking on `Execute`. You should now see the sync running in the `Executions` tab. You can navigate to the Logs tab to check the logs and validate everything runs as expected. Also, you can download the extracted CSV files from the `Outputs` tab: ![Kestra Flow Execution Screenshot](/images/docs/deployment/kestra-execution.png) ## Step 4: Schedule the flow To run the flow periodically, we can add a trigger to run it on a schedule. Back in the Flow editor, add the following section: ```yaml copy triggers: - id: schedule type: io.kestra.core.models.triggers.types.Schedule cron: "0 9 * * *" # every day at 9am timezone: US/Eastern ``` This cron expression will run the flow every day at 09:00. You can use [crontab.guru](https://crontab.guru/) to generate cron expressions and replace the one in the example above. Kestra also supports these special values for `cron`: ```text @yearly @annually @monthly @weekly @daily @midnight @hourly ``` With this in place, remember to click `Save` again. Your CloudQuery sync will now run on a regular schedule. ## Next steps ### Caching To speed up your syncs and save bandwidth, you can cache the downloaded integrations locally. Kestra does not provide cache for container-based tasks, but you can use a shared volume to cache the plugins. To do this, you need to configure Kestra to [enable volume mounts](https://kestra.io/docs/scripts/bind-mount) first. Then configure your CloudQuery task to use the shared volume: ```yaml - id: run_sync type: io.kestra.plugin.cloudquery.CloudQueryCLI env: CLOUDQUERY_API_KEY: "{{ secret('CLOUDQUERY_API_KEY') }}" # Mount the shared volume to the CloudQuery container docker: image: ghcr.io/cloudquery/cloudquery:latest volumes: - "/tmp/cq:/app/.cq" commands: # Use the shared volume for downloading plugins - cloudquery sync config.yml --log-console --log-level=debug --cq-dir=/app/.cq ``` The shared volume mounts to the `/tmp/cq` directory in the example above on the host machine. You can adjust the path to your needs. ### Production deployment For production deployments, deploy Kestra to a cloud environment such as Kubernetes. For more information, see the [Kestra Deployment with Kubernetes guide](https://kestra.io/docs/installation/kubernetes) and if you have any questions, you can reach the Kestra team [via Community Slack](https://kestra.io/slack). --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/kubernetes-cronjob # Deploying CloudQuery using Kubernetes CronJobs Set up a Kubernetes CronJob to run CloudQuery syncs on a regular schedule. Due to its flexibility and standardization across cloud providers, Kubernetes is commonly used by DevOps & Platform Engineers when deploying workloads and microservices. ## Prerequisites - A CloudQuery API key. More information on generating an API Key can be found [here](/cli/managing-cloudquery/deployments/generate-api-key) - A Kubernetes Cluster - A PostgreSQL database server - Credentials (with Read privileges) for a supported API - in the example we'll use a DigitalOcean account ## Step 1: Create a Kubernetes Secret for your API Keys To keep our credentials separate from our manifests, we need to create a secret that we'll use to supply our environment variables We can do this in a single command: ```bash copy kubectl create secret generic cloudquery-secret \ --from-literal=CLOUDQUERY_API_KEY= \ --from-literal=DIGITALOCEAN_TOKEN= \ --from-literal=SPACES_ACCESS_KEY_ID= \ --from-literal=SPACES_SECRET_ACCESS_KEY= \ --from-literal=PG_CONNECTION_STR= ``` ## Step 2: Create the CronJob Manifest A CronJob is a Kubernetes object that allows you to run a job on a schedule. It's similar to a Kubernetes Workload, except that it is intended for Jobs (short-lived containers) that conduct a task and then shutdown, instead of Pods (long-lived containers intended for services). The CronJob manifest has two key elements, the `jobTemplate`, and the `schedule`. ```yaml apiVersion: batch/v1 kind: CronJob metadata: name: cloudquery labels: app: cloudquery spec: schedule: "0 0 * * *" jobTemplate: {} ``` The Schedule expression follows the same format as crontab; that is a space separated list as follows `minute hour day(of month) month day(of week)`. Which in the above example would be midnight (i.e. 00:00) every day. To learn more about cron schedule expressions, check out https://crontab.guru/ or the Kubernetes documentation https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ The `jobTemplate` describes what to run, how to run it, and any volumes that it needs. As this follows the same structure as a Kubernetes Workload, we won't go too deep into the details here. ```yaml copy apiVersion: batch/v1 kind: CronJob metadata: name: cloudquery labels: app: cloudquery spec: schedule: "0 0 * * *" jobTemplate: spec: template: spec: containers: - name: cloudquery image: ghcr.io/cloudquery/cloudquery:latest imagePullPolicy: IfNotPresent args: ["sync", "/config/config.yml", "--log-console", "--log-format", "json"] envFrom: - secretRef: name: cloudquery-secret volumeMounts: - name: config mountPath: "/config" readOnly: true restartPolicy: Never volumes: - name: config configMap: name: cloudquery-config items: - key: "config" path: "config.yml" ``` Looking at this completed manifest, there are three key elements to pay attention to: `args`, `envFrom`, `volumes` In the `args`, you'll see the arguments to be passed to CloudQuery, in `envFrom` you're instructing Kubernetes to use the secret you created earlier, and in `volumes` you're telling Kubernetes where to find the configuration file. ## Step 3: Defining the ConfigMap to hold the CloudQuery Config When using Kubernetes configuration files are commonly stored on a special type of object called a ConfigMap. ConfigMaps enable you to define collections of text files that can be mounted as directories by Pods and Jobs. A ConfigMap definition is relatively simple, containing a `data` object where files are defined as key-value pairs. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: cloudquery-config data: {} ``` In this case, you want to define a ConfigMap to store a CloudQuery configuration file, that defines a Source integration and a Destination integration. ```yaml copy apiVersion: v1 kind: ConfigMap metadata: name: cloudquery-config data: config: | kind: source spec: # Source spec section name: digitalocean path: cloudquery/digitalocean registry: cloudquery version: "VERSION_SOURCE_DIGITALOCEAN" tables: - "digitalocean_droplets" - "digitalocean_databases" - "digitalocean_accounts" - "digitalocean_storage_volumes" - "digitalocean_floating_ips" - "digitalocean_firewalls" - "digitalocean_load_balancers" - "digitalocean_billing_history" destinations: ["postgresql"] --- kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" registry: "cloudquery" version: "VERSION_DESTINATION_POSTGRESQL" spec: connection_string: ${PG_CONNECTION_STR} ``` In this example we're using the DigitalOcean source integration and the PostgreSQL destination integration. But you can find out more about building configuration files and integrations [here](https://www.cloudquery.io/hub/plugins/source). ## Step 4: Apply the manifest Apply the configuration from a terminal using `kubectl apply` ```bash copy kubectl apply -f cronjob.yaml -f configmap.yaml ``` ## Step 5: Query the data You can manually trigger the cronjob to run early using: ```bash copy kubectl create job --from=cronjob/ ``` After which you can query your SQL database to see the results. ## Summary You now have CloudQuery running as a Kubernetes CronJob syncing to PostgreSQL. If you have any questions, check out the video above or join our [Community](https://community.cloudquery.io) to chat with us! ## Next Steps - [Docker Deployment](/cli/managing-cloudquery/deployments/docker) - Docker fundamentals for container-based deployments - [Running Syncs in Parallel](/cli/managing-cloudquery/running-in-parallel) - Distribute sync workloads across pods - [Monitoring](/cli/managing-cloudquery/monitoring/overview) - Set up observability for Kubernetes syncs --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/deployments/overview # Overview CloudQuery can run locally, but if you want to deploy in a remote non-ephemeral environment to sync periodically and store the data in a managed PostgreSQL, the recommended way is to deploy on Kubernetes (EKS or GKE) with our [helm-charts](https://github.com/cloudquery/helm-charts). We have also written some guides to help you get started on different platforms. For production deployments, consider [security best practices](/cli/managing-cloudquery/security) and [monitoring setup](/cli/managing-cloudquery/monitoring/overview). ## Tutorials {/* vale off */} export function Item({ slug, title }) { return
  • {title}
    • } export function List() { const items = Object.entries(meta).filter(([slug]) => slug !== 'overview').map(([slug, title]) => ) return
      {items}
    } {/* vale on */} ## Next Steps - [Choosing a Deployment](/cli/managing-cloudquery/deployments/choosing-a-deployment) - Compare deployment options and orchestrators - [Docker Deployment](/cli/managing-cloudquery/deployments/docker) - Get started with Docker containers - [Generate API Key](/cli/managing-cloudquery/deployments/generate-api-key) - Create an API key for headless deployments - [Monitoring](/cli/managing-cloudquery/monitoring/overview) - Set up observability for production syncs --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/environment-variables # Environment variables ## CLI environment variables The CloudQuery CLI reads the following environment variables to control its behavior: | Variable | Description | Default | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------ | ------- | | `CLOUDQUERY_API_KEY` | API key for headless authentication (CI/CD pipelines, automated workflows). Replaces the need for `cloudquery login`. | N/A | | `OTEL_ENDPOINT` | OpenTelemetry collector endpoint for CLI-level logs and traces. When set, the CLI sends telemetry data to this endpoint. | N/A | | `OTEL_ENDPOINT_INSECURE` | Set to `true` to disable TLS when connecting to the OpenTelemetry endpoint. | `false` | | `CQ_TELEMETRY_LEVEL` | Controls what telemetry data the CLI sends. Valid values: `none`, `errors`, `stats`, `all`. | `all` | | `CQ_NO_TELEMETRY` | **Deprecated.** Use `CQ_TELEMETRY_LEVEL=none` instead. When set to any value, disables telemetry. | N/A | For details on OpenTelemetry integration, see the [monitoring overview](/cli/managing-cloudquery/monitoring/overview). For telemetry details, see the [telemetry page](/cli/advanced/telemetry). ## Configuration file variable substitution CloudQuery configuration `.yml` files support substitution of values from environment variables. Use this to keep sensitive data (like passwords & tokens) or variable data (that you want to change without touching CloudQuery configuration) outside the configuration file and load them from environment variables at run-time. ## Environment variable substitution example Inside `postgresql.yml`: ```yaml copy kind: "destination" spec: name: "postgresql" spec: connection_string: ${PG_CONNECTION_STRING} ``` CloudQuery sources `PG_CONNECTION_STRING` from the environment and substitutes it before processing. ## File variable substitution example Inside `postgresql.yml`: ```yaml copy kind: "destination" spec: name: "postgresql" spec: connection_string: ${file:./path/to/secret/file} ``` CloudQuery reads the local path `./path/to/secret/file` and substitutes the file contents before processing. ## Environment variables with multi-line JSON Multi-line JSON, such as those required by the service account key for the GCP integration, can be imported by using pipe '|' operator. The substitution should be in the next line and it should be indented by a single tab before. You don't need to escape any characters while passing the variable. Inside `gcp.yml`: ```yaml copy kind: "source" spec: name: "gcp" spec: service_account_key_json: | ${GCP_SERVICE_ACCOUNT_KEY_JSON} ``` ## JSON files in older versions If the file or environment variable being substituted in contains JSON, import it as-is. If you're using CloudQuery version 3.5.0 or prior, wrap it in single quotes and content should be escaped with newlines removed. ```yaml copy kind: "destination" spec: name: "bigquery" spec: service_account_key_json: '${file:./path/to/secret/file.json}' # single quotes only for CLI versions 3.5.0 or prior ``` ## Time variable substitution example CloudQuery configuration files support value substitution using both relative and fixed timestamps. Relative timestamps can be specified as `now`, `x seconds [ago|from now]`, `x minutes [ago|from now]`, `x hours [ago|from now]`, `x days [ago|from now]`, or `until`. Fixed timestamps follow the RFC3339 (e.g. `2025-03-21T08:53:19+00:00`) or `dateOnly` formats (e.g. `2025-01-01`) Inside `aws.yml`: ```yaml copy kind: source spec: # Source spec section name: aws spec: table_options: aws_cloudtrail_events: lookup_events: - start_time: ${time:5 days ago} end_time: ${time:now} aws_cloudwatch_metrics: - list_metrics: namespace: AWS/RDS metric_name: DatabaseConnections get_metric_statistics: - start_time: ${time:2022-01-01} # dateOnly format end_time: ${time:2022-04-02T15:04:05Z} # RFC3339 format ``` For the `lookup_events` option, CloudQuery replaces `end_time` with the current time and `start_time` with the time 5 days ago. For the `get_metric_statistics` option `end_time` and `start_time` will be replaced by default with RFC3339 formatted value of the fixed time. You can also modify the output format by using the pipe operator `|` followed by your desired format. The time format should follow Go’s time layout specification: https://go.dev/src/time/format.go. For example, to display the date in `YYYY-MM-DD` format, use: `${time:now|2006-01-02}`. ## Next Steps - [Configuration Guide](/cli/core-concepts/configuration) - Set up source and destination configurations - [Security](/cli/managing-cloudquery/security) - Best practices for managing secrets - [Deployment Options](/cli/managing-cloudquery/deployments/overview) - Use environment variables in production deployments --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/migrations # How CloudQuery handles changes to existing tables CloudQuery handles schema changes by avoiding breaking changes and adding columns instead, because users commonly build views on top of synced tables. Migration strategies are implemented in the destination integrations, as source integrations are database agnostic and send back JSON objects. CloudQuery has [two modes](/cli/integrations/destinations#migrate_mode) of migrating to a new schema, `safe` which is supported by all destinations, and `forced` which is only supported by a few selected destinations at the moment. Refer to the specific destination documentation to see what modes of migration it supports. The `safe` mode is the default and will not run migrations that would result in data loss, and will print an error instead. The `forced` mode will run migrations that may result in data loss and the migration should always succeed without errors. Schema changes that require data loss only succeed with `forced` mode: | Schema change | Reasoning | | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Adding a new column that is a **primary key** or a **not null** column | New syncs **can't** succeed without back-filling the data, or dropping and re-adding the table | | Removing a column that is a **primary key** or a **not null** column | New syncs **can't** succeed as the column will not be populated with data, so dropping and re-adding the table is required | | Changing a column type | New syncs **can't** succeed without casting existing data into the new type, which is not always possible and can have performance implications in production environments | Schema changes that don't require data loss pass in both modes: | Schema change | Reasoning | | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | Adding a new column that is neither a **primary key** nor a **not null** column | New syncs **can** succeed by adding the new column to the existing table | | Removing a column that is neither a **primary key** nor a **not null** column | New syncs **can** succeed by ignoring the column removal | ## Next Steps - [Managing Versions](/cli/advanced/managing-versions) - Understand integration version management - [Destination Integrations](/cli/integrations/destinations) - Configure migration and write modes - [Source Integrations](/cli/integrations/sources) - Configure source schema options --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/monitoring/otel-datadog # OpenTelemetry and Datadog Send OpenTelemetry traces, metrics, and logs directly to Datadog using one of the options below. First, you will need to [setup OpenTelemetry with Datadog](https://docs.datadoghq.com/opentelemetry/). There are multiple ways to configure OpenTelemetry with Datadog. We'll show only a subset of them here, and you can find more information in the link above. ## Option 1: Using an OpenTelemetry collector To configure an OpenTelemetry collector with Datadog, you need to create a configuration file, for example `otel_collector_config.yaml` with the content below: ```yaml receivers: otlp: protocols: http: endpoint: "0.0.0.0:4318" processors: batch/datadog: send_batch_max_size: 1000 send_batch_size: 100 timeout: 10s exporters: datadog: api: site: ${env:DATADOG_SITE} key: ${env:DATADOG_API_KEY} service: pipelines: metrics: receivers: [otlp] processors: [batch/datadog] exporters: [datadog] traces: receivers: [otlp] processors: [batch/datadog] exporters: [datadog] logs: receivers: [otlp] processors: [batch/datadog] exporters: [datadog] ``` Then run the collector with the following command (replacing `DATADOG_SITE` and `DATADOG_API_KEY` with your own values): ```bash docker run \ -p 4318:4318 \ -e DATADOG_SITE=$DATADOG_SITE \ -e DATADOG_API_KEY=$DATADOG_API_KEY \ --hostname $(hostname) \ -v $(pwd)/otel_collector_config.yaml:/etc/otelcol-contrib/config.yaml \ otel/opentelemetry-collector-contrib:0.104.0 ``` > For additional ways to run the collector, see the [official documentation](https://docs.datadoghq.com/opentelemetry/collector_exporter/deployment#running-the-collector). ## Option 2: Direct OTel Ingestion by the Datadog Agent via a configuration file [Locate](https://docs.datadoghq.com/agent/configuration/agent-configuration-files/) your `datadog.yaml` file and add the following configuration: ```yaml otlp_config: receiver: protocols: http: endpoint: 0.0.0.0:4318 logs: enabled: true logs_enabled: true ``` [Restart](https://docs.datadoghq.com/agent/configuration/agent-commands/#restart-the-agent) the Datadog agent for the change to take effect. ## Option 3: Direct OTel ingestion by the Datadog Agent via environment variables Pass the `DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT` environment variable to the Datadog agent with a value of `0.0.0.0:4318`. If you're using Docker compose, you can find an example below: ```yaml version: "3.0" services: agent: image: gcr.io/datadoghq/agent:7 volumes: - /var/run/docker.sock:/var/run/docker.sock:ro - /proc/:/host/proc/:ro - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro environment: DD_API_KEY: redacted DD_SITE: "datadoghq.eu" DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT: "0.0.0.0:4318" DD_LOGS_ENABLED: "true" DD_OTLP_CONFIG_LOGS_ENABLED: "true" ports: - "4318:4318" ``` [Restart](https://docs.datadoghq.com/agent/configuration/agent-commands/#restart-the-agent) the Datadog agent for the change to take effect. > For additional ways to configure the Datadog agent, see the [official documentation](https://docs.datadoghq.com/opentelemetry/interoperability/otlp_ingest_in_the_agent#enabling-otlp-ingestion-on-the-datadog-agent). ## Start CloudQuery Configured with Datadog Once you have the agent or collector ready, you can specify the endpoint in the source spec: ```yaml kind: source spec: name: "aws" path: "cloudquery/aws" registry: "cloudquery" version: "VERSION_SOURCE_AWS" tables: ["aws_s3_buckets"] destinations: ["postgresql"] otel_endpoint: "0.0.0.0:4318" otel_endpoint_insecure: true spec: ``` Run `cloudquery sync spec.yml --log-level debug`. Running with `--log-level debug` is recommended to get more detailed logs about requests retries and errors. After ingestion starts, you should start seeing traces in the Datadog [**APM Traces Explorer**](https://app.datadoghq.com/apm/traces). You can also validate metrics and logs in the [**Metrics Summary**](https://app.datadoghq.com/metric/summary) and [**Log Explorer**](https://app.datadoghq.com/logs). ![Datadog](/images/docs/monitoring/cq_otel_datadog.png) We also provide a Datadog dashboard you can download from [here](/assets/datadog-dashboard.json) and import it into your Datadog account: 1. Click "New Dashboard" 2. In the name field, type "CloudQuery Sync Dashboard", then click "New Dashboard" 3. Click "Configure" -> "Import dashboard JSON…" 4. Drag the JSON file into the window, or copy-paste the contents. ![Datadog](/images/docs/monitoring/cq_otel_datadog_dashboard.png) ## Next Steps - [OpenTelemetry with Grafana](/cli/managing-cloudquery/monitoring/otel-grafana) - Alternative monitoring with Grafana - [Monitoring Overview](/cli/managing-cloudquery/monitoring/overview) - All monitoring options for CloudQuery - [Performance Tuning](/cli/advanced/performance-tuning) - Use metrics to optimize sync performance --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/monitoring/otel-grafana # OpenTelemetry and Grafana Visualize CloudQuery OpenTelemetry traces, metrics and logs with Grafana. We will use [Docker Compose][Compose] to run [Grafana][Grafana] and related services, so make sure you have it installed on your machine. ## Step 1: Creating a Docker Compose file We will use [Tempo][Tempo] for ingesting traces, [Loki][Loki] for logs, [Prometheus][Prometheus] for metrics, and the [OpenTelemetry collector][Collector] for collecting and forwarding the data to each service. Create a file named `docker-compose.yml` with the following content: ```yaml version: "3.8" services: tempo: image: grafana/tempo:latest command: ["-config.file=/etc/tempo.yaml"] volumes: - tempo_data:/tmp - ./tempo/tempo.yaml:/etc/tempo.yaml ports: - "3200" - "4318" loki: image: grafana/loki:latest ports: - "3100" command: -config.file=/etc/loki/local-config.yaml collector: image: otel/opentelemetry-collector-contrib:latest ports: - "4318:4318" # 4318 needs to be exposed to the host for the collector to ingest data - "8090" volumes: - ./collector/collector.yaml:/etc/otelcol-contrib/config.yaml prometheus: image: prom/prometheus:latest command: - "--enable-feature=remote-write-receiver" - "--config.file=/etc/prometheus/prometheus.yaml" ports: - "9090" volumes: - prometheus:/prometheus - ./prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml grafana: image: grafana/grafana-enterprise volumes: - grafana_data:/var/lib/grafana - ./grafana/datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml - ./grafana/dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml - ./grafana/cloudquery-dashboard.json:/var/lib/grafana/dashboards/cloudquery-dashboard.json environment: GF_FEATURE_TOGGLES_ENABLE: "tempoApmTable" ports: - "3000:3000" # 3000 needs to be exposed to the host for the Grafana UI volumes: prometheus: driver: local grafana_data: driver: local tempo_data: driver: local ``` This Docker Compose file configures [Prometheus][Prometheus], [Tempo][Tempo], an [OpenTelemetry collector][Collector] and Grafana with a custom configuration, and [Loki][Loki] with the default configuration. ## Step 2: Configure Prometheus Create a file with the path `prometheus/prometheus.yaml` with the following content: ```yaml global: scrape_interval: 15s scrape_configs: - job_name: "opentelemetry" static_configs: - targets: ["collector:8090"] ``` This configuration will tell [Prometheus][Prometheus] to scrape the [OpenTelemetry][OpenTelemetry] collector every 15 seconds. ## Step 3: Configure Tempo Create a file with the path `tempo/tempo.yaml` with the following content: ```yaml server: http_listen_port: 3200 distributor: receivers: otlp: protocols: http: storage: trace: backend: local wal: path: /tmp/tempo/wal local: path: /tmp/tempo/blocks # Needed for aggregation functions, e.g. quantile_over_time # Visit https://grafana.com/docs/tempo/latest/traceql/metrics-queries/ for more information query_frontend: search: max_duration: 0 metrics: max_duration: 0 overrides: metrics_generator_processors: ["local-blocks"] metrics_generator: processor: local_blocks: filter_server_spans: false storage: path: /var/tempo/generator/wal traces_storage: path: /var/tempo/generator/traces ``` This configuration will tell [Tempo][Tempo] to listen on port 3200 and receive [OpenTelemetry][OpenTelemetry] traces via HTTP on the default port of 4318. ## Step 4: Configure the OpenTelemetry collector Create a file with the path `collector/collector.yaml` with the following content: ```yaml receivers: otlp: protocols: http: endpoint: "0.0.0.0:4318" processors: batch: exporters: prometheus: endpoint: collector:8090 otlphttp: endpoint: http://tempo:4318 loki: endpoint: http://loki:3100/loki/api/v1/push service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlphttp] metrics: receivers: [otlp] processors: [batch] exporters: [prometheus] logs: receivers: [otlp] exporters: [loki] ``` This configuration will tell the [OpenTelemetry collector][Collector] to receive traces, metrics, and logs and forward them to [Tempo][Tempo], [Prometheus][Prometheus] and [Loki][Loki], respectively. ## Step 5: Configure Grafana Data Sources Create a file with the path `grafana/datasources.yaml` with the following content: ```yaml apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy orgId: 1 url: http://prometheus:9090 basicAuth: false isDefault: false version: 1 editable: true uid: prometheus - name: Loki type: loki access: proxy orgId: 1 url: http://loki:3100 basicAuth: false isDefault: false version: 1 editable: true uid: loki - name: Tempo type: tempo access: proxy orgId: 1 url: http://tempo:3200 basicAuth: false isDefault: true version: 1 editable: true apiVersion: 1 uid: tempo ``` This configuration will tell [Grafana][Grafana] to use [Prometheus][Prometheus], [Loki][Loki], and [Tempo][Tempo] as data sources. ## Step 6: Download the CloudQuery Grafana Dashboard Create a file with the path `grafana/cloudquery-dashboard.json` with the content from [here](/assets/grafana-dashboard.json). > If you'd like to import the dashboard to an existing Grafana instance, you can download an external version of it from [here](/assets/grafana-dashboard-external.json). ## Step 7: Configure Grafana with the CloudQuery Dashboard Create a file with the path `grafana/dashboards.yaml` with the following content: ```yaml apiVersion: 1 providers: - name: CloudQuery folder: CloudQuery type: file allowUiUpdates: true options: path: /var/lib/grafana/dashboards ``` This configuration will tell [Grafana][Grafana] to load the [CloudQuery][CloudQuery] dashboard. ## Step 8: Start the services Run `docker-compose up` to start the services. Once the services are up and running, you should be able to access [Grafana][Grafana] at [http://localhost:3000](http://localhost:3000) with the default credentials `admin:admin`. ## Step 9: Configure a Source Integration with OpenTelemetry You can use the example source configuration below to start a sync with [OpenTelemetry][OpenTelemetry] enabled: ```yaml kind: source spec: name: "aws" path: "cloudquery/aws" registry: "cloudquery" version: "VERSION_SOURCE_AWS" tables: ["aws_s3_buckets"] destinations: ["postgresql"] otel_endpoint: "0.0.0.0:4318" otel_endpoint_insecure: true spec: ``` ## Step 10: Run the sync Run `cloudquery sync spec.yml --log-level debug`. Running with `--log-level debug` is recommended to get more detailed logs about requests retries and errors. After ingestion starts, you can access the [dashboard](http://localhost:3000/d/6_bNYpGVz/cloudquery-dashboard?orgId=1) to see sync insights, traces, metrics, and logs. [CloudQuery]: / [OpenTelemetry]: https://opentelemetry.io/ [Compose]: https://docs.docker.com/compose/install/ [Grafana]: https://grafana.com/ [Tempo]: https://grafana.com/oss/tempo/ [Loki]: https://grafana.com/oss/loki/ [Prometheus]: https://prometheus.io/ [Collector]: https://opentelemetry.io/docs/collector ## Next Steps - [OpenTelemetry with Datadog](/cli/managing-cloudquery/monitoring/otel-datadog) - Alternative monitoring with Datadog - [Monitoring Overview](/cli/managing-cloudquery/monitoring/overview) - All monitoring options for CloudQuery - [Dashboards & Visualizations](/cli/core-concepts/dashboards) - Use Grafana dashboards for CloudQuery data --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/monitoring/overview # Overview Monitor CloudQuery using any of these methods: - [Logging](#logging) - [OpenTelemetry](#opentelemetry-preview) - [Datadog](/cli/managing-cloudquery/monitoring/otel-datadog) - [Grafana](/cli/managing-cloudquery/monitoring/otel-grafana) ## Logging CloudQuery utilizes structured [logging](/cli/cli-reference/cloudquery) (in plain and JSON formats) which can be analyzed by local tools such as `jq`, `grep` and remote aggregations tools like `loki`, `datadog` or any other popular log aggregation that supports structured logging. ## OpenTelemetry (Preview) > **Preview:** OpenTelemetry support is in preview. The configuration interface (`otel_endpoint`, `otel_endpoint_insecure`) is functional but may change in future releases. We do not currently provide stability guarantees for this feature. ELT workloads can be long running and sometimes it is necessary to better understand what calls are taking the most time, to optimize those on the integration side, ignore them or split them to a different workload. CloudQuery supports [OpenTelemetry](https://opentelemetry.io/) traces, metrics and logs out of the box. Configure it using `otel_endpoint` and `otel_endpoint_insecure` in your source spec (see [Configuration](/cli/core-concepts/configuration)). To collect OpenTelemetry data you need a [backend](https://opentelemetry.io/docs/concepts/components/#exporters) that supports the OpenTelemetry protocol. For example you can use [Jaeger](https://opentelemetry.io/docs/instrumentation/go/exporters/#jaeger) to visualize and analyze traces. To start Jaeger locally you can use Docker: ```bash docker run -d \ -e COLLECTOR_OTLP_ENABLED=true \ -p 16686:16686 \ -p 4318:4318 \ jaegertracing/all-in-one:1.58 ``` and then specify in the source spec the following: ```yaml kind: source spec: name: "aws" path: "cloudquery/aws" registry: "cloudquery" version: "VERSION_SOURCE_AWS" tables: ["aws_s3_buckets"] destinations: ["postgresql"] otel_endpoint: "localhost:4318" otel_endpoint_insecure: true # this is only in development when running local jaeger spec: ``` After that you can open [http://localhost:16686](http://localhost:16686) and see the traces: ![jaeger](/images/docs/jaeger.png) In production, it is common to use an OpenTelemetry [collector](https://opentelemetry.io/docs/concepts/components/#collector) that runs locally or as a gateway to batch the traces and forward it to the final backend. This helps with performance, fault-tolerance and decoupling of the backend in case the tracing backend changes. ## CLI-level OpenTelemetry In addition to per-integration `otel_endpoint` configuration in source specs, the CLI itself supports global OpenTelemetry via environment variables: | Variable | Description | | ------------------------ | -------------------------------------------------------------- | | `OTEL_ENDPOINT` | OpenTelemetry collector endpoint for CLI-level logs and traces | | `OTEL_ENDPOINT_INSECURE` | Set to `true` to disable TLS when connecting to the endpoint | When `OTEL_ENDPOINT` is set, the CLI sends its own logs and traces to the specified collector, independent of any per-integration OpenTelemetry configuration. ### Correlating CLI and integration traces Use the `--invocation-id` flag to set a consistent identifier across CLI traces and integration traces: ```bash cloudquery sync config.yaml --invocation-id ``` This is useful in distributed environments where you need to correlate CLI-level traces with integration-level traces across services. When not specified, a random UUID is generated for each invocation. For the full list of CLI environment variables, see the [environment variables reference](/cli/managing-cloudquery/environment-variables). ## Next Steps - [OpenTelemetry with Datadog](/cli/managing-cloudquery/monitoring/otel-datadog) - Send traces and metrics to Datadog - [OpenTelemetry with Grafana](/cli/managing-cloudquery/monitoring/otel-grafana) - Visualize metrics in Grafana - [Performance Tuning](/cli/advanced/performance-tuning) - Use monitoring data to optimize syncs - [Telemetry](/cli/advanced/telemetry) - Configure CLI telemetry settings --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/proxy-configuration # Using CloudQuery CLI with a Proxy Server If you run the CloudQuery CLI in an environment that requires a proxy server for outgoing traffic, you'll need to set it via environment variables. To configure a proxy server for HTTPS traffic set the `HTTPS_PROXY` environment variable. For HTTP traffic set the `HTTP_PROXY` environment variable. Example: ```bash copy export HTTPS_PROXY=http://example.com:3128/proxy export HTTP_PROXY=http://example.com:3128/proxy cloudquery sync [files or directories] ``` `HTTPS` in `HTTPS_PROXY` variable name means that it is a proxy for HTTPS requests, and not the protocol of the proxy, so its value can start with `http://`. ## Next Steps - [Environment Variables](/cli/managing-cloudquery/environment-variables) - Set proxy variables and other configuration - [Security](/cli/managing-cloudquery/security) - CloudQuery security best practices - [Deployment Options](/cli/managing-cloudquery/deployments/overview) - Deploy CloudQuery in production environments --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/rate-limiting # Rate Limiting There is one main lever to control the rate at which CloudQuery fetches resources from cloud providers. This setting is called `concurrency` available in most source integrations, and it can be specified as part of the integration source spec (Each spec is described in the relevant page in the [hub](https://www.cloudquery.io/hub/plugins/source)). ## Concurrency `concurrency` provides rough control over the number of concurrent requests that will be made while performing a sync. Setting this to a low number will reduce the number of concurrent requests, reducing the memory used and making the sync less likely to hit rate limits. The trade-off is that syncs will take longer to complete. ## Next Steps - [Performance Tuning](/cli/advanced/performance-tuning) - Optimize overall sync performance - [Source Integrations](/cli/integrations/sources) - Configure concurrency per source integration - [Running Syncs in Parallel](/cli/managing-cloudquery/running-in-parallel) - Distribute workloads across processes --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/running-in-parallel # Running CloudQuery in Parallel Running multiple instances of `cloudquery sync` in parallel can be useful when a single sync is too slow, for example when syncing a large number of accounts, or when fetching from large accounts. ## Splitting Syncs Automatically Starting from version [v6.8.0](https://github.com/cloudquery/cloudquery/releases/tag/cli-v6.8.0) of the CloudQuery CLI, you can use the `--shard` flag to automatically split a sync into smaller parts that can be run in parallel. For example, to split a sync into 4 parts, you can run: ```bash cloudquery sync config.yml --shard 1/4 cloudquery sync config.yml --shard 2/4 cloudquery sync config.yml --shard 3/4 cloudquery sync config.yml --shard 4/4 ``` The `shard` flag will automatically split the sync into parts, ensure each part gets a unique source name, and that the parts don't overlap. It's recommended to run the parts in parallel, as the sync will be faster than running a single sync. You can find an example of how to run the syncs in parallel in the [GitHub Actions Deployment Guide](/cli/managing-cloudquery/deployments/github-actions#running-cloudquery-in-parallel-to-speed-up-sync-time) section. ### Supported Source Integrations for Sharding | Source Integration | Minimal Version | | ------------------ | ------------------------------------------------------------------------------------ | | AWS | [v27.20.0](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs) | | Azure | [v14.8.0](https://www.cloudquery.io/hub/plugins/source/cloudquery/azure/latest/docs) | | GCP | [v16.3.0](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs) | ## Splitting Syncs Manually If you are using an older version of the CloudQuery CLI, or if you want to manually split a sync, you can do so by creating different configurations for each part of the sync, using the guidelines below. ### Unique Names Every source and destination integration configuration must have a unique `name`. This is required because the `name` is written into the database (`_cq_source_name`), and is used to later delete stale resources. For instance, a configuration with multiple source integrations could look like: ```yaml copy kind: source spec: name: aws1 path: cloudquery/aws registry: cloudquery ... --- kind: source spec: name: aws2 path: cloudquery/aws registry: cloudquery ... --- kind: destination spec: name: "postgresql" path: cloudquery/postgresql registry: cloudquery ... ``` If the names are not unique, then the different integrations may delete/overwrite each other's resources. ### No Overlapping Syncs When splitting a sync into multiple source-integration configurations to be run in parallel, these syncs must not overlap - the set of Account/Table/Region that every source-integration grabs must not intersect. For instance, in GCP, if the first source-integration fetches resource `A` from project `1`, the second source-integration can fetch resource `B` from project `1`, or resource `A` from project `2`, but can never fetch resource `A` from project `1`. For another example, if the first source-integration fetches from region `europe-west1` in project `1`, the second source-integration can fetch from region `europe-west1` in project `2`, or from region `europe-west2` in project `1`, but can never fetch from region `europe-west1` in project `1`. If the configurations overlap, the behavior is undefined, and the database may contain duplicate rows. ## Next Steps - [Performance Tuning](/cli/advanced/performance-tuning) - Further optimize sync speed - [Deployment Options](/cli/managing-cloudquery/deployments/overview) - Choose a deployment model for parallel syncs - [Destination Integrations](/cli/integrations/destinations) - Configure sync_group_id for parallel writes --- Source: https://www.cloudquery.io/docs/cli/managing-cloudquery/security # Security Key security points for self-hosted CloudQuery instances. Follow these best practices when running CloudQuery on your own infrastructure. ## Integration Authentication Credentials - Integration Authentication Credentials should always be read-only. - The machine where CloudQuery is running should be secured with the correct permissions, as it contains the credentials to your cloud infrastructure. ## CloudQuery Database Even though in most cases the CloudQuery database contains only configuration and meta-data, you should protect it and keep it secure with correct access and permissions. ## Next Steps - [Environment Variables](/cli/managing-cloudquery/environment-variables) - Manage secrets with environment variables - [Generate API Key](/cli/managing-cloudquery/deployments/generate-api-key) - Create API keys for authenticated access - [Proxy Configuration](/cli/managing-cloudquery/proxy-configuration) - Route traffic through proxies --- Source: https://www.cloudquery.io/docs {/* This file is intentionally empty. Nextra uses it as a routing placeholder. */} --- Source: https://www.cloudquery.io/docs/platform/advanced-topics/api-automation # Automating with the Platform API The CloudQuery Platform API gives you programmatic access to everything in the platform UI. This guide walks through common automation patterns: triggering syncs from a CI/CD pipeline, checking policy violations, provisioning users, and managing platform configuration as code. For a full API overview, see [Platform API](/platform/reference/api-reference). For interactive endpoint documentation, see the [Interactive API Reference ↗](https://platform-multi-tenant-api-docs.cloudquery.io/). ## Prerequisites - A CloudQuery Platform deployment with at least one integration and sync configured - An API key with the appropriate role for your use case (created in **Organization Settings → API Keys**) ### Choosing the right API key role | Role | Use case | |---|---| | `ci` | CI/CD pipelines: trigger syncs, check policy violations | | `general:read` | Read-only scripts: list resources, pull reports, fetch audit logs | | `general:write` | Automation that creates or modifies syncs, policies, or alerts | | `admin:read` | Scripts that read user, team, or platform settings | | `admin:write` | Provisioning automation: manage users, teams, SSO config | Store API keys in a secrets manager (AWS Secrets Manager, HashiCorp Vault, GitHub Actions secrets) rather than in plaintext config files. --- ## Triggering syncs from CI/CD A common pattern is to trigger a sync at the end of a deployment pipeline so your cloud inventory stays current after infrastructure changes. ### Trigger a sync and wait for completion Syncs are identified by name. The following script triggers a run, captures the run ID, and polls until it completes or fails: ```bash #!/bin/bash RUN_ID=$(curl -s -X POST \ -H "Authorization: Bearer $CLOUDQUERY_API_KEY" \ "https://$PLATFORM_URL/api/syncs/$SYNC_NAME/runs" \ | jq -r '.id') echo "Sync run started: $RUN_ID" while true; do STATUS=$(curl -s \ -H "Authorization: Bearer $CLOUDQUERY_API_KEY" \ "https://$PLATFORM_URL/api/syncs/$SYNC_NAME/runs/$RUN_ID" \ | jq -r '.status') echo "Status: $STATUS" if [ "$STATUS" = "completed" ]; then echo "Sync completed successfully" break elif [ "$STATUS" = "failed" ]; then echo "Sync failed" exit 1 fi sleep 30 done ``` --- ## Enforcing policies in CI/CD Use the Platform API to check policy violations after a sync. This lets you fail a pipeline when violations are detected, for example, blocking a deployment if it would introduce a compliance violation. ```bash #!/bin/bash VIOLATIONS=$(curl -s \ -H "Authorization: Bearer $CLOUDQUERY_API_KEY" \ "https://$PLATFORM_URL/api/policies/$POLICY_ID/violations" \ | jq '.total_violations') echo "Policy violations: $VIOLATIONS" if [ "$VIOLATIONS" -gt 0 ]; then echo "Policy check failed: $VIOLATIONS violation(s) found" exit 1 fi echo "Policy check passed" ``` --- ## Managing configuration as code Instead of clicking through the UI, define policies and syncs in scripts or IaC tooling and apply them via the API. This gives you version control, code review, and repeatable deployments. ### Create a policy ```bash curl -s -X POST \ -H "Authorization: Bearer $CLOUDQUERY_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Public S3 Buckets", "description": "Detects S3 buckets with public access enabled", "query": "SELECT account_id, name, region FROM aws_s3_buckets WHERE bucket_policy_is_public = true" }' \ "https://$PLATFORM_URL/api/policies" ``` ### List existing syncs ```bash curl -s \ -H "Authorization: Bearer $CLOUDQUERY_API_KEY" \ "https://$PLATFORM_URL/api/syncs" \ | jq '[.syncs[] | {name, status: .last_run_status}]' ``` For the full set of create/update endpoints for syncs, alerts, and reports, see the [Interactive API Reference ↗](https://platform-multi-tenant-api-docs.cloudquery.io/). --- ## Provisioning users programmatically Automate user onboarding and offboarding instead of managing it through the UI. This is useful when integrating with an identity provider or HR system. ### Add a user to the platform ```bash curl -s -X POST \ -H "Authorization: Bearer $CLOUDQUERY_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "email": "newuser@example.com", "role": "general:read" }' \ "https://$PLATFORM_URL/api/users" ``` ### List users ```bash curl -s \ -H "Authorization: Bearer $CLOUDQUERY_API_KEY" \ "https://$PLATFORM_URL/api/users" \ | jq '[.users[] | {email, role}]' ``` ### Remove a user ```bash curl -s -X DELETE \ -H "Authorization: Bearer $CLOUDQUERY_API_KEY" \ "https://$PLATFORM_URL/api/users/$USER_ID" ``` --- ## Pulling audit logs into external systems Export the audit log to feed into a SIEM, data warehouse, or compliance reporting tool. Use `per_page` and `page` to paginate through results: ```bash #!/bin/bash # Fetch the first 1000 audit log entries and write to a file curl -s \ -H "Authorization: Bearer $CLOUDQUERY_API_KEY" \ "https://$PLATFORM_URL/api/audit-logs?per_page=1000&page=1" \ | jq '.' > audit-log-$(date +%Y%m%d).json ``` Filter by time range using `start_time` and `end_time`: ```bash curl -s \ -H "Authorization: Bearer $CLOUDQUERY_API_KEY" \ "https://$PLATFORM_URL/api/audit-logs?start_time=2024-01-01T00:00:00Z&end_time=2024-01-31T23:59:59Z" ``` --- ## Next steps - [Platform API](/platform/reference/api-reference): API overview, authentication, and categories - [API Keys](/platform/production-deployment/api-keys): create and manage API keys - [Policies](/platform/features/policies): write SQL-based compliance controls - [Alerts](/platform/features/alerts): configure notifications for policy violations - [Platform API Reference ↗](https://platform-multi-tenant-api-docs.cloudquery.io/): full endpoint documentation --- Source: https://www.cloudquery.io/docs/platform/advanced-topics/custom-columns # Custom Columns With Custom Columns, you can define new columns on any asset table and populate them via SQL expressions (e.g., extracting tags or fields). Once added, these columns become: - Filterable in the UI - Searchable via SQL - Queryable in reports and dashboards --- ## Creating a Custom Column 1. Navigate to **Organization settings** → **Custom columns**. 2. Click **Add custom column**. 3. Fill in the form: - **Column label**: the display name shown in the UI (e.g., “Team”) - **Column name**: the backend identifier in `snake_case` (e.g., `team`) - **Description** _(optional)_: purpose of the column - **Value expression**: a ClickHouse SQL expression to extract or transform the value (e.g., `tags['team']`) 4. Click **Create custom column**. Once saved, the column appears in the Asset Inventory and can be filtered and searched like native fields. --- ## Supported SQL Expressions You can use any valid [ClickHouse SQL expression](https://clickhouse.com/docs/en/sql-reference/functions/) when the value type is expression. Here are some common examples: ### Tag Extraction ```sql tags['team'] tags['cost_center'] ``` ### JSON Parsing ```sql JSONExtractString(tags['metadata'], 'department') JSONExtract(tags['finance'], 'budget', 'Float64') ``` ### Normalization / Cleanup ```sql lower(tags['env']) -- normalize prod, Prod, PROD multiIf( tags['env'] = 'production', 'prod', tags['env'] = 'staging', 'stage', tags['env'] ) ``` ### Conditional Defaults ```sql COALESCE(NULLIF(tags['team'],''),'unknown') ``` ### Lifecycle Metadata ```sql parseDateTimeBestEffort(tags['deletion_date']) toDate(now() + INTERVAL 30 DAY) ``` ## Example Use Cases | Column | Type | Expression / Source | Description | | ------------------ | ------ | ------------------------------------------ | --------------------------------- | | team | String | tags\['team'] | Surface team ownership | | `normalized_env` | String | lower(tags\['env']) | Normalize inconsistent tag values | | `scheduled_delete` | Date | parseDateTimeBestEffort(tags\['deletion']) | Track lifecycle cleanup dates | ## Query Examples ```sql SELECT account, region, name, tags, team FROM cloud_assets WHERE team == '' ``` ## Programmatic access Custom columns can be created and managed via the Platform API, useful for IaC-style column management. See the [Platform API Reference](/platform/reference/api-reference) (`custom-columns` section) for endpoint details. ## Next Steps - [Asset Inventory](/platform/features/asset-inventory) - Use custom columns in the asset inventory - [SQL Console](/platform/features/sql-console) - Query custom column data with SQL --- Source: https://www.cloudquery.io/docs/platform/advanced-topics/performance-tuning # Performance Tuning This article describes performance tuning of Syncs run on CloudQuery Platform. For CloudQuery CLI specific options, see [/advanced/performance-tuning](/cli/advanced/performance-tuning) ## Identifying Slow Tables To improve sync performance, start by identifying which tables are taking the longest to sync. Open sync run details to see the individual tables synced and browse through the tables with the highest amount of rows and check their run time. Consider whether you actually need the tables or services to be synced.
    ![Sync run details showing individual table sync times and row counts to identify slow tables](/images/platform/performance-1.png)
    ## Tune Concurrency to work around rate limiting The main lever to control the rate at which CloudQuery fetches resources from cloud providers is the `concurrency` option, available in most source integrations. Specify it as part of the integration source configuration when using YAML, or as an independent input when configuring a new Integration. The `concurrency` option provides rough control over the number of concurrent requests that will be made while performing a sync. Setting this to a low number will reduce the number of concurrent requests, reducing the memory used and making the sync less likely to hit rate limits. The trade-off is that syncs will take longer to complete. ## Adjust Batch Size Most destination integrations have batching related settings that can be adjusted to improve performance. Tuning these can improve performance, but it can also increase the memory usage of the sync process. Here are the batching related settings you will come across: - `batch_size`: The number of rows that are inserted into the destination at once. The default value for this setting is usually between 1000 to 10000 rows, depending on the destination integration. - `batch_size_bytes`: Maximum size of items that may be grouped together to be written in a single write. This is useful for limiting the memory usage of the sync process. The default value for this varies between 4 MB to 100 MB, depending on the destination integration. - `batch_timeout`: Maximum interval between batch writes. Even if data stops coming in, the batch will be written after this interval. The default value for this setting is usually between 10 seconds and 1 minute, depending on the destination integration. Some destination integrations (such as file or S3 destinations) start a new object or file for every batch, and some buffer the data in memory to be written at once. You should check the documentation for the destination integration you are using to see what the default values are and consider how they can be adjusted to suit your use case. Here's a conservative example for the PostgreSQL destination integration that reduces the overall memory usage, but may also increase the time it takes to sync: ```yaml kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" registry: "cloudquery" version: "vX.X.X" # replace with latest from https://www.cloudquery.io/hub/plugins/destination/cloudquery/postgresql spec: connection_string: "postgres://user:pass@localhost:5432/mydb?sslmode=disable" # replace with your connection string batch_size: 10000 # 10000 rows, default batch_size_bytes: 4194304 # 4 MB, dramatically tuned down from the 100 MB default batch_timeout: "30s" # 30 seconds, tuned down from 60 seconds ``` With this configuration, the PostgreSQL destination integration will write 10,000 rows at a time, or 4 MB of data at a time, or every 30 seconds, whichever comes first. ## Use a Different Scheduler This option is available only when setting up an integration via API or using YAML configuration. By default, CloudQuery syncs will fetch all tables in parallel, writing data to the destination(s) as they come in. However, the `concurrency` setting, mentioned above, places a limit on how many **table-clients** can be synced at a time. What "table-client" means depends on the source integration and the table. In AWS, for example, a client is usually a combination of account and region. Get all the combinations of accounts and regions for all tables, and you have all the table-clients for a sync. For the GCP source integration, clients generally map to projects. The default CloudQuery scheduler, known as `dfs`, will sync up to `concurrency / 100` table-clients at a time (we are ignoring child relations for the purposes of this discussion). Let's take an example GCP cloud estate with 5000 projects, syncing 100 tables. This makes for approximately 500,000 table-client pairs, and a concurrency of 10,000 will allow 100 table-client pairs to be synced at a time. The `dfs` scheduler will start with the first table and its first 100 projects, and then move on to finish all projects for that table before moving on to the next table. This means, in practice, only one table is really being synced at a time! Usually this works out fine, as long as the cloud platform's rate limits are aligned with the clients. But if rate limits are applied per-table, rather than per-project, `dfs` can be suboptimal. A better strategy in this case would be to choose the first client for every table before moving on to the next client. This is what the `round-robin` scheduler does. Only some integrations support this setting. The following example configuration enables `round-robin` scheduling for the GCP source integration: ```yaml kind: source spec: name: "gcp" path: "cloudquery/gcp" registry: "cloudquery" version: "vX.X.X" # replace with latest from https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp tables: ["gcp_storage_*", "gcp_compute_*"] destinations: ["postgresql"] spec: scheduler: "round-robin" project_ids: ... ``` Finally, the `shuffle` strategy aims to provide a balance between `dfs` and `round-robin` by randomizing the order in which table-client pairs are chosen. The following example enables `shuffle` for the GCP integration, which can help reduce the likelihood of hitting rate limits by randomly mixing the underlying services to which API calls that are made concurrently, rather than hitting a single API with all calls at once: ```yaml kind: source spec: name: "gcp" path: "cloudquery/gcp" registry: "cloudquery" version: "vX.X.X" # replace with latest from https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp tables: ["gcp_storage_*", "gcp_compute_*"] destinations: ["postgresql"] spec: project_ids: ... scheduler: "shuffle" # ... ``` The `shuffle` scheduler is the **default** for the AWS source integration. ## Next Steps - [Monitoring Sync Status](/platform/syncs/monitoring-sync-status) - Track sync performance - [Setting up a Sync](/platform/syncs/setting-up-a-sync) - Configure sync settings - [CLI Performance Tuning](/cli/advanced/performance-tuning) - Performance tuning for self-hosted CLI syncs --- Source: https://www.cloudquery.io/docs/platform/advanced-topics/understanding-platform-views # Understanding Platform Views When you query data on CloudQuery Platform (whether in the [Asset Inventory](/platform/features/asset-inventory), the [SQL Console](/platform/features/sql-console), or through [Policies](/platform/features/policies)), you're querying views, not the raw synced tables directly. Understanding how these views work helps when writing SQL queries, debugging sync results, or configuring [data access controls](/platform/production-deployment/user-management/limiting-access-to-data). Platform views only apply to the default ClickHouse destination. If you sync to a custom destination (e.g. PostgreSQL or BigQuery), views are not created and you query the tables directly. See [Syncs](/platform/core-concepts/syncs) for more on write modes with custom destinations. As described in [Syncs](/platform/core-concepts/syncs), the platform stores synced data in tables prefixed with `raw_` and creates views on top of them. This page explains how those views are constructed. ## Cloud Assets View When a sync runs on CloudQuery Platform, the sync is configured (through use of a transformer) to prefix all output tables with `raw_` (e.g., `raw_aws_ec2_instances`, `raw_gcp_compute_instances`). Each raw table row includes a `_cq_sync_group_id`, a unique identifier for the sync run that produced it. As tables complete syncing, the platform builds a **snapshot table** that combines and normalizes data from all raw tables into a single schema. This snapshot contains columns like `cloud`, `account`, `name`, `region`, `resource_type`, and `tags`, providing a consistent structure regardless of the source integration. Historical (full) tables and incremental tables are handled differently: - **Historical tables** (the majority of tables) are filtered to only the latest complete `_cq_sync_group_id`, so stale data from previous syncs is excluded. - **Incremental tables** include all rows, since incremental tables only ever add new data. During this process, rows are deduplicated so that only the most recent version of each record (by `_cq_id`) is included. Finally, the `cloud_assets` view is atomically pointed at the new snapshot table. Old snapshot tables are cleaned up automatically. This atomic swap means you always query a complete, consistent dataset, never partial sync results. ![Diagram showing how raw tables flow through `cloud_assets_historical` and `cloud_assets_incremental` into the unified `cloud_assets` view](/images/platform/platform-views-1.png) For example, you can query the `cloud_assets` view directly in the SQL Console: ```sql SELECT account, resource_type, name, tags FROM cloud_assets WHERE resource_type = 'aws_ec2_instances' ``` This returns only the latest snapshot of EC2 instances, even though the underlying `raw_aws_ec2_instances` table may contain data from multiple syncs. ## Table Views Table views follow the same pattern but are simpler. For every raw table, the platform creates a view with the original table name (e.g., `raw_aws_ec2_instances` becomes `aws_ec2_instances`). Behind the scenes, the platform: 1. Creates a new snapshot table from the raw table, filtered to the latest complete `_cq_sync_group_id` and deduplicated. 2. Atomically replaces the view to point at the new snapshot. 3. Drops the old snapshot table. This keeps data consistent during syncs and switches atomically, while still allowing records to be appended efficiently into ClickHouse.
    ![Diagram illustrating how table views are defined.](/images/platform/platform-views-2.png) Diagram illustrating how table views are defined.
    ### What Happens During a Sync? While a sync is in progress, queries against views continue to return the **previous complete snapshot**. The view only switches to include new data once a table's sync is fully complete and the new snapshot is built. This means you never see partial or in-progress data. ## Related Pages - [Syncs](/platform/core-concepts/syncs): how sync modes and `raw_` prefixing work - [Asset Inventory](/platform/features/asset-inventory): the UI built on the `cloud_assets` view - [SQL Console](/platform/features/sql-console): query views directly with SQL - [Custom Columns](/platform/advanced-topics/custom-columns): enrich the `cloud_assets` view with custom metadata ## Next Steps - [SQL Console](/platform/features/sql-console) - Query platform views with SQL - [Historical Snapshots](/platform/features/sql-console/historical-snapshots) - Query point-in-time data snapshots - [Data Management](/platform/production-deployment/data-management) - Manage underlying data storage --- Source: https://www.cloudquery.io/docs/platform/core-concepts/data-model # Data Model When CloudQuery Platform syncs data from your cloud providers, the data lands in ClickHouse tables organized by integration and resource type. The platform then builds normalized views on top of this raw data, giving you both detailed per-resource tables and a unified cross-cloud inventory. ## How Tables Are Organized Each source integration creates one table per resource type. For example, the AWS integration creates tables like `aws_ec2_instances`, `aws_s3_buckets`, and `aws_iam_roles`. The Azure integration creates `azure_compute_virtual_machines`, `azure_storage_accounts`, and so on. During a sync, data is first written to staging tables prefixed with `raw_` (e.g. `raw_aws_ec2_instances`). Once a table's sync completes, the platform creates a view with the original name (`aws_ec2_instances`) that points to the latest complete snapshot. This means the data you query is always consistent - it switches atomically when a sync finishes, rather than showing partial results mid-sync. You can query these per-resource tables directly in the [SQL Console](/platform/features/sql-console) using standard ClickHouse SQL. For the full technical details on how staging tables, views, and snapshots work, see [Understanding Platform Views](/platform/advanced-topics/understanding-platform-views). ## The Cloud Assets Table The platform also maintains a unified table called `cloud_assets` that normalizes resources from all your integrations into a common schema. This is what powers the [Asset Inventory](/platform/features/asset-inventory), and it's what you query when you need to search across clouds. Every resource in `cloud_assets` has these columns: | Column | Description | | --- | --- | | `cloud` | Cloud provider (`aws`, `azure`, `gcp`, `k8s`, etc.) | | `account` | Account or subscription ID | | `account_name` | Human-readable account name | | `name` | Resource name | | `region` | Cloud region or location | | `resource_type` | Source table name (e.g. `aws_ec2_instances`) | | `resource_type_label` | Human-readable resource type label | | `resource_category` | Category like Compute, Storage, Database, Networking | | `tags` | Resource tags as a `Map(String, String)` | | `supports_tags` | Whether this resource type supports tagging | Cost columns are also available when AWS Cost & Usage data is synced: | Column | Description | | --- | --- | | `cost_7d_unblended` | 7-day raw cost (before discounts and credits) | | `cost_30d_unblended` | 30-day raw cost (before discounts and credits) | The `cloud_assets` table is a good starting point for cross-cloud queries. For example, finding all resources in a specific account across all clouds: ```sql SELECT cloud, resource_type, name, region FROM cloud_assets WHERE account = '123456789012' ORDER BY cloud, resource_type ``` When you need the full resource details (like an EC2 instance's security groups or an S3 bucket's encryption settings), query the integration-specific table directly. ## System Columns Every table in CloudQuery Platform includes system columns prefixed with `_cq_`. These track sync metadata and enable the platform's deduplication and consistency features. | Column | Type | Description | | --- | --- | --- | | `_cq_id` | UUID | Unique identifier for each resource record | | `_cq_parent_id` | UUID (can be null) | Parent resource ID, for hierarchical relationships (e.g. a subnet belonging to a VPC) | | `_cq_sync_time` | DateTime64 | Timestamp of when this record was synced | | `_cq_source_name` | String | Name of the source integration (e.g. `aws`, `gcp`) | | `_cq_sync_group_id` | UInt64 | Groups all records from a single sync run together | You don't normally need to query these columns directly, but they're useful for debugging sync issues or understanding when data was last refreshed. ## Custom Columns You can extend the `cloud_assets` schema with [custom columns](/platform/advanced-topics/custom-columns) - computed fields defined by ClickHouse SQL expressions. These are useful for extracting values from tags, normalizing data, or adding organization-specific metadata. For example, you could create a custom column that extracts a team owner from tags: ```sql tags['team'] ``` Custom columns appear alongside native columns in the Asset Inventory and are indexed for search. ## Next Steps - [Filters & Queries](/platform/core-concepts/filters-and-queries) - search and query your synced data - [Asset Inventory](/platform/features/asset-inventory) - browse resources using the `cloud_assets` table - [SQL Console](/platform/features/sql-console) - run ClickHouse SQL queries against any table - [Understanding Platform Views](/platform/advanced-topics/understanding-platform-views) - how staging tables, views, and snapshots work under the hood - [Custom Columns](/platform/advanced-topics/custom-columns) - add computed columns to `cloud_assets` - [Syncs](/platform/core-concepts/syncs) - how data gets into these tables --- Source: https://www.cloudquery.io/docs/platform/core-concepts/filters-and-queries # Filters & Queries Once your cloud data is synced to CloudQuery Platform, there are two ways to find what you're looking for. **Filters** give you a fast search bar in the Asset Inventory - type an IP address, instance ID, or structured query and get results across all your clouds. **Queries** give you full ClickHouse SQL in the SQL Console for joins, aggregations, and anything filters can't express. ## Filters The [Asset Inventory](/platform/features/asset-inventory) provides filters for finding resources by any attribute and an input for a full text search. To filter resources by their properties, click the **Filter by** and select one of the properties, an operator, and the available values. You can add additional filters and change the logical operator (`AND`/`OR`) between them by clicking the logical operator in the input box. ![Combined filter with a specified account name and region](/images/platform/asset-inventory/filter.png) To perform a full-text search, type your search query in the designated input. ![full text search input](/images/platform/asset-inventory/search.png) The full text search input takes an arbitrary string and searches for its existence in the cloud resources and their metadata (including their JSON columns). You can use logical operators in the full text search input as well. Use backslash `\` to escape special characters that have other meaning, such as asterisk `*` . Example full text search queries: - `"picture-service"`: finds all resources with the "picture-service" in one of the indexed columns. - `"picture-service" OR "auth-service"`: finds all resources with the "picture-service" or "auth-service" in one of the indexed columns. - `"test" AND NOT "latest"`: finds all resources with "test" but not containing "latest". ### Saved filters Save filters for reuse by clicking **Save** after entering a filter expression. Saved filters can be: - Reapplied from the saved filters panel in the Asset Inventory - Used as scopes when creating [policies](/platform/features/policies) - Managed via the [REST API](https://platform-multi-tenant-api-docs.cloudquery.io/) ## Queries Queries are ClickHouse SQL statements you can run in the [SQL Console](/platform/features/sql-console). Queries can perform joins, aggregations, and complex conditions that go beyond what filters support. For example, find all unassigned EC2 images by joining the `aws_ec2_images` table with the `aws_ec2_instances` table: ```sql SELECT img.image_id, inst.image_id as ec2_image, -- should all be NULL to show unassigned img.name, img.creation_date, img.state, img.tags FROM aws_ec2_images img LEFT JOIN aws_ec2_instances inst ON inst.image_id = img.image_id WHERE inst.image_id IS NULL -- AMIs not associated with any EC2 instance AND img.state = 'available' AND img.creation_date <= NOW() - INTERVAL 30 DAY ORDER BY img.creation_date ASC; ``` Queries use [ClickHouse SQL syntax](https://clickhouse.com/docs/en/sql-reference). For more on saved queries, tagging, and alert integration, see the [SQL Console](/platform/features/sql-console) documentation. Insights uses the same filtering model to help you triage security, compliance, and cost findings. See [Insights](/platform/features/insights). ## Next Steps - [Asset Inventory](/platform/features/asset-inventory) - browse, filter, and search your synced cloud resources - [SQL Console](/platform/features/sql-console) - run ClickHouse SQL queries against your synced data - [Policies](/platform/features/policies) - use saved filters as scopes for compliance policies - [Alerts](/platform/features/alerts) - trigger alerts based on query results - [Query Examples](/platform/features/sql-console/query-examples) - security, compliance, and FinOps query examples - [AI Query Writer](/platform/features/sql-console/ai-query-writer) - generate SQL queries using natural language --- Source: https://www.cloudquery.io/docs/platform/core-concepts/hub-directory # Integration Directory Browse all available integrations and reports on the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source). Each link below goes to the relevant Hub documentation page for that integration or report. ## Source Integrations Source integrations connect CloudQuery to your cloud providers, security tools, and SaaS platforms to sync resource data into the Asset Inventory. [Browse all source integrations →](https://www.cloudquery.io/hub/plugins/source) ### Cloud Infrastructure | Integration | Description | Hub Docs | | ------------------------ | ------------------------------------------------------ | ---------------------------------------------------------------------------------------- | | AWS | EC2, S3, IAM, Lambda, RDS, and 300+ other AWS services | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs) | | AWS Cost & Usage Reports | AWS billing and cost data from CUR | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/awscur/latest/docs) | | GCP | Compute, Storage, IAM, BigQuery, GKE, and more | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs) | | Azure | VMs, Storage, Active Directory, AKS, and more | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/azure/latest/docs) | | Kubernetes | Cluster resources, workloads, RBAC, and networking | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) | | Oracle Cloud | Oracle Cloud Infrastructure resources | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/oracle/latest/docs) | | DigitalOcean | Droplets, spaces, networking, and more | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/digitalocean/latest/docs) | ### Security & Identity | Integration | Description | Hub Docs | | ---------------- | ------------------------------------------- | -------------------------------------------------------------------------------------------- | | Okta | Identity and access visibility | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/okta/latest/docs) | | Entra ID | Microsoft Azure Active Directory / Entra ID | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/entraid/latest/docs) | | CrowdStrike | Endpoint security data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/crowdstrike/latest/docs) | | SentinelOne | Endpoint detection and response data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/sentinelone/latest/docs) | | Wiz | Cloud security posture data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/wiz/latest/docs) | | Palo Alto Cortex | Palo Alto Networks Cortex data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/palo-alto-cortex/latest/docs) | | Snyk | Vulnerability and dependency data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/snyk/latest/docs) | | Cloudflare | DNS, firewall, and network data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/cloudflare/latest/docs) | ### Engineering & DevOps | Integration | Description | Hub Docs | | ----------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | GitHub | Repositories, teams, members, actions, and security advisories | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/github/latest/docs) | | GitLab | Projects, merge requests, pipelines, and users | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/gitlab/latest/docs) | | Backstage | Software catalog and developer portal data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/backstage/latest/docs) | | Datadog | Monitoring data, dashboards, and alerts | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/datadog/latest/docs) | | PagerDuty | Incident, on-call schedule, and service data | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/pagerduty/latest/docs) | --- ## Destination Integrations Destination integrations define where CloudQuery writes your synced data. [Browse all destination integrations →](https://www.cloudquery.io/hub/plugins/destination) ### Data Warehouses & Lakes | Integration | Description | Hub Docs | | ----------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | PostgreSQL | SQL database for joins, queries, and compliance reporting | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/postgresql/latest/docs) | | BigQuery | Google Cloud analytics and AI | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/bigquery/latest/docs) | | Snowflake | Enterprise cloud data warehouse | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/snowflake/latest/docs) | | Databricks | Unified analytics and AI platform | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/databricks/latest/docs) | | ClickHouse | Column-oriented analytics database | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/clickhouse/latest/docs) | | DuckDB | In-process analytical database | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/duckdb/latest/docs) | ### Storage & Files | Integration | Description | Hub Docs | | ------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------- | | File (S3/GCS/Local) | CSV, JSON, or Parquet files in object stores or local disk | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/file/latest/docs) | | GCS | Google Cloud Storage | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/gcs/latest/docs) | | Azure Blob | Azure Blob Storage | [Docs](https://www.cloudquery.io/hub/plugins/destination/cloudquery/azblob/latest/docs) | --- ## Reports Pre-built reports for visualizing security, compliance, cost, and asset inventory data in CloudQuery Platform. [Browse all reports →](https://www.cloudquery.io/hub/reports) ### Asset Inventory & Multi-Cloud | Report | Description | Hub | | ----------------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------- | | Multi-cloud Asset Inventory | Overview of assets across AWS, GCP, and Azure | [View](https://www.cloudquery.io/hub/reports/multi-cloud-asset-inventory) | | Asset End of Life | Track resources approaching deprecation or end-of-life | [View](https://www.cloudquery.io/hub/reports/assets-end-of-life) | | Tags Overview | Tagged vs untagged resources across all cloud providers | [View](https://www.cloudquery.io/hub/reports/tags-overview) | | IP Address Report | Public IP addresses across your cloud infrastructure | [View](https://www.cloudquery.io/hub/reports/ip-address-report) | | Kubernetes & Container Report | Cluster distribution, node counts, pods, and container images | [View](https://www.cloudquery.io/hub/reports/kubernetes-container-report) | ### Security & Compliance | Report | Description | Hub | | -------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------- | | Identity and Access Overview | User account security across AWS, GCP, and Azure | [View](https://www.cloudquery.io/hub/reports/identity-and-access-overview) | | User Access Review | User accounts grouped by integration | [View](https://www.cloudquery.io/hub/reports/user-access-review) | | AWS Security Best Practices | Common AWS misconfigurations and security gaps | [View](https://www.cloudquery.io/hub/reports/aws-security-best-practices) | | AWS Well-Architected | Public resources, MFA adoption, GuardDuty findings, and more | [View](https://www.cloudquery.io/hub/reports/aws-well-architected) | | AWS Security Hub Overview | Open and resolved findings by severity | [View](https://www.cloudquery.io/hub/reports/aws-security-hub-overview) | | AWS Cloud Security: Data Overview | RDS encryption, accessibility, and backup status | [View](https://www.cloudquery.io/hub/reports/aws-cloud-security-data-overview) | | AWS Secrets | Secrets without rotation, unused secrets by account | [View](https://www.cloudquery.io/hub/reports/aws-secrets) | | AWS CloudTrail Log Detections | API call patterns, failed logins, IAM policy changes | [View](https://www.cloudquery.io/hub/reports/aws-cloudtrail-log-detections) | | AWS Backup Health | Backup compliance for EC2, RDS, S3, and DynamoDB | [View](https://www.cloudquery.io/hub/reports/aws-backup-health) | | AWS VPC Details | Instances, routing, and public/private network topology | [View](https://www.cloudquery.io/hub/reports/aws-vpc-details) | | Azure: Attack Surface Security Posture | VM access rules, public databases, and storage exposure | [View](https://www.cloudquery.io/hub/reports/azure-attack-surface-security-posture) | | Service Quota and Limits Report | Current usage vs. service quotas across AWS | [View](https://www.cloudquery.io/hub/reports/service-quota-and-limits-report) | ### Cost & FinOps | Report | Description | Hub | | ------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------- | | AWS Cost and Usage | Cost by service, region, compute, storage, and network | [View](https://www.cloudquery.io/hub/reports/aws-cost-and-usage) | | Cost Saving Opportunities | Idle, unattached, untagged, and oversized resources | [View](https://www.cloudquery.io/hub/reports/cost-saving-opportunities) | ### Engineering & Developer Tools | Report | Description | Hub | | -------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------ | | GitHub Repository Insights | PRs, issues, Dependabot alerts, and code scanning | [View](https://www.cloudquery.io/hub/reports/github-repository-insights) | | GitHub Security Insights | Dependabot trends, unprotected branches, PRs without review | [View](https://www.cloudquery.io/hub/reports/github-security-insights) | --- ## Next Steps - [Integration Guides](/platform/integration-guides/overview): Step-by-step setup guides for AWS, GCP, Azure, GitHub, and Kubernetes - [Asset Inventory](/platform/features/asset-inventory): Browse and filter synced resources - [SQL Console](/platform/features/sql-console): Query synced data with ClickHouse SQL - [Reports](/platform/features/reports): Pre-built dashboards for security and compliance --- Source: https://www.cloudquery.io/docs/platform/core-concepts/integrations # Integrations Integrations are how CloudQuery Platform connects to your cloud providers, SaaS apps, and destination databases. When you set up a sync on the platform, you're configuring a source integration (where data comes from) and one or more destination integrations (where data goes). The platform handles the connection lifecycle, scheduling, and error handling for you. CloudQuery brings together data from many sources through an integration-based architecture, with source, transformer and destination integrations acting as independent components. For the full technical details on integration architecture, gRPC communication, and building custom integrations, see the [CLI integration docs](/cli/core-concepts/integrations). ``` Sources Destinations ┌──────────┐ ┌──────────────────┐ │ AWS │──┐ ┌──│ S3 → ClickHouse │ (default) ├──────────┤ │ ┌──────────┐ │ ├──────────────────┤ │ GCP │──┼─│ Sync │─┘ │ S3 │ ├──────────┤ │ └──────────┘ └──────────────────┘ │ Azure │──┤ ├──────────┤ │ │ GitHub │──┘ ├──────────┤ │ 70+ │ │ more │ └──────────┘ ``` A [sync](/platform/core-concepts/syncs) connects one or more source integrations to one or more destination integrations, forming a data pipeline. Browse all available integrations on the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source). ## Source Integrations Source integrations connect to your cloud providers, SaaS applications, and APIs to extract configuration and asset data. Each source integration: - Defines the schema (tables) for the resources it syncs. - Authenticates with the supported API, SaaS service and/or cloud provider. - Extracts data from the supported APIs and transforms them into the defined schema. - Sends the data for further processing by the rest of the pipeline. ### Browse Source Integrations Browse the 70+ source integrations supported by the platform below. Full configuration reference and supported tables for each integration live on the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source). You can jump directly to any integration below, or see the [Integration Directory](/platform/core-concepts/hub-directory) for a complete organized listing. **Cloud Infrastructure** | Integration | Hub Docs | | ------------------------ | ---------------------------------------------------------------------------------------- | | AWS | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs) | | AWS Cost & Usage Reports | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/awscur/latest/docs) | | GCP | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs) | | Azure | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/azure/latest/docs) | | Kubernetes | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) | | Oracle Cloud | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/oracle/latest/docs) | | DigitalOcean | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/digitalocean/latest/docs) | **Security & Identity** | Integration | Hub Docs | | ---------------- | -------------------------------------------------------------------------------------------- | | Okta | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/okta/latest/docs) | | Entra ID | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/entraid/latest/docs) | | CrowdStrike | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/crowdstrike/latest/docs) | | SentinelOne | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/sentinelone/latest/docs) | | Wiz | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/wiz/latest/docs) | | Palo Alto Cortex | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/palo-alto-cortex/latest/docs) | | Snyk | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/snyk/latest/docs) | | Cloudflare | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/cloudflare/latest/docs) | **Engineering & DevOps** | Integration | Hub Docs | | ----------- | ------------------------------------------------------------------------------------- | | GitHub | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/github/latest/docs) | | GitLab | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/gitlab/latest/docs) | | Backstage | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/backstage/latest/docs) | | Datadog | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/datadog/latest/docs) | | PagerDuty | [Docs](https://www.cloudquery.io/hub/plugins/source/cloudquery/pagerduty/latest/docs) | [Browse all 70+ source integrations →](https://www.cloudquery.io/hub/plugins/source) ### Setting Up Source Integrations The platform provides guided setup for the most popular integrations: - [AWS (Guided Setup)](/platform/integration-guides/aws-onboarding-wizard): automated IAM role creation with CloudFormation - [AWS (Manual Setup)](/platform/integration-guides/setting-up-an-aws-integration): manual IAM role configuration - [GCP](/platform/integration-guides/setting-up-a-gcp-integration): service account and organization-wide access - [Azure](/platform/integration-guides/setting-up-an-azure-integration): service principal configuration - [GitHub](/platform/integration-guides/setting-up-a-github-integration): personal access token setup - [Kubernetes](/platform/integration-guides/setting-up-a-k8s-integration): cluster access via kubeconfig For any integration not listed above, use the [General Integration Setup Guide](/platform/integration-guides/general-integration-setup-guide). ## Destinations Destination integrations receive data from source integrations and write it to a database, data warehouse, message queue, or file storage system. ### Default Destination: S3 → ClickHouse When you create a sync on CloudQuery Platform, data is first written to S3 and then processed through a data pipeline into the built-in ClickHouse database. This powers platform features like the [Asset Inventory](/platform/features/asset-inventory), [SQL Console](/platform/features/sql-console), [Policies](/platform/features/policies), and [Reports](/platform/features/reports). ### Additional Destinations You can also sync data to additional destinations alongside the default ClickHouse destination. To set up a new destination, see the [General Destination Setup Guide](/platform/integration-guides/general-destination-setup-guide). Popular additional destinations include PostgreSQL, BigQuery, and Snowflake. Platform features like Asset Inventory and SQL Console depend on the default ClickHouse destination. Additional destinations serve as auxiliary targets for your data. ## Transformers Transformers are available only with the CloudQuery CLI Transformers are optional components that sit in between the source and destination. They allow you to modify data before it is written to the database, such as modifying the table name, obfuscating secrets, or adding additional columns. See the [CLI transformer docs](/cli/integrations/transformers) for configuration details and available transformers. **Using the CLI instead?** See [CLI Integrations](/cli/core-concepts/integrations) for the self-hosted configuration reference and architecture details. ## Next Steps - [Syncs](/platform/core-concepts/syncs) - how the platform manages syncs, write modes, and table views - [Setting up a sync](/platform/syncs/setting-up-a-sync) - configure your first sync on the platform - [Quickstart](/platform/quickstart/your-first-sync) - end-to-end walkthrough from account creation to querying data - [Integration guides](/platform/integration-guides) - step-by-step setup for AWS, GCP, Azure, GitHub, Kubernetes, and more - [General integration setup guide](/platform/integration-guides/general-integration-setup-guide) - add any source integration to the platform - [General destination setup guide](/platform/integration-guides/general-destination-setup-guide) - add a custom destination beyond the default ClickHouse - [Browse source integrations](https://www.cloudquery.io/hub/plugins/source) - find integrations for your cloud providers and SaaS apps --- Source: https://www.cloudquery.io/docs/platform/core-concepts/syncs # Syncs A CloudQuery sync fetches data from a source [integration](/platform/core-concepts/integrations) and delivers it to a destination. For example, you can sync [AWS](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs) data to the built-in ClickHouse database, or sync [GCP](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs) data to S3. ``` Sync Source ───────> schedule ─ tables ───────> S3 → ClickHouse (AWS) ``` You can also sync data to additional destinations like PostgreSQL, BigQuery, or Snowflake simultaneously. For CLI-specific sync configuration, write modes, and managing incremental state manually, see the [CLI sync docs](/cli/core-concepts/syncs). The platform delivers data to the destination as soon as the source produces it. Integrations may batch writes for performance reasons, but data generally arrives at the destination as the sync progresses. ## What the Platform Manages for You When syncing to the default S3 → ClickHouse destination, the platform handles write mode, incremental state, and table management automatically. You configure the schedule, select your source integration, and choose one or more destinations (or use the default S3 → ClickHouse). The platform takes care of the rest. Specifically: - **Write mode** is configured at the destination level, not per-sync. The default destination uses append mode. The platform creates views on top of the raw tables so you always query the latest snapshot of your data. - **Incremental table state** (cursors and backends) is managed automatically. You don't need to configure or maintain state backends. - **Table prefixing and views** - synced tables are stored with a `raw_` prefix, and the platform creates queryable views (e.g. `cloud_assets`) for use in the [Asset Inventory](/platform/features/asset-inventory), [SQL Console](/platform/features/sql-console), and [Policies](/platform/features/policies). See [Data Model](/platform/core-concepts/data-model) for how tables and views are organized. The sections below explain how write modes and incremental tables work under the hood. To add destinations like PostgreSQL or BigQuery, see the [General Destination Setup Guide](/platform/integration-guides/general-destination-setup-guide). ## Table Sync Modes Table syncs come in two flavors: full and incremental. A single sync can combine both types, and which type is used for a particular table depends on the table definition. This is indicated in the table's documentation in the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source). ### Full Table Syncs This is the normal mode of operation for most tables. Every sync fetches a full snapshot from the corresponding APIs. How that data is written to the destination depends on the destination's write mode: - **append** - new rows are added alongside existing data from previous syncs. - **overwrite** - new data replaces existing data, but stale rows from previous syncs are kept. - **overwrite-delete-stale** - new data replaces existing data, and rows from previous syncs that are no longer present are deleted at the end of the sync. The default S3 → ClickHouse destination only supports append mode. The platform creates views to surface the latest data. ### Incremental Table Syncs Some APIs lend themselves to being synced incrementally. Rather than fetch all past data on every sync, an incremental table only fetches data that has changed since the last sync. This is done by storing metadata in a state **backend**. The metadata is known as a **cursor**, and it marks where the last sync ended so the next sync can resume from the same point. Incremental syncs are more efficient than full syncs, especially for tables with large amounts of data, because only the changed subset needs to be retrieved. Incremental tables are marked as "incremental" in [integration table documentation](https://www.cloudquery.io/hub/plugins/source), along with which columns are used for the cursor value. On CloudQuery Platform, the state backend is managed for you. For CLI users managing their own state, see [Managing Incremental Tables](/cli/advanced/managing-incremental-tables). ## What Happens After a Sync Once data lands in your destination, you can: - **Browse resources** in the [Asset Inventory](/platform/features/asset-inventory): filter and search across all synced cloud assets. - **Query with SQL** in the [SQL Console](/platform/features/sql-console): run ad-hoc queries, use the [AI Query Writer](/platform/features/sql-console/ai-query-writer) to generate them from natural language, or ask the [AI Assistant](/platform/features/ai-assistant) questions directly from the Platform Home page. - **Enforce policies** with [Policies](/platform/features/policies): define SQL-based checks that run continuously against your synced data. - **Build reports** with [Reports](/platform/features/reports) and [Alerts](/platform/features/alerts): track metrics over time and get notified when conditions are met. **Using the CLI instead?** See [CLI Syncs](/cli/core-concepts/syncs) for the self-hosted sync configuration, including parallel execution and state backend management. ## Next Steps - [Setting up a sync](/platform/syncs/setting-up-a-sync) - configure and schedule syncs on the platform - [Monitoring sync status](/platform/syncs/monitoring-sync-status) - track sync progress and troubleshoot failures - [Integrations](/platform/core-concepts/integrations) - how source, destination, and transformer integrations work - [Filters & Queries](/platform/core-concepts/filters-and-queries) - search synced data in Asset Inventory and SQL Console - [Asset Inventory](/platform/features/asset-inventory) - browse and search your synced cloud resources - [Understanding Platform Views](/platform/advanced-topics/understanding-platform-views) - how the platform creates views from raw synced tables - [Performance Tuning](/platform/advanced-topics/performance-tuning) - optimize sync performance on the platform - Browse available integrations on the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source) --- Source: https://www.cloudquery.io/docs/platform/features/ai-assistant/custom-context # Custom Context Custom context lets you give the AI Assistant organization-specific knowledge about your cloud environment. This knowledge is injected at the start of a conversation, so the assistant can reason about things that aren't visible in your resource data, like how your teams are organized, what your tagging conventions mean, or which environments map to which accounts. Custom contexts are created by organization administrators and are available to all users. ## What to put in a custom context Custom context is a plain-text document. Useful content includes: - **Environment structure**: which AWS accounts or GCP projects belong to production, staging, or dev - **Tagging conventions**: what tags like `team`, `cost-center`, or `env` mean in your organization - **Naming patterns**: how your teams name VPCs, buckets, or service accounts - **Ownership information**: which team owns which set of resources - **Business context**: compliance requirements or policies that affect how you interpret your infrastructure **Example:** ``` Our AWS account IDs map as follows: - 123456789012: production - 234567890123: staging - 345678901234: development All production resources are tagged with env=prod. The "team" tag maps to our Slack channels. Security team owns all resources tagged with team=security. ``` ## Creating and managing contexts 1. Open the AI Assistant on the Platform Home page 2. Click **Add Context**, then **Manage Context documents** 3. Create a new context document, give it a name, and paste your content 4. Save the document To delete or edit a context, return to **Manage Context documents** from the same menu. ## Auto-include Each context document has an **Auto-include** toggle. When enabled, the document is automatically added to every new conversation for all users in the organization. This is the recommended setting for shared organizational knowledge that should always be available to the assistant. You can also select context documents manually at the start of a conversation if you want to include them selectively. ## Timing Custom context cannot be added to a conversation after the assistant has sent its first response. If you forget to include a context, start a new conversation. ## Limits | Limit | Value | | --- | --- | | Custom contexts per organization | 20 | | Maximum size per context | 4,096 bytes (~4 KB) | | Contexts per conversation | 10 | Keep context documents focused and concise. The byte limit is intentional; a document that tries to describe everything will use token budget that the assistant needs for reasoning. ## Next steps - [AI Assistant Overview](/platform/features/ai-assistant): How the assistant works and how to configure data access modes --- Source: https://www.cloudquery.io/docs/platform/features/ai-assistant # AI Assistant The AI Assistant is the Home page of CloudQuery Platform. It lets you ask questions about your cloud infrastructure in plain language and get answers, without writing SQL or knowing which tables to query. Unlike the [AI Query Writer](/platform/features/sql-console/ai-query-writer) in the SQL Console, which is scoped to generating SQL, the AI Assistant is a full conversational agent. It can explore your schema, reason across multiple tables, and, depending on your organization's settings, execute queries and return results directly in the chat. ## Data access modes The AI Assistant operates in one of two modes, configured per organization: | Mode | What the assistant can do | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | **Data Privacy** (default) | Reads table names and column definitions only. Returns SQL queries for you to run manually or open in the SQL Console. | | **Full Access** | Executes queries on your behalf and returns results directly in the chat. | Data Privacy mode is the default. Every organization starts with it enabled, and it cannot be bypassed by individual users. ### Enabling Full Access Full Access mode is controlled by organization administrators: 1. Go to **Organization Settings → Platform Settings** 2. Find the **AI assistant data access** section 3. Toggle **Enable AI agent data access** on 4. The setting applies to all users in the organization immediately Full Access mode allows the AI Assistant to execute SQL queries against your CloudQuery database and include results in its responses. Enable it only if your organization's policies permit AI systems to read live infrastructure data. ## How to use it Open the Platform Home page and type your question in the chat input. The assistant searches your schema, figures out which tables are relevant, and responds with either a direct answer (Full Access mode) or an executable SQL query (Data Privacy mode). **Example prompts:** - `Show count of virtual machines across all cloud environments` - `Which S3 buckets are publicly accessible?` - `List IAM users that have not rotated their access keys in the last 90 days` - `What EC2 instances are running in production?` You can follow up in the same conversation to refine results, narrow scope, or ask clarifying questions. The assistant retains context across messages within a session. If you ask for a SQL query directly, the assistant will generate one. You can open it in the [SQL Console](/platform/features/sql-console), save it as a policy, or run it as-is. You can also use the AI assistant to summarize [Insights](/platform/features/insights) and create or edit [Reports](/platform/features/reports). ## Rate limits The following limits apply per organization: | Limit | Value | | ------------------- | --------- | | Requests per minute | 10 | | Tokens per request | 500,000 | | Tokens per day | 5,000,000 | These limits are subject to change during the initial launch period. ## What the AI Assistant cannot do - **No web access.** The assistant cannot fetch external URLs, load documentation, or analyze websites. - **No off-topic conversations.** The assistant has guardrails that keep it focused on cloud infrastructure and CloudQuery-related topics. Unrelated requests will be declined. - **No persistent memory.** Each new conversation starts fresh. Use [Custom Context](/platform/features/ai-assistant/custom-context) to carry organization-specific knowledge into every session automatically. ## Data privacy In the default Data Privacy mode, the assistant never reads or transmits row data from your CloudQuery database. It only sees table names and column definitions. | The assistant can see | The assistant cannot see | | --------------------------------------- | -------------------------------------------- | | Table names in your CloudQuery database | Row data of any kind | | Column names and their data types | Values stored in any column or field | | SQL syntax validation results | Credentials, connection strings, or API keys | In Full Access mode, the assistant executes SQL queries and reads result sets. Only the rows returned by those queries are processed; the assistant does not have unrestricted access to your CloudQuery database. The assistant is powered by [Claude](https://www.anthropic.com/claude) (Anthropic) running on AWS Bedrock. AWS Bedrock does not use customer data to train or improve foundation models. Your queries, schema information, and query results are not retained for model training purposes. ## FAQ ### Which AI model does this use? [Claude Sonnet 4.5 and Haiku 4.5](https://www.anthropic.com/claude) (Anthropic), running on AWS Bedrock. Sonnet 4.5 handles the main agent loop; Haiku 4.5 handles a lightweight scope-check that runs before every request. ### How is this different from the AI Query Writer in the SQL Console? The [AI Query Writer](/platform/features/sql-console/ai-query-writer) is a focused tool inside the SQL Console that always operates in schema-only mode and always returns a SQL query. The AI Assistant is a broader conversational agent that lives on the Platform Home page, supports both Data Privacy and Full Access modes, and can answer questions directly without requiring you to run SQL yourself. ### Can I use both? Yes. They serve different workflows. Use the AI Assistant for exploration and questions; use the SQL Console and AI Query Writer when you want to author, save, and reuse queries. ### Is my data used to train the model? No. AWS Bedrock does not use customer data to train or improve foundation models. ### Who can change the data access mode? Only organization administrators. The setting is at **Organization Settings → Platform Settings → AI Assistant Data Access** and applies to all users in the organization. ## Next steps - [Custom Context](/platform/features/ai-assistant/custom-context): Give the assistant knowledge about your specific environment - [SQL Console](/platform/features/sql-console): Save and run the queries the assistant generates - [AI Query Writer](/platform/features/sql-console/ai-query-writer): The SQL-focused assistant inside the SQL Console - [MCP Server](/platform/features/mcp-server): Connect AI tools like Claude Desktop or Cursor directly to your CloudQuery database --- Source: https://www.cloudquery.io/docs/platform/features/alerts # Alerts Alerts functionality is being consolidated into [Policies](/platform/features/policies) in an upcoming release. Alerts will continue to work as documented here until then. Alerts send notifications when conditions you define are triggered. Each alert is based on a SQL query. If the query returns any rows after a sync, the alert fires and sends notifications to the configured [destinations](/platform/features/notification-destinations). You can notify a Slack channel and open a Jira ticket at the same time. CloudQuery Platform alerts are not meant to replace your incident management system. They feed events into your existing tools when your infrastructure does not comply with the rules you set. ## How Alerts Work Alerts are a feature of [SQL queries](/platform/core-concepts/filters-and-queries#queries). You configure a query to trigger an alert with a severity, a message, and one or more [notification destinations](/platform/features/notification-destinations). The platform evaluates alert queries after every sync. If a query returns any rows, the platform triggers the alert and sends notifications to the configured destinations. To review and triage the findings that trigger alerts, use [Insights](/platform/features/insights). The alert triggers only once per violation cycle. For the alert to trigger again, the query must return zero rows at least once (resetting the alert to "inactive"), then return rows again on a subsequent evaluation. Notifications are sent on every state change. When a query first returns rows, a "triggered" notification is sent. If the query continues to return rows on subsequent evaluations, no additional notification is sent. When the query returns zero rows, the alert resets to "inactive" and a resolution notification is sent. ## Configuring an Alert 1. Go to the [SQL Console](/platform/features/sql-console) and write a SQL query, or load a saved query. For example, to get notified about unattached EBS volumes: ```sql SELECT * FROM aws_ec2_ebs_volumes WHERE attachments='[]'; ``` 2. Click **Configure Alert** to open the alert configuration dialog.
    ![Alert configuration dialog opened from the SQL Console with options for query title and notification settings](/images/platform/alerts-1.png)
    3. Enter a **Query title**. This is used as the alert message. You can change it later.
    ![Configure alert title](/images/platform/alerts-2.png)
    4. Select one or more [notification destinations](/platform/features/notification-destinations). If none exist yet, click **Add notification destination** to create one.
    ![Configure notification destinations](/images/platform/alerts-3.png)
    5. Click **Save alert**. ## Configuring Notification Destinations For full details on creating and managing Slack and webhook destinations, see the [Notification Destinations](/platform/features/notification-destinations) page. When configuring a webhook destination for alerts, you can use the following placeholder variables in the request body: | Variable | Description | | ---------------------- | ------------------------------------------------ | | `{{query_name}}` | The name of the query the alert is configured on | | `{{query_url}}` | Direct URL to the query in the SQL Console | | `{{alert_status}}` | Current alert state: `triggered` or `inactive` | | `{{alert_severity}}` | The severity level configured for the alert | | `{{alert_message}}` | The custom message configured for the alert | | `{{alert_violations}}` | The number of rows returned by the query |
    ![Configure notification destination](/images/platform/alerts-4.png)
    You can test the destination using the **Send test notification** button. Test notifications send the body as-is, without replacing placeholders. Here's an example of a CloudQuery alert delivered to Slack, showing an unencrypted S3 bucket with resource metadata: ![Slack notification from CloudQuery showing an alert for an unencrypted S3 bucket with account ID, region, and resource details](/images/platform/alerts-slack-example.png) ## Editing and Managing Alerts To edit a saved alert, go to the [SQL Console](/platform/features/sql-console) and click the **Saved queries** button in the top right corner. Switch to **Saved queries with alerts** to see queries that have alerts configured.
    ![Saved queries panel showing saved SQL queries with configured alerts and a dropdown to edit alert settings](/images/platform/alerts-5.png)
    Use the dropdown menu on the right to edit the alert. You can change the alert message, severity, and destinations. You can also disable the alert so the query is not evaluated. ## Troubleshooting ### Notifications Are Not Being Sent Alerts do not send new notifications when they are already in the "triggered" state. The alert must reset to "inactive" (query returns zero rows) before it can trigger again. 1. Open the [SQL Console](/platform/features/sql-console) and run the alert query to check if it returns rows. 2. If it does, fix or update the query so it returns zero rows, then run a sync. Any sync with any integration triggers alert evaluation. 3. To isolate whether the issue is with CloudQuery or the receiving endpoint, add a test destination using [Webhook.site](https://webhook.site/). If the test destination receives notifications but your real destination does not, the issue is on the receiving end. Allow up to 10 minutes after a sync for alert evaluation and notification delivery. ### A Destination Cannot Parse the Request Body Make sure the `Content-Type` header matches what the receiving service expects. Most endpoints require `application/json`. ## Related Features - [Policies](/platform/features/policies): the next generation of detective controls, replacing alerts - [Notification Destinations](/platform/features/notification-destinations): configure Slack and webhook endpoints - [SQL Console](/platform/features/sql-console): write and test the SQL queries that power your alerts ## Programmatic access Alerts can be managed via the Platform API. See the [Platform API Reference](/platform/reference/api-reference) (`alerts` section) for endpoint details. ## Next Steps - [Notification Destinations](/platform/features/notification-destinations) - Configure Slack, Jira, and webhook notification targets - [Policies](/platform/features/policies) - Define compliance rules that trigger alerts - [SQL Console](/platform/features/sql-console) - Write the SQL queries that power your alerts --- Source: https://www.cloudquery.io/docs/platform/features/asset-inventory # Asset Inventory The Asset Inventory is a unified view of all your cloud resources across every connected integration. Resources from AWS, GCP, Azure, and any other synced provider appear in a single inventory with consistent schemas, searchable attributes, and organized categories. ## How It Works After you [set up integrations](/platform/core-concepts/integrations) and [run a sync](/platform/syncs/setting-up-a-sync), your cloud resources are normalized into a common schema and loaded into the Asset Inventory. The inventory updates after each sync, so it reflects the latest state of your infrastructure. ## Navigating the Inventory The Asset Inventory has three main areas: - **Left sidebar**: browse resources by category - **Main table**: view, sort, and filter resources - **Detail panel**: inspect individual resource attributes (opens when you click a row) ## Resource Categories The left sidebar organizes assets into categories: - **Compute**: EC2 instances, VMs, managed instance groups - **Storage**: S3 buckets, disks, blob storage - **Networking**: VPCs, subnets, load balancers, DNS - **Databases**: RDS, Cloud SQL, Cosmos DB - **Containers**: EKS, GKE, AKS clusters and workloads - **Identity**: IAM users, roles, service accounts - **Serverless**: Lambda functions, Cloud Functions - **Logging**: CloudTrail, audit logs - **Threat detection**: GuardDuty, Security Command Center - And more (Administration, Batch Processing, Queues, Endpoint Management, Vulnerability Management) Select **All Resources** to view everything, or select a category to filter to those resource types. Each category shows the total resource count. You can also search category names using the search field at the top of the sidebar. ## Search, Filters, Grouping The top part of the Asset Inventory allows you to search, filter, and group resources by their properties. To filter resources by their properties, click the **Filter by** and select one of the properties, an operator, and the available values. You can add additional filters and change the logical operator (`AND`/`OR`) between them by clicking the logical operator in the input box. ![Combined filter with a specified account name and region](/images/platform/asset-inventory/filter.png) To perform a full-text search, type your search query in the designated input. ![full text search input](/images/platform/asset-inventory/search.png) The full text search input takes an arbitrary string and searches for its existence in the cloud resources and their metadata (including their JSON columns). You can use logical operators in the full text search input as well. Use backslash `\` to escape special characters that have other meaning, such as asterisk `*` . Example full text search queries: - `"picture-service"`: finds all resources with the "picture-service" in one of the indexed columns. - `"picture-service" OR "auth-service"`: finds all resources with the "picture-service" or "auth-service" in one of the indexed columns. - `"test" AND NOT "latest"`: finds all resources with "test" but not containing "latest". ## Saved Filters Save frequently used filters for reuse. Saved filters can be shared with your team and reapplied from the saved filters panel. They are also available via the [REST API](https://platform-multi-tenant-api-docs.cloudquery.io/) and can be used as scopes when creating [policies](/platform/features/policies). ## Resource Details Click any row in the table to open the detail panel on the right side. The detail view has three tabs: - **Resource details**: the resource's cloud provider, name, type, region, CloudQuery ID, last sync time, tags, and custom column values - **Related resources**: resources connected to the selected asset based on table relationships, with the ability to filter by resource type - **Insights**: [Insights](/platform/features/insights) findings that affect this resource, so you can see security, cost, and compliance issues in context ## Customizing Columns Click the column settings icon to show or hide columns in the table. You can: - Toggle individual columns on or off - Show or hide all columns at once - Restore the default column set Default columns include: Cloud, Resource Type, Account, Name, Region, and Tags. [Custom columns](/platform/advanced-topics/custom-columns) appear alongside the defaults and can be toggled the same way. ## Exporting Data Use the export button to download the current filtered and sorted view as a file. ## Going Deeper with SQL For queries that go beyond the search syntax (joins across tables, aggregations, or complex conditions), use the [SQL Console](/platform/features/sql-console) to write ClickHouse SQL directly against your synced data. To analyze your inventory for security risks, compliance gaps, and cost optimization opportunities, use [Insights](/platform/features/insights). ## Related Features - [SQL Console](/platform/features/sql-console): run ClickHouse SQL queries against your synced data - [Policies](/platform/features/policies): define detective controls that continuously evaluate your resources - [Reports](/platform/features/reports): build visualizations and dashboards from your asset data - [Insights](/platform/features/insights): analyze your inventory for security risks, compliance gaps, and cost optimization - [Custom columns](/platform/advanced-topics/custom-columns): add computed or enrichment columns to the inventory ## Next Steps - [SQL Console](/platform/features/sql-console) - Write advanced SQL queries against your cloud data - [Policies](/platform/features/policies) - Create rules to monitor asset compliance - [Alerts](/platform/features/alerts) - Get notified when asset conditions are met --- Source: https://www.cloudquery.io/docs/platform/features/insights # Insights Insights surfaces security, compliance, and cost findings across your synced cloud infrastructure. Each insight identifies a category of resources that may require attention, with severity levels, affected resource counts, and actionable mitigation guidance. ## Sources and Generation Insights are generated asynchronously after each sync completes. When a sync lands new data for a supported table, the platform queues a job that evaluates the data against insight rules and joins findings to your asset inventory. The **Last synced** timestamp on each insight shows when it was last evaluated. ### Built-in Sources These activate automatically when you connect the corresponding integration: | Source | Cloud | Insight Categories | What It Surfaces | | ----------------------- | ----- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | **AWS Health** | AWS | Security, Availability, Maintenance, Cost, Account | Upcoming and open AWS Health events with severity based on required action and event type | | **AWS Security Hub** | AWS | Security | Security findings from Security Hub, including remediation guidance when available | | **AWS CUR Cost** | AWS | Cost | Resource-level cost data from Cost and Usage Reports, with 30-day and 7-day trends and spike detection | | **GCP Security Center** | GCP | Security | Active findings from Security Command Center (project and organization level) | | **Azure Advisor** | Azure | Security, Cost, Performance, Operational Excellence | Recommendations from Azure Advisor across all categories | Each source maps findings to resources in your asset inventory using ARN or resource ID matching, so you can see which specific assets are affected. ### Third-Party Sources Insights can also include findings from external security tools: - **Wiz**: data findings (PII detection, financial data exposure) and security findings ### Custom Sources You can define your own insight rules using [Policies](/platform/features/policies) for organization-specific checks. ## Triaging Insights A typical workflow after your first sync: 1. Open **Insights** from the sidebar. Findings are grouped by severity by default 2. Click **Add filter**, select **Severity**, and check **Critical** and **High** to focus on what matters most 3. Click a finding to open its detail page: check the **Evidence** panel to understand what was detected and the **Mitigation** panel for remediation steps 4. In the resources table, click an affected resource to open its detail drawer and inspect its attributes or check **Related Resources** for connected assets 5. Click **Saved filters** in the top-right corner, then **Save current filters** to save this combination for quick access later ## Insights List The main Insights page displays a table of all detected findings with the following columns: | Column | Description | | -------------------- | ------------------------------------------------------------------------- | | **Severity** | Low, Medium, High, or Critical | | **Finding** | Title describing the insight | | **Source Category** | Sub-category within the source (e.g., Policies, Data Findings) | | **Source** | Where the finding originated (e.g., AWS Health, Wiz, GCP Security Center) | | **Insight Category** | Category of the finding (e.g., Security, Cost, Compliance, PII) | | **Account** | The affected cloud account | | **Resource Type** | Types of resources involved | | **Resources** | Number of affected resources | All columns are sortable. Click any column header to sort ascending or descending. ### Filtering and Grouping The filter bar above the table lets you narrow results by any combination of: - **Severity**: Low, Medium, High, Critical - **Source**: the integration that generated the finding - **Source Category**: sub-category within the source - **Insight Category**: Security, Cost, Compliance, etc. - **Resource Type**: the type of cloud resource affected - **Account**: the cloud account - **Tags**: resource tags - **Ownership Tags**: one entry per tag key if you've configured [Resource Ownership](#configuring-resource-ownership) **Adding a filter:** Click **Add filter**, select a field from the left pane, then check one or more values in the right pane. Your selection appears immediately as a chip below the toolbar. **Editing a filter:** Click the value portion of any chip to reopen its value picker and toggle values inline. Click the **×** on a chip to remove that filter entirely. **Clearing all filters:** Click **Clear filters** to reset to the default (unfiltered) view. **Group By:** Use the **Group by** dropdown to group findings by severity, source, source category, insight category, resource type, or account. When you apply a filter, the grouping automatically shifts to the next ungrouped field: for example, filtering by severity switches grouping to source. Select **None** to show a flat list. ### Saved Filters Click **Saved filters** in the top-right corner of the page to open the saved filters drawer. From there you can apply, rename, or delete existing saved filters, or click **Save current filters** to save your current filter and group-by configuration. The save button is only enabled when at least one non-default filter is active. Saved filters are also available via the [Platform API](https://platform-api-docs.cloudquery.io/). ## Insight Detail Click any insight to open its detail page. The header shows the insight title, severity, affected resource count, and last synced timestamp. The **resources table** lists all affected resources. Use the search field to find resources by name, and use the dropdown filters to narrow by account, cloud provider, region, or resource type. Click any resource to open its detail drawer, which has three tabs: - **Resource Details**: cloud provider, name, type, region, tags, and custom column values - **Related Resources**: connected resources (e.g., an EC2 instance's security groups or attached volumes). You can search and click through to drill into nested relationships. Relationships are pre-configured for common resource types across AWS, GCP, and Azure. This tab also appears in the [Asset Inventory](/platform/features/asset-inventory). - **Insights**: other insights affecting this same resource Side panels on the detail page show additional context when available: - **Evidence**: what was detected and why it matters - **Mitigation**: steps to resolve the finding - **Additional Properties**: source-specific metadata such as aggregated cost values ## Configuring Resource Ownership Ownership tags let you filter and group insights by who owns the affected resources. Once configured, each tag appears as an "Owner: {tag}" filter in the Insights filter bar. See [Resource Ownership](/platform/production-deployment/resource-ownership) for setup instructions. ## Next Steps - [**Policies**](/platform/features/policies): create custom rules that generate insights specific to your organization - [**Alerts**](/platform/features/alerts): get notified when new findings match conditions you define - [**Asset Inventory**](/platform/features/asset-inventory): browse and search the resources that insights are evaluated against - [**Reports**](/platform/features/reports): build dashboards that visualize insight trends over time - [**SQL Console**](/platform/features/sql-console): write queries directly against your synced data for analysis beyond what insights surfaces --- Source: https://www.cloudquery.io/docs/platform/features/mcp-server # CloudQuery MCP Server The **CloudQuery MCP (Model Context Protocol) Server** lets you query your CloudQuery asset inventory using Claude Desktop, Cursor IDE, or any LLM tool that supports the [MCP protocol](https://modelcontextprotocol.io). To explore findings through the platform UI instead, see [Insights](/platform/features/insights). If you're not a CloudQuery Platform customer but you are a CLI user, you can still use the CloudQuery MCP server in CLI, PostgreSQL, or Snowflake Mode. This server supports four operating modes: - **CLI Mode** – for getting started with CloudQuery CLI and generating configuration files using natural language - **Platform Mode** – for CloudQuery Platform customers - **PostgreSQL Mode** – for CLI users syncing to a PostgreSQL destination - **Snowflake Mode** – for CLI users syncing to a Snowflake destination (Requires MCP server >= 1.8.0) --- ### Quick Start Download the [latest version](https://github.com/cloudquery/mcp-releases/releases/latest) of the MCP binary for your platform. Unzip the binary and ensure it's executable. ### Requirements #### CLI Mode The default mode and available **to all users**. ##### Tools | Tool | Description | | ------------------------------ | ------------------------------------------------------------------------------------- | | `cli-list-source-plugins` | List source integrations available on CloudQuery Hub (e.g. aws, gcp, azure, github). | | `cli-list-destination-plugins` | List destination integrations (e.g. PostgreSQL, BigQuery, Snowflake, ClickHouse). | | `cli-get-plugin-docs` | Get documentation for a source or destination integration (name and kind required). | | `cli-list-source-tables` | List all tables for a given source integration. | | `cli-get-cli-docs` | Get CloudQuery CLI documentation (e.g. login, sync). | #### Platform Mode This mode is for **CloudQuery Platform customers**. You’ll need: - Your **CloudQuery Platform API Key** - Your **deployment’s API URL** ##### Tools | Tool | Description | | ------------------------------ | ---------------------------------------------------------------------- | | `list-installed-plugins` | List integrations installed in the CloudQuery Platform deployment. | | `table-search-regex` | Search for tables by regex; includes the unified `cloud_assets` table. | | `table-schemas` | Get column definitions, types, and JSON schemas for given tables. | | `column-search` | Search for columns by regex across all tables. | | `known-good-queries` | List curated ClickHouse query examples (filter by name). | | `execute-clickhouse-sql-query` | Execute a ClickHouse SQL query against the asset inventory. | #### PostgreSQL Mode This mode is available to **all CloudQuery CLI users** who sync to a PostgreSQL destination. You'll need: - The **Connection String or DSN to the PostgreSQL database** ##### Tools | Tool | Description | | ----------------------------- | ----------------------------------------------------------------- | | `postgres-list-plugins` | List integrations present in the database from synced CloudQuery data. | | `postgres-table-search-regex` | Search for tables by regex (e.g. `aws_ec2.*`, `.*storage.*`). | | `postgres-table-schemas` | Get column definitions and types for given tables. | | `postgres-column-search` | Search for columns by regex across all tables. | | `execute-postgres-query` | Run a PostgreSQL SQL query against the database. | > The MCP server must be able to reach the PostgreSQL database from its host. #### Snowflake Mode (Requires MCP server >= 1.8.0) This mode is available to **all CloudQuery CLI users** who sync to a Snowflake destination. You'll need: - The **Snowflake connection string** in the format: `user:password@account/database/schema?warehouse=warehouse_name` ##### Tools | Tool | Description | | ------------------------------ | ----------------------------------------------------------------- | | `snowflake-list-plugins` | List integrations present in the database from synced CloudQuery data. | | `snowflake-table-search-regex` | Search for tables by regex (e.g. `aws_ec2.*`, `.*storage.*`). | | `snowflake-table-schemas` | Get column definitions and types for given tables. | | `snowflake-column-search` | Search for columns by regex across all tables. | | `execute-snowflake-query` | Run a Snowflake SQL query against the database. | > The MCP server must be able to reach the Snowflake database from its host. --- ### Environment Variables #### CLI Mode If no environment variables are set, the MCP server will default to CLI mode. #### PostgreSQL Mode Configure the following environment variables to enable PostgreSQL mode. - `POSTGRES_CONNECTION_STRING` - `postgres://user:password@host:port/database` > If no `search_path` is specified in the connection string, the MCP server automatically defaults to `public` schema. You can override this by explicitly setting `search_path` in your connection string (e.g., `postgres://user:password@host:port/database?search_path=myschema`). > The MCP server supports reading `.env` files. ##### Kerberos Authentication For Kerberos/GSSAPI authentication, include the appropriate parameters in the connection string: - `postgres://username@REALM.EXAMPLE.COM@host:port/database?krbsrvname=postgres` - `postgres://username@host:port/database?krbsrvname=postgres&gsslib=gssapi` #### Snowflake Mode (Requires MCP server >= 1.8.0) Configure the following environment variables to enable Snowflake mode. - `SNOWFLAKE_CONNECTION_STRING` - Snowflake connection string in the format: `user:password@account/database/schema?warehouse=warehouse_name` Example connection strings: - `user:password@account/database/schema` - Basic authentication - `user:password@account/database/schema?warehouse=COMPUTE_WH` - With warehouse specified - `user:password@account.region/database/schema?warehouse=COMPUTE_WH` - With region For more connection string options, see the [Snowflake Go Driver documentation](https://pkg.go.dev/github.com/snowflakedb/gosnowflake). > The MCP server supports reading `.env` files. #### Platform Mode Configure the following environment variables to enable Platform mode. - `CQ_PLATFORM_API_URL` - `https://your-deployment.cloudquery.io/api` - `CQ_PLATFORM_API_KEY` - Your CloudQuery Platform API key #### Optional - `CQAPI_LOG_LEVEL`: Log level (`debug`, `info`, `warn`, `error`, default: `info`) - `LOG_PRETTY`: Enable pretty console logging (auto-detected by default) - `LOG_COLOR`: Enable colored logging (auto-detected by default) You can place these in a `.env` file or export them in your shell. --- ### IDE Integration #### Claude Desktop Add the following configuration to: - macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` - Linux: `~/.config/Claude/claude_desktop_config.json` **CLI Mode** ```json { "mcpServers": { "cloudquery": { "command": "/absolute/path/to/cq-platform-mcp", "args": [], "env": {} } } } ``` **Platform Mode** ```json { "mcpServers": { "cloudquery": { "command": "/absolute/path/to/cq-platform-mcp", "args": [], "env": { "CQ_PLATFORM_API_KEY": "your_api_key", "CQ_PLATFORM_API_URL": "https://your-deployment.cloudquery.io/api" } } } } ``` **PostgreSQL Mode** ```json { "mcpServers": { "cloudquery": { "command": "/absolute/path/to/cq-platform-mcp", "args": [], "env": { "POSTGRES_CONNECTION_STRING": "postgres://user:pass@localhost:5432/db?sslmode=disable" } } } } ``` **Snowflake Mode (Requires MCP server >= 1.8.0)** ```json { "mcpServers": { "cloudquery": { "command": "/absolute/path/to/cq-platform-mcp", "args": [], "env": { "SNOWFLAKE_CONNECTION_STRING": "user:password@account/database/schema?warehouse=COMPUTE_WH" } } } } ``` > The `"command"` must be an **absolute path** to the binary. --- #### Cursor IDE 1. Open Cursor 2. Go to **Settings** → **Cursor Settings** → **Tools and Integrations** 3. Add a new MCP server configuration: **Platform Mode Example** ```json { "name": "cloudquery", "command": "/path/to/cq-platform-mcp", "args": [], "env": { "CQ_PLATFORM_API_KEY": "your_api_key", "CQ_PLATFORM_API_URL": "https://your-deployment.cloudquery.io/api" } } ``` **PostgreSQL Mode Example** ```json { "name": "cloudquery", "command": "/path/to/cq-platform-mcp", "args": [], "env": { "POSTGRES_CONNECTION_STRING": "postgres://user:password@localhost:5432/database?sslmode=disable" } } ``` **Snowflake Mode Example (Requires MCP server >= 1.8.0)** ```json { "name": "cloudquery", "command": "/path/to/cq-platform-mcp", "args": [], "env": { "SNOWFLAKE_CONNECTION_STRING": "user:password@account/database/schema?warehouse=COMPUTE_WH" } } ``` --- #### VS Code IDE Integration There are multiple ways to integrate MCP servers in VS Code, as described in the [VS Code MCP documentation](https://code.visualstudio.com/docs/copilot/chat/mcp-servers#_use-mcp-tools-in-agent-mode). 1. Open VS Code IDE 2. Go to `MCP: Open User Configuration` (can also be done at the workspace or folder level) via the keyboard shortcut `Ctrl+Shift+P` (Windows/Linux) or `Cmd+Shift+P` (Mac) 3. Add a new MCP server with the following configuration: **CLI Mode** ```json { "servers": { "CloudQuery": { "type": "stdio", "command": "/path/to/cq-platform-mcp", "env": {} } } } ``` **Platform Mode** ```json { "inputs":[ { "type":"promptString", "id":"cloudquery-platform-api-key", "description":"CloudQuery Platform API Key", "password":true }, { "type":"promptString", "id":"cloudquery-platform-api-url", "description":"CloudQuery Platform API URL", "password":false } ], "servers":{ "CloudQuery":{ "type":"stdio", "command": "/path/to/cq-platform-mcp", "env":{ "CQ_PLATFORM_API_KEY": "${input:cloudquery-platform-api-key}", "CQ_PLATFORM_API_URL": "${input:cloudquery-platform-api-url}" } } } } ``` **PostgreSQL Mode** ```json { "inputs":[ { "type":"promptString", "id":"cloudquery-postgres-connection-string", "description":"CloudQuery PostgreSQL Connection String", "password":true } ], "servers":{ "CloudQuery":{ "type":"stdio", "command": "/path/to/cq-platform-mcp", "env":{ "POSTGRES_CONNECTION_STRING": "${input:cloudquery-postgres-connection-string}" } } } } ``` **Snowflake Mode (Requires MCP server >= 1.8.0)** ```json { "inputs":[ { "type":"promptString", "id":"cloudquery-snowflake-connection-string", "description":"CloudQuery Snowflake Connection String", "password":true } ], "servers":{ "CloudQuery":{ "type":"stdio", "command": "/path/to/cq-platform-mcp", "env":{ "SNOWFLAKE_CONNECTION_STRING": "${input:cloudquery-snowflake-connection-string}" } } } } ``` --- ### macOS Security Note If macOS blocks the binary, use: ```bash xattr -d com.apple.quarantine /absolute/path/to/cq-platform-mcp ``` Or allow it in **System Preferences → Security & Privacy → General**. --- ### Streamable HTTP Server The MCP server can also be run as a Streamable HTTP server, which allows you to connect to it remotely. To enable this, set the `HTTP_ADDRESS` environment variable to the desired address and port, e.g.: ```bash export HTTP_ADDRESS=":8080" ``` The server will then listen for incoming HTTP requests under the path `/mcp` on the specified address and port. ### About MCP MCP (Model Context Protocol) is a standard for tools like Claude Desktop to interact with external structured data sources. The CloudQuery MCP Server exposes your asset inventory to these tools. ## Next Steps - [AI Assistant](/platform/features/ai-assistant) - Built-in conversational agent on the Platform Home page with optional full data access - [AI Query Writer](/platform/features/sql-console/ai-query-writer) - Use the built-in AI query writer in the platform - [SQL Console](/platform/features/sql-console) - Run queries directly in the platform - [API Keys](/platform/production-deployment/api-keys) - Generate API keys for MCP server authentication --- Source: https://www.cloudquery.io/docs/platform/features/notification-destinations # Notification Destinations Notification destinations define where CloudQuery Platform sends notifications when [alerts](/platform/features/alerts) are triggered or [policy](/platform/features/policies) violations are detected. There are two destination types: **Slack** (native integration with OAuth and channel selection) and **Webhook** (HTTP POST to any endpoint). ## Creating a Notification Destination 1. In the sidebar, click your user icon and select **Organization settings**. 2. Open the **Notification destinations** section. 3. Click **Add notification destination**. 4. Enter a **Destination name** to identify this destination when configuring alerts (e.g., "Slack - Security Channel" or "PagerDuty Webhook"). 5. Select a **Destination type**: **Webhook** or **Slack**. The fields that follow depend on the type you select. ## Slack Destinations The Slack destination type connects to your Slack workspace via OAuth and sends notifications directly to selected channels. No webhook URL or custom payload is required. ### Configuration 1. Select **Slack** from the **Destination type** dropdown. 2. Click **Connect workspace** to start the OAuth flow. A new window opens where you authorize CloudQuery to post to your Slack workspace. 3. After authorization, select one or more channels from the **Slack channels** dropdown. 4. Optionally, enter a **Custom message** to prepend to the alert notification. 5. Click **Save notification destination**. ### How Slack Notifications Work When an alert triggers, CloudQuery posts a message to each selected channel. The message includes the alert details (query name, status, severity, violations). If you set a custom message, it appears above the alert details. ## Webhook Destinations Webhook destinations send an HTTP POST request to any URL you specify. Use this type for services like PagerDuty, Opsgenie, Microsoft Teams, or custom HTTP endpoints. ### Configuration 1. Select **Webhook** from the **Destination type** dropdown. 2. Set the **Destination URL**: the HTTP(S) endpoint to receive notifications. 3. Configure the **Web request body**: the JSON payload sent with each notification. Use [placeholder variables](#placeholder-variables) to include dynamic alert data. 4. Add **HTTP Headers**: set `Content-Type: application/json` for most destinations. 5. Click **Save notification destination**. ### Placeholder Variables Use these variables in the web request body to include dynamic data from the triggering alert: | Variable | Description | | ---------------------- | ------------------------------------------------ | | `{{query_name}}` | The name of the query the alert is configured on | | `{{query_url}}` | Direct URL to the query in the SQL Console | | `{{alert_status}}` | Current alert state: `triggered` or `inactive` | | `{{alert_severity}}` | The severity level configured for the alert | | `{{alert_message}}` | The custom message configured for the alert | | `{{alert_violations}}` | The number of rows returned by the query | ### Example: Slack Incoming Webhook If you prefer to use a [Slack Incoming Webhook](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks) instead of the native Slack integration, select **Webhook** as the destination type, set the webhook URL, add the `Content-Type: application/json` header, and use this request body: ```json { "text": "*{{alert_status}}*: {{query_name}}\nSeverity: {{alert_severity}}\nViolations: {{alert_violations}}\n<{{query_url}}|View in CloudQuery>" } ``` ### Example: Generic Webhook For services like PagerDuty, Opsgenie, or custom HTTP endpoints: ```json { "title": "CloudQuery Alert: {{query_name}}", "status": "{{alert_status}}", "severity": "{{alert_severity}}", "message": "{{alert_message}}", "violations": "{{alert_violations}}", "url": "{{query_url}}" } ``` ## Testing a Destination After saving, click **Send test notification** to verify the endpoint is reachable and the payload format is correct. For webhook destinations, test notifications send the body as-is, without replacing placeholder variables. For Slack destinations, a test message is sent to the selected channels. ## Managing Destinations Notification destinations are shared across all alerts. You can: - Edit a destination's configuration at any time - Delete a destination (alerts using it will stop sending to that endpoint) - Add the same destination to multiple alerts **Note:** The destination type (Webhook or Slack) cannot be changed after creation. To switch types, create a new destination. ## Troubleshooting ### Notifications Are Not Being Received - Verify the destination URL is reachable from the CloudQuery Platform server (webhook destinations only). - Check that the `Content-Type` header matches what the receiving service expects (usually `application/json`). - Use the **Send test notification** button to confirm the endpoint responds. - Alerts only send notifications on state changes. See [Alerts: How Alerts Work](/platform/features/alerts#how-alerts-work) for details. ### Slack Connection Failed - Verify that the Slack integration is configured for your CloudQuery Platform instance. If you see "Slack integration is not configured," contact your administrator. - Make sure you authorized CloudQuery in the correct Slack workspace. - If channels are not appearing in the dropdown, confirm the workspace connection completed and refresh the page. ### Destination Cannot Parse the Request Body Ensure the `Content-Type` header is set correctly. Most webhook endpoints expect `application/json`, but some services require `application/x-www-form-urlencoded` or other content types. ## Related Features - [Policies](/platform/features/policies): configure notifications for policy violations - [Alerts](/platform/features/alerts): trigger notifications when SQL queries return results ## Programmatic access Notification destinations can be managed via the Platform API. See the [Platform API Reference](/platform/reference/api-reference) (`alerts` section) for endpoint details. ## Next Steps - [Alerts](/platform/features/alerts) - Create SQL-based alerts that send notifications - [Policies](/platform/features/policies) - Monitor compliance and trigger notifications on violations - [Reports](/platform/features/reports) - Schedule reports with notifications --- Source: https://www.cloudquery.io/docs/platform/features/policies # Policies Policies are SQL-based detective controls built into CloudQuery Platform. You define criteria for your cloud infrastructure, and CloudQuery continuously evaluates whether your existing resources meet that criteria. Unlike IaC scanners that catch misconfigurations at deployment time, CloudQuery Policies operate at runtime. They detect issues in resources that already exist, including ones created through the console, by third-party tools, or by IaC that has since drifted. ## Policies Dashboard The Policies page displays a summary of your policy health: - **Total active violations** across all enabled policies - **High severity violations** from policies marked `high` or `critical` - **Violations this week** from evaluations in the last 7 days - **Violations by domain** chart showing the distribution across your policy domains - **Violations over time** chart tracking trends across evaluation periods ![CloudQuery Platform Policies dashboard showing policy groups and violation counts](/images/platform/policies/policies-dashboard.png) ## Creating a Policy Policy creation is a 3-step wizard: details, rule logic, and alerting. ### Step 1: Details 1. Go to **Policies** in the sidebar and click **Create Policy**. 2. Enter a **Policy name**. 3. Select a **Domain**. This categorizes the policy's purpose: - Compliance - FinOps - Governance - Operations - Security 4. Select a **Severity**: `critical`, `high`, `medium`, or `low`. 5. Optionally add a **Description** to explain what this policy checks and why. 6. Optionally assign the policy to one or more **Policy Groups** for organizing related policies together. ### Step 2: Rule Logic {#step-2-rule-logic} Define the SQL query that the policy evaluates. You can either: - **Use a saved query**: select from previously saved queries, searchable by name, SQL content, or tags - **Write a new query**: opens the [SQL Console](/platform/features/sql-console) where you write and test your query. The platform validates the query with a dry run before allowing you to proceed. The SQL editor provides table and column autocomplete. Each row returned by the query is one violation. ### Step 3: Alerting Optionally select one or more **notification destinations** ([Slack, webhooks](/platform/features/notification-destinations)) to receive alerts when violations are detected. You can also create a new notification destination directly from this step. Click **Save new policy** to create the policy. ## Writing Policies in SQL Policy rules are ClickHouse SQL queries against CloudQuery's normalized tables. These are the same tables that cover EC2 instances, RDS databases, EBS volumes, and everything else you have synced. You don't need to learn Rego, OPA, or a vendor-specific policy language. ### Example: Find Untagged Expensive EC2 Instances ```sql SELECT instance_id, instance_type, region, JSONExtractString(tags, 'Environment') AS env, JSONExtractString(tags, 'CostCenter') AS cost_center FROM aws_ec2_instances WHERE JSONExtractString(tags, 'CostCenter') = '' AND JSONExtractString(state, 'Name') = 'running' AND instance_type LIKE '%xlarge' ``` ### Example: Find Unencrypted RDS Instances ```sql SELECT db_instance_identifier, engine, region, db_instance_status, JSONExtractString(tags, 'Team') AS team FROM aws_rds_instances WHERE storage_encrypted = false AND db_instance_status = 'available' ``` ### Example: Find Unattached EBS Volumes ```sql SELECT volume_id, volume_type, size, region, create_time, JSONExtractString(tags, 'CostCenter') AS cost_center FROM aws_ec2_ebs_volumes WHERE length(attachments) = 0 ``` ## Managing Policies ### Editing a Policy Open a policy from the policies list and click **Edit**. You can update the name, domain, severity, description, query, policy groups, and notification destinations. In edit mode, you can jump directly to any step in the wizard. Changes take effect on the next evaluation cycle. ### Pausing and Resuming Policies can be set to **active** or **paused**. A paused policy stops evaluating resources until you resume it. Use the action menu on the policy detail page to change status. ### Deleting a Policy Use the action menu on the policy detail page to delete a policy. This removes the policy and its violation history. ## Policy Groups Bundle related policies into Policy Groups to organize by compliance standard (CIS, SOC 2, HIPAA) or by team responsibility. Each group has a name and description, and policies can belong to multiple groups. Policy groups provide an aggregate view of violation counts by severity across all policies in the group. To create a group, go to the Policies page and click **Create policy group**. You can also assign policies to groups during policy creation. Policy groups can also be managed programmatically via the [Platform API Reference](/platform/reference/api-reference) (`policies` section). ## Notifications and Actions When a policy violation is detected, you can notify your team and trigger downstream actions: - Send alerts to Slack (native integration) or any HTTP endpoint via [webhook notification destinations](/platform/features/notification-destinations) - Use webhooks to trigger downstream systems like Jira, PagerDuty, Lambda (via API Gateway), or custom workflows - View all violations in a unified dashboard ![CloudQuery Platform policy alerting configuration with Slack and webhook options](/images/platform/policies/alerting-config.png) ## Tracking Violations over Time The policy detail page includes a violations-over-time chart. Use it to measure the impact of new policies, track remediation progress, and catch regressions. The dashboard also shows trends across evaluation periods, broken down by domain. Policy violations also appear as findings in [Insights](/platform/features/insights), where you can triage them by severity, filter by resource type or account, and track remediation. ## Use Cases - **FinOps**: idle resources, oversize VMs, missing cost tags - **Security**: public buckets, unencrypted volumes, exposed ports - **Compliance**: region restrictions, tag hygiene, audit trails - **Governance**: naming conventions, tagging standards, organizational policies - **Operations**: old AMIs, unsupported instance types, lifecycle policies ## Video Walkthrough Watch how to create a CloudQuery Policy from scratch, from writing the SQL query to configuring alerts and reviewing violations. ## Related Features - [SQL Console](/platform/features/sql-console): write and test the SQL queries that power your policies - [Notification Destinations](/platform/features/notification-destinations): configure Slack and webhook endpoints for policy alerts - [Asset Inventory](/platform/features/asset-inventory): browse the resources your policies evaluate - [Reports](/platform/features/reports): build dashboards that visualize policy compliance trends - [Historical Snapshots](/platform/features/sql-console/historical-snapshots): combine with policies to track compliance drift over time ## Programmatic access Policies and policy groups can be managed via the Platform API. See the [Platform API Reference](/platform/reference/api-reference) (`policies` section) for endpoint details. ## Next Steps - [Alerts](/platform/features/alerts) - Get notified when policies detect violations - [Reports](/platform/features/reports) - Generate compliance reports from policy results - [Asset Inventory](/platform/features/asset-inventory) - Browse resources affected by policy violations - [Query Examples](/platform/features/sql-console/query-examples) - Security and compliance query examples --- Source: https://www.cloudquery.io/docs/platform/features/reports/built-in-report-templates # Built-in Report Templates CloudQuery Platform includes pre-built report templates that you can use as-is or customize. Each template is a YAML-defined report with pre-configured widgets and SQL queries. To use a template, open **Reports**, click **New Report**, and select **Use a template**. You can search by integration name or category. Select a template to preview it, then click **Save** to create the report. After saving, you can edit the YAML to customize it. See the [Reports YAML documentation](/platform/features/reports/reports-yaml-documentation-with-examples) for the full reference. Each template lists which integrations are required or recommended in the report's **Tips for using this report** popup. Make sure you have the relevant [integrations syncing](/platform/syncs/setting-up-a-sync) before using a template. ## Cloud Security and Compliance These reports focus on security posture, vulnerability detection, and compliance monitoring. | Template | Description | Key integrations | | -------- | ----------- | ---------------- | | **Asset End of Life Report** | Track assets nearing end of life to stay ahead of security and operational risks | AWS, GCP, Azure | | **AWS Backup Health** | Overview of your AWS data resilience posture: backup coverage, job status, and vault health | AWS | | **AWS: Security Hub Overview** | High-level view of AWS Security Hub findings by severity and resource type | AWS | | **AWS Cloud Security: Data Security Overview** | Security analysis of your AWS RDS instances, covering encryption, public access, and backup status | AWS | | **AWS Security Best Practices** | Identify and address security risks across your AWS environment | AWS | | **Azure: Attack Surface Security Posture** | Analyze potential security risks in your Azure environment | Azure | | **AWS CloudTrail Log Detections** | Visibility into security events, anomalous activity, and API calls from CloudTrail logs | AWS | ## Multi-Cloud Governance These reports provide cross-provider visibility into assets, identity, access, and tagging. | Template | Description | Key integrations | | -------- | ----------- | ---------------- | | **Multi-cloud Asset Inventory** | Overview of assets across multiple cloud providers in a single view | AWS, GCP, Azure | | **Tags Overview** | Discover how resources are tagged across providers. Find untagged and inconsistently tagged resources | AWS, GCP, Azure | | **Identity and Access Overview** | Assess user account security across multiple cloud providers | AWS, GCP, Azure | | **User Access Review** | Combines user account information from 10+ integrations into a single access review | AWS, GCP, Azure, GitHub, Okta, and more | | **IP Address Report** | Find public IP addresses and their connected resources across providers | AWS, GCP, Azure | ## Infrastructure and Operations These reports cover network topology, architecture review, and container workloads. | Template | Description | Key integrations | | -------- | ----------- | ---------------- | | **AWS VPC Details** | Detailed view of a single AWS VPC, including public and private instances, subnets, and routing | AWS | | **AWS Well-Architected** | Evaluate your cloud workloads against [AWS Well-Architected Framework](https://aws.amazon.com/architecture/well-architected/) pillars | AWS | | **Kubernetes & Container Report** | Overview of Kubernetes clusters, nodes, workloads, and resource usage across EKS, GKE, and AKS | AWS, GCP, Azure | ## Cost Management | Template | Description | Key integrations | | -------- | ----------- | ---------------- | | **Cost Saving Opportunities** | Find unused or forgotten resources across multiple providers to cut unnecessary costs | AWS, GCP, Azure | ## Other Reports | Template | Description | Key integrations | | -------- | ----------- | ---------------- | | **GitHub Repository Insights** | Overview of repository activity and health: commits, PRs, issues, and contributors | GitHub | | **GitHub Security Insights** | Security metrics across your GitHub repositories: Dependabot alerts, secret scanning, and code scanning | GitHub | For interactive exploration of individual findings behind these reports, use [Insights](/platform/features/insights). ## Customizing Templates After saving a report from a template, you can open the report editor to modify the YAML. Common customizations include: - Adjusting SQL queries to filter by specific accounts, regions, or tags - Adding or removing widgets - Changing visualization types (table, pie chart, bar chart, line chart) - Adding report-level [filters](/platform/features/reports/reports-yaml-documentation-with-examples#filters) for interactive filtering See the [Reports YAML documentation](/platform/features/reports/reports-yaml-documentation-with-examples) for the full reference on widget types and configuration options. ## Next Steps - [Reports](/platform/features/reports) - Learn about creating and managing reports - [Reports YAML Documentation](/platform/features/reports/reports-yaml-documentation-with-examples) - Customize reports with YAML - [Policies](/platform/features/policies) - Complement reports with continuous policy monitoring --- Source: https://www.cloudquery.io/docs/platform/features/reports # Reports Reports turn your synced cloud data into visual dashboards. Use them to monitor security posture, compliance status, cost optimization opportunities, and resource inventory across all your connected [integrations](/platform/core-concepts/integrations). Reports are defined as code (YAML). You can create your own reports from scratch or customize one of the [built-in report templates](/platform/features/reports/built-in-report-templates). ## Creating a Report from a Template 1. Open **Reports** from the main navigation. 2. Click **Use a template** to browse available templates.
    ![Built-in Report Templates](/images/platform/report-template.png) Built-in report templates
    3. Search by integration name or report category. For example, type "aws" to find all reports that use the AWS integration. 4. Select a template to see a preview, then click **Save** to create the report. See [built-in report templates](/platform/features/reports/built-in-report-templates) for a full list of available templates and their use cases. ## Creating a Report from Scratch 1. From the Reports page, click **New Report** and select **Build from scratch**. 2. The report editor opens with YAML code on the right and a live preview on the left.
    ![CloudQuery Report Editor](/images/platform/report-scratch.png) CloudQuery report editor
    3. Edit the YAML to define your widgets. For example, to change a widget to a pie chart: ```yaml display: type: pie_chart ``` 4. Click **Preview Changes** to see the updated visualization. See the [Reports YAML documentation](/platform/features/reports/reports-yaml-documentation-with-examples) for the full reference on widget types, filters, and configuration options. ## Report Layout Reports have a standardized layout. Filters appear at the top right. Below the filters, a button shows which integrations are required or recommended to populate the report's data. Widget layout is automated by the platform. You can control widget width using the `width` property in the YAML, but the overall arrangement is managed automatically. ## Visualizations Each visualization has a dropdown menu in its header bar that lets you: - Open the underlying SQL query in the [SQL Console](/platform/features/sql-console) - Copy the SQL query to your clipboard Table visualizations also include a search box and a full-screen view.
    ![Detail of a table visualization header bar with search, full screen button, and a dropdown menu with SQL query actions.](/images/platform/visualizations.png)
    Table visualization header with search, full screen, and SQL query actions
    ## Troubleshooting ### No Data in a Report Check the **Tips for using this report** button at the top right of the report. The popup lists the integrations the report needs. Make sure you have a [sync set up](/platform/syncs/setting-up-a-sync) for the required integrations.
    ![Example Tips for using this report.](/images/platform/report-issues.png)
    ### Some Widgets Are Empty The resource type may not exist in your environment, or your integration may not sync that table. Open the widget query in the [SQL Console](/platform/features/sql-console) using the dropdown menu and check which tables are used. Verify your integration has those tables selected for syncing. ### No Widgets Visible The report YAML is likely malformed. Check the indentation and spacing. Try removing individual widgets to isolate which one causes the issue. ### Slow Report Performance Reports that query CloudTrail data can be slow due to large data volumes. To improve performance, limit the CloudTrail sync to a shorter time frame or to specific event types using [Table Options](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs#table-options) on the AWS integration. ## Related Features - [Built-in report templates](/platform/features/reports/built-in-report-templates): pre-built reports for common use cases - [Reports YAML documentation](/platform/features/reports/reports-yaml-documentation-with-examples): full reference for report code - [SQL Console](/platform/features/sql-console): write and test the queries that power report widgets - [Historical Snapshots](/platform/features/sql-console/historical-snapshots): use snapshot tables in report queries for trend analysis - [Policies](/platform/features/policies): complement reports with automated detective controls ## Programmatic access Reports and report templates can be managed via the Platform API. See the [Platform API Reference](/platform/reference/api-reference) (`reports` section) for endpoint details. ## Next Steps - [Built-in Report Templates](/platform/features/reports/built-in-report-templates) - Browse 40+ ready-to-use report templates - [Reports YAML Documentation](/platform/features/reports/reports-yaml-documentation-with-examples) - Create custom reports with YAML - [SQL Console](/platform/features/sql-console) - Write custom queries for your reports - [Alerts](/platform/features/alerts) - Set up alerts alongside your reports --- Source: https://www.cloudquery.io/docs/platform/features/reports/reports-yaml-documentation-with-examples/full-report-example # Full Report Example This example report contains every widget and filter type. Copy it into your report editor to experiment with different visualizations and filters. ```yaml report: title: Report with all widgets shortDescription: This reports tests all widgets longDescription: | ## What does this report show? All widgets and their possible visualizations logo: aws tags: - tag1 - tag2 filters: - type: generic filterLabel: Option filterExpression: account_id_filtered = '?' options: - label: account 1 value: 1 - label: account 2 value: 2 - label: account 3 value: 3 - type: generic filterLabel: Account filterExpression: account_id = '?' query: | select value, label from ( select 1 as value, 'account 1' as label union all select 2 as value, 'account 2' as label union all select 3 as value, 'account 3' as label ) order by value - type: date filterLabel: Event Date filterExpression: event_time < '?' defaultValue: now() - interval 30 DAY - type: date filterLabel: Expiration filterExpression: expiration_date < '?' defaultValue: now() + interval 30 DAY mode: future - type: search filterExpression: ip_address LIKE '%?%' filterLabel: IP Address - type: search filterExpression: average_load >= '?' filterLabel: minimum average load (%) defaultValue: 50 groups: - title: Group with filtered widgets description: Widgets using the filters above widgets: - title: Widget with a generic option filter display: type: table query: | select account_id as account_id_filtered, account_name from ( select 1 as account_id, 'account 1' as account_name union all select 2 as account_id, 'account 2' as account_name ) where 1=1 and account_id_filtered = '?' - title: Widget with a generic filter display: type: table query: | select account_id, account_name from ( select 1 as account_id, 'account 1' as account_name union all select 2 as account_id, 'account 2' as account_name ) where 1=1 and account_id = '?' - title: widget filtered by Event Date display: type: table query: | select event_time from ( select now() as event_time union all select now() - interval 10 day as event_time ) where 1=1 and event_time < '?' - title: widget filtered by Expiration Date display: type: table query: | select expiration_date from ( select now() as expiration_date union all select now() + interval 10 day as expiration_date ) where 1=1 and expiration_date < '?' - title: widget using search without default value display: type: table query: | select ip_address from ( select '192.168.1.1' as ip_address union all select '192.168.1.2' as ip_address union all select '192.168.2.1' as ip_address union all select '192.168.3.1' as ip_address ) where 1=1 and ip_address LIKE '%?%' - title: widget using search with default value display: type: table query: | select ip_address, average_load from ( select '192.168.1.1' as ip_address, 30 as average_load union all select '192.168.1.2' as ip_address, 40 as average_load union all select '192.168.2.1' as ip_address, 50 as average_load union all select '192.168.3.1' as ip_address, 60 as average_load ) where 1=1 and average_load >= '?' - title: Unfiltered widgets description: simple widgets with no filters applied widgets: - title: label widget display: type: label query: | select 1 - title: bar chart widget display: type: bar_chart entryLabel: average query: | select 'row 1' as x, 2 as y union all select 'row 2' as x, 3 as y - title: pie chart widget display: type: pie_chart total: show: true label: pie chart label query: | select 'slice 1' as slice, 10 as value union all select 'slice 2' as slice, 20 as value union all select 'slice 3' as slice, 5 as value - title: table widget display: type: table query: | select 'EU' as region, 2 as count, toDate(now()) as date union all select 'US' as region, 3 as count, toDate(now()) as date - title: table widget with search display: type: table query: | select 'EU' as region, 2 as count, toDate(now()) as date, concat('eu','abc') as search_string union all select 'US' as region, 3 as count, toDate(now()) as date, concat('us','def') as search_string - title: bar chart with multiple series display: type: bar_chart width: 100% query: | WITH toDate('2024-01-01') AS start_date, toDate('2024-01-05') AS end_date SELECT date, map( 'series 1', randUniform(10, 50), 'series 2', randUniform(0, 100), 'series 3', randUniform(10, 30) ) FROM ( SELECT addDays(start_date, number) AS date FROM numbers(dateDiff('day', start_date, end_date) + 1) ) ORDER BY date; - title: XY-chart display: type: xy_chart width: 100% queries: - title: Random series 1 query: | WITH toDate('2024-01-01') AS start_date, toDate('2024-01-05') AS end_date SELECT date, rand() / 1000000000 as random_normal FROM ( SELECT addDays(start_date, number) AS date FROM numbers(dateDiff('day', start_date, end_date) + 1) ) ORDER BY date; - title: Random series 2 query: | WITH toDate('2024-01-01') AS start_date, toDate('2024-01-05') AS end_date SELECT date, rand() / 1000000000 AS random_normal FROM ( SELECT addDays(start_date, number) AS date FROM numbers(dateDiff('day', start_date, end_date) + 1) ) ORDER BY date; ``` ## Next Steps - [Reports YAML Documentation](/platform/features/reports/reports-yaml-documentation-with-examples) - YAML reference for report configuration - [Built-in Report Templates](/platform/features/reports/built-in-report-templates) - Browse templates to use as starting points - [Reports](/platform/features/reports) - Learn about creating and managing reports --- Source: https://www.cloudquery.io/docs/platform/features/reports/reports-yaml-documentation-with-examples # Reports YAML Documentation with Examples Each report is defined as YAML code with three main sections: [report properties](#report-properties), [widgets](#widgets), and optional [filters](#filters). ## Report Properties A report starts with the report definition section with the following properties: | Property | Description | | -------- | ----------- | | `title` | The report title displayed in the main heading and in the list of reports | | `shortDescription` | Brief description displayed as a subtitle in the report | | `longDescription` | Markdown-formatted text displayed in the **Tips for using this report** popup | | `logo` | One of the built-in images for report cards (see values below) | | `tags` | Array of labels for filtering reports in the main Reports view | | `filters` | Optional array of report-level filters (see [Filters](#filters)) | | `widgets` | Array of visualizations displayed in the report (see [Widgets](#widgets)) | ```yaml report: title: <> shortDescription: <> longDescription: | ## What does this report show? <> logo: <> tags: - tag1 - tag2 filters: - type: generic ... widgets: - title: "first widget" ... ``` ## Widgets A widget has the following properties: **title:** The title of the widget | Property | Description | | -------- | ----------- | | `title` | The widget title | | `display` | Display options including `type` (see widget types below) and optional `width` | | `query` | ClickHouse SQL query that populates the widget | ```yaml - title: <> display: type: <> query: | select if(length(tags) = 0, 'Untagged Resources', 'Tagged Resources') AS tags_status, count(*) from cloud_assets where resource_category != '' and supports_tags = true group by tags_status order by tags_status asc ``` ### Label A label shows text or a single number. ```yaml - title: label widget display: type: label query: | select 1 ```
    ![Report label widget displaying a single value result from a SQL query](/images/platform/report-widget-label.png) A label widget
    ### Table The table widget displays the query result as-is. ``` - title: table widget display: type: table query: | select 'EU' as region, 2 as count, toDate(now()) as date union all select 'US' as region, 3 as count, toDate(now()) as date ```
    ![Report table widget displaying SQL query results with region, count, and date columns](/images/platform/report-widget-table.png) A table widget
    ### Pie Chart The pie chart visualizes the first two columns of the query result. The first column must be a string, the second must be a number. Optionally, display a total in the center of the pie chart using the `total` property. ```yaml - title: pie chart widget display: type: pie_chart total: show: true label: pie chart inner label query: | select 'slice 1' as slice, 10 as value union all select 'slice 2' as slice, 20 as value union all select 'slice 3' as slice, 5 as value ```
    ![A pie chart widget with the sum of all values displayed](/images/platform/report-widget-pie-chart.png) A pie chart widget with the sum of all values displayed
    ### Bar Chart The bar chart visualizes the first two columns of the query result. The first column must be a string, the second must be a number. Use the `entryLabel` property to customize the label shown on hover. ```yaml - title: bar chart widget display: type: bar_chart entryLabel: average query: | select 'row 1' as x, 2 as y union all select 'row 2' as x, 3 as y ```
    ![Report bar chart widget visualizing two data rows with labeled bars and hover detail on hover](/images/platform/report-widget-bar-chart.png) A bar chart widget
    ### Bar Chart with Multiple Series To visualize multiple series, put the values and series names in a single column of the `map` type. ```yaml - title: bar chart with multiple series display: type: bar_chart width: 100% query: | WITH toDate('2024-01-01') AS start_date, toDate('2024-01-05') AS end_date SELECT date, map( 'series 1', randUniform(10, 50), 'series 2', randUniform(0, 100), 'series 3', randUniform(10, 30) ), FROM ( SELECT addDays(start_date, number) AS date FROM numbers(dateDiff('day', start_date, end_date) + 1) ) ORDER BY date; ```
    ![Multi-series bar chart using the map column type](/images/platform/report-widget-multiple-series-chart.png) Multi-series bar chart using the map column type
    A practical example using [Historical Snapshots](/platform/features/sql-console/historical-snapshots) data, showing storage bucket counts by day and cloud provider: ```sql SELECT date, map( 'aws', sumIf(bucket_count, cloud = 'aws'), 'gcp', sumIf(bucket_count, cloud = 'gcp') ) AS storage_counts FROM ( SELECT toDate(_cq_snapshot_time) AS date, 'aws' AS cloud, count() AS bucket_count FROM snapshot_aws_s3_buckets GROUP BY date UNION ALL SELECT toDate(_cq_snapshot_time) AS date, 'gcp' AS cloud, count() AS bucket_count FROM snapshot_gcp_storage_buckets GROUP BY date ) GROUP BY date ORDER BY date; ``` ### X-Y Chart The X-Y chart is a line chart that displays multiple data series over the same X axis. Unlike other widgets, it accepts an array of `queries` instead of a single `query`. Each query has a `title` (series name) and the SQL query. ```yaml - title: XY-chart display: type: xy_chart width: 100% queries: - title: Random series 1 query: | WITH toDate('2024-01-01') AS start_date, toDate('2024-01-05') AS end_date SELECT date, rand() / 1000000000 as random_normal FROM ( SELECT addDays(start_date, number) AS date FROM numbers(dateDiff('day', start_date, end_date) + 1) ) ORDER BY date; - title: Random series 2 query: | WITH toDate('2024-01-01') AS start_date, toDate('2024-01-05') AS end_date SELECT date, rand() / 1000000000 AS random_normal FROM ( SELECT addDays(start_date, number) AS date FROM numbers(dateDiff('day', start_date, end_date) + 1) ) ORDER BY date; ```
    ![An X-Y chart widget with a hover detail displayed](/images/platform/report-widget-xy-chart.png) An X-Y chart widget with a hover detail displayed
    ## Filters Add top-level filters to a report to let users interactively filter widget data. Each filter is defined by its **type**, **label**, **expression**, and optionally the available values. | Property | Description | | -------- | ----------- | | `type` | Filter type: `generic` (dropdown), `date` (date picker), or `search` (free-form input) | | `filterLabel` | Label displayed to the user | | `filterExpression` | Placeholder expression used in widget SQL queries. CloudQuery replaces it when the user selects a value | | `defaultValue` | Optional default value applied on first load | To connect a filter to a widget, include the `filterExpression` in the widget SQL `WHERE` clause. Each filter expression needs to be unique within the report. There cannot be two filters with the same filter expression, such as `account_id = '?'` . If you need to have two filters filtering on the same column name, consider using the `as` keyword in the SQL queries to rename the columns to match the specific filter expression. For example, you can define the filter expression as `account = '?'` . Then in the SQL query, define the where clause using this exact expression: ```yaml ... query: | select account, name from cloud_assets where 1=1 and account = '?' ... ``` Use `1=1` before the filter expression so that when no filter value is selected, the widget shows unfiltered results. The examples below show each filter type with a matching widget. ### Generic Filter with Statically Defined Options This filter is a dropdown with a static list of options defined in the code. Each option has a label displayed to the user and value used in the SQL query. ```yaml - type: generic filterLabel: Option filterExpression: account_id = '?' options: - label: account 1 value: 1 - label: account 2 value: 2 - label: account 3 value: 3 ```
    ![A generic filter with statically defined options](/images/platform/report-generic-filter-options.png) A generic filter with statically defined options
    An example of a table widget using this filter: ```yaml - title: Widget with a generic option filter display: type: table query: | select account_id as account_id_filtered, account_name from ( select 1 as account_id, 'account 1' as account_name union all select 2 as account_id, 'account 2' as account_name ) where 1=1 and account_id = '?' ``` ### Generic Filter with Options from a SQL Query This filter looks the same as above, but loads options dynamically from a SQL query. You can use any CloudQuery table to populate the values. ```yaml - type: generic filterLabel: Account filterExpression: account_id = '?' query: | select value, label from ( select 1 as value, 'account 1' as label union all select 2 as value, 'account 2' as label union all select 3 as value, 'account 3' as label ) order by value ``` ### Multiple Select Filter Building on the above filters, you can enable multi-select functionality by adding the `multiple: true` attribute. This allows users to select multiple values from the dropdown, filtering results that match any of the selected options. An example filter configuration: ```yaml - type: generic filterLabel: Accounts filterExpression: account_id IN (?) multiple: true options: - label: account 1 value: 1 - label: account 2 value: 2 - label: account 3 value: 3 ``` An example widget using this multi-select filter: ```yaml - title: Widget with a multiple generic option filter display: type: table query: | select account_id as account_id_filtered, account_name from ( select 1 as account_id, 'account 1' as account_name union all select 2 as account_id, 'account 2' as account_name union all select 3 as account_id, 'account 3' as account_name ) where 1=1 and account_id IN (?) ``` ### Date Selector The date selector provides a calendar dropdown with preset buttons for past intervals (7 days ago, 14 days ago, and so on). ```yaml - type: date filterLabel: Event Date filterExpression: event_time < '?' defaultValue: now() - interval 30 DAY ```
    ![A date selector filter](/images/platform/report-date-selector.png) A date selector filter
    An example widget using the filter above: ```yaml - title: widget filtered by Event Date display: type: table query: | select event_time from ( select now() as event_time union all select now() - interval 10 day as event_time ) where 1=1 and event_time < '?' ``` ### Date Selector with Future Dates Add `mode: future` to change the preset buttons to point to future dates: ```yaml - type: date filterLabel: Expiration filterExpression: expiration_date < '?' defaultValue: now() + interval 30 DAY mode: future ```
    ![A date selector filter with future mode](/images/platform/report-date-selector-future.png) A date selector filter with future mode
    ### Free-Form Search Filter This free-form search input lets users type in any value. ```yaml - type: search filterExpression: ip_address LIKE '%?%' filterLabel: IP Address ```
    ![A free-form search filter](/images/platform/report-search-filter.png) A free-form search filter
    An example widget using this filter: ```yaml - title: widget using search without default value display: type: table query: | select ip_address from ( select '192.168.1.1' as ip_address union all select '192.168.2.1' as ip_address union all select '192.168.3.1' as ip_address ) where 1=1 and ip_address LIKE '%?%' ``` ### Free-Form Search Filter with a Default Value Add a `defaultValue` property to the above to always load a fresh report with this filter set automatically. ```yaml - type: search filterExpression: average_load >= '?' filterLabel: minimum average load (%) defaultValue: 50 ``` An example widget using this filter: ```yaml - title: widget using search with default value display: type: table query: | select ip_address, average_load from ( select '192.168.1.1' as ip_address, 30 as average_load union all select '192.168.1.2' as ip_address, 40 as average_load union all select '192.168.2.1' as ip_address, 50 as average_load union all select '192.168.3.1' as ip_address, 60 as average_load ) where 1=1 and average_load >= '?' ``` ## Advanced Options ### Setting Widget Width Use the `width` property to set the widget width as a percentage of the view port. Available on all widget types. ```yaml - title: label widget display: type: label width: 50% query: | select 1 ``` ### Adding Search to a Table Widget Add a `search_string` column to the SQL query. The column is not displayed in the table. Instead, a search box appears in the table header that filters rows based on this column. Use `concat()` to make multiple columns searchable. ```yaml - title: table widget with search display: type: table query: | select 'EU' as region, 2 as count, toDate(now()) as date, concat('eu','abc') as search_string union all select 'US' as region, 3 as count, toDate(now()) as date, concat('us','def') as search_string ```
    ![A table widget with the search functionality enabled](/images/platform/report-table-search.png) A table widget with the search functionality enabled
    ### Groups Wrap widgets in collapsible groups to organize the report into sections. **title:** The title of the group **description**: The subtitle displayed below the title. **widgets**: An array of widgets. ``` groups: - title: first group description: first group with widgets widgets: - title: label widget display: type: label query: | select 1 ```
    ![Report layout with multiple widget groups organized into labeled sections for structured dashboards](/images/platform/report-groups.png) A report with groups
    ## Next Steps - [Full Report Example](/platform/features/reports/reports-yaml-documentation-with-examples/full-report-example) - Complete YAML report example - [Built-in Report Templates](/platform/features/reports/built-in-report-templates) - Browse ready-to-use templates - [SQL Console](/platform/features/sql-console) - Test queries before adding them to reports --- Source: https://www.cloudquery.io/docs/platform/features/sql-console/ai-query-writer # AI Query Writer
    ![CloudQuery Platform SQL Console with the AI Query Writer assistant panel for generating SQL queries](/images/platform/ai-query-1.png)
    The SQL Console includes a built-in AI assistant that writes ClickHouse SQL from plain-language descriptions. Describe what you want to find, and the assistant figures out which tables are relevant, inspects their schemas, and generates a query you can run or refine. CloudQuery Platform also includes the [AI Assistant](/platform/features/ai-assistant) on the Home page, a full conversational agent that can answer questions directly and supports both schema-only and full data access modes. ## How to use it Click the AI assistant button at the top of the [SQL Console](/platform/features/sql-console), describe what you're looking for, and submit. The assistant typically responds in a few seconds. ## How it works The assistant is powered by [Claude](https://www.anthropic.com/claude) (Anthropic) running on AWS Bedrock. Each request follows this sequence automatically: 1. **Table discovery**: searches the CloudQuery catalog for tables relevant to your question 2. **Schema inspection**: reads the column names and types for those tables 3. **Syntax validation**: checks the generated SQL against the CloudQuery database before returning it ## Data privacy The AI Query Writer can only see the *structure* of your data: table names and column definitions. It never reads, queries, or transmits the actual contents of your CloudQuery database. Here is exactly what the model can and cannot access: | The model can see | The model cannot see | | --------------------------------- | -------------------------------------------- | | Table names in your CloudQuery database | Row data of any kind | | Column names and their data types | Values stored in any column or field | | SQL syntax validation results | Credentials, connection strings, or API keys | This is not configurable. The AI Query Writer always operates in schema-only mode. There is no setting that would allow it to read your data. AWS Bedrock does not use customer data to train or improve foundation models. Your queries and schema information are not retained for model training purposes. ## Conversation context The assistant remembers the last 20 messages within a session. This means you can iterate (ask a follow-up question to refine a query, correct a mistake, or narrow the scope) without starting over each time. ## Limitations - The model occasionally generates SQL referencing a table or column that doesn't exist. When this happens it usually self-corrects within the same request after observing the error. - Tables with many hundreds of columns may require the model to paginate through the schema, which can add a few seconds. - The assistant is scoped to SQL generation tasks. Off-topic requests are declined. - Queries involving deeply nested structures (such as AWS resource tags stored as JSON, or arrays of security group rules) may need manual adjustment after generation. ## FAQ ### How do I access this feature? This feature is available to all CloudQuery Platform users. Click the AI assistant button at the top of the [SQL Console](/platform/features/sql-console). ### Which AI model does this use? [Claude Sonnet 4.5 and Haiku 4.5](https://www.anthropic.com/claude) (Anthropic), running on AWS Bedrock. The models only receive the information described in the [Data privacy](#data-privacy) section above. ### Does the AI have access to my data? No. The AI Query Writer only sees table names and column definitions; it cannot query or read any row data from your CloudQuery database. See [Data privacy](#data-privacy) for the full breakdown. ### Is there an AI feature that can also reason about my actual data? Yes. The [AI Assistant](/platform/features/ai-assistant) on the Platform Home page supports a Full Access mode where it executes queries and returns results directly in the chat. This mode is enabled by organization administrators in Platform Settings. The [CloudQuery MCP Server](/platform/features/mcp-server) is another option: it lets you connect Claude Desktop, Cursor, or other MCP-compatible tools to your CloudQuery database directly and requires a separately issued API key. ## Next Steps - [AI Assistant](/platform/features/ai-assistant) - The conversational AI on the Platform Home page with full data access support - [SQL Console](/platform/features/sql-console) - Save and manage your AI-generated queries - [Query Examples](/platform/features/sql-console/query-examples) - Browse curated security, compliance, and cost queries - [CloudQuery MCP Server](/platform/features/mcp-server) - Use the CloudQuery MCP server with AI assistants that can also read your data --- Source: https://www.cloudquery.io/docs/platform/features/sql-console/historical-snapshots # Historical Snapshots Historical Snapshots capture the state of your cloud data at regular intervals, giving you the ability to track how your infrastructure changes over time. Use them for trend analysis, compliance auditing, incident investigation, and understanding resource lifecycle. ## How It Works CloudQuery automatically creates daily snapshots of your tables and stores them in dedicated `snapshot_` tables. Snapshots are taken daily, even when no sync is running, so there are no gaps in your historical record. Each snapshot row includes a `_cq_snapshot_time` column that records when the data was captured. Snapshot tables automatically adapt to schema changes, always matching the latest integration version. ## Table Naming Convention Snapshot tables follow the pattern `snapshot_[original_table_name]`: | Original table | Snapshot table | | --------------------------- | ------------------------------------- | | `aws_ec2_instances` | `snapshot_aws_ec2_instances` | | `gcp_compute_instances` | `snapshot_gcp_compute_instances` | | `azure_storage_accounts` | `snapshot_azure_storage_accounts` | ## Browsing Snapshots Open the [SQL Console](/platform/features/sql-console) and select the **Historical Snapshots** tab in the left sidebar to see all available snapshot tables.
    ![Historical Snapshots tab in the SQL Console showing available snapshot tables for time-travel queries](/images/platform/snapshots-browse.png)
    You can query snapshot tables the same way you query any other table, and you can join snapshot tables with regular tables in a single query. ## Example Queries ### Asset History Tracking Look up the state of a resource that has since been deleted or modified: ```sql -- Find parameters of a deleted EC2 instance SELECT * FROM snapshot_aws_ec2_instances WHERE instance_id = 'i-0f73af6b372abb8c7' AND toDate(_cq_snapshot_time) = toDate('2025-07-08') ``` ### Trend Analysis Track resource counts over time to spot growth trends or unexpected changes: ```sql -- Daily count of EC2 instances SELECT toDate(_cq_snapshot_time) as day, count(*) as instance_count FROM snapshot_aws_ec2_instances GROUP BY day ORDER BY day ``` ### Change Detection Compare two snapshots to find what changed between dates: ```sql -- Find EC2 instances that existed last week but not today SELECT yesterday.instance_id, yesterday.instance_type, yesterday.region FROM snapshot_aws_ec2_instances AS yesterday LEFT JOIN snapshot_aws_ec2_instances AS today ON yesterday.instance_id = today.instance_id AND toDate(today._cq_snapshot_time) = today() WHERE toDate(yesterday._cq_snapshot_time) = today() - 7 AND today.instance_id IS NULL ``` ### Historical Reports Snapshot tables work with [Reports](/platform/features/reports), so you can build visualizations that show trends over time. This query shows storage bucket counts by cloud provider per day: ```sql SELECT date, map( 'aws', sumIf(bucket_count, cloud = 'aws'), 'gcp', sumIf(bucket_count, cloud = 'gcp'), 'azure', sumIf(bucket_count, cloud = 'azure') ) AS storage_counts FROM ( SELECT toString(toDate(_cq_snapshot_time)) AS date, 'aws' AS cloud, count() AS bucket_count FROM snapshot_aws_s3_buckets GROUP BY date UNION ALL SELECT toString(toDate(_cq_snapshot_time)) AS date, 'gcp' AS cloud, count() AS bucket_count FROM snapshot_gcp_storage_buckets GROUP BY date UNION ALL SELECT toString(toDate(_cq_snapshot_time)) AS date, 'azure' AS cloud, count() AS bucket_count FROM snapshot_azure_storage_containers GROUP BY date ) GROUP BY date ORDER BY date; ``` ![Storage buckets count over time](/images/platform/snapshots-storage-over-time.png) ## Related Features - [SQL Console](/platform/features/sql-console): run queries against snapshot and live tables - [Reports](/platform/features/reports): visualize snapshot query results in dashboards - [Policies](/platform/features/policies): combine with snapshots to track compliance drift over time ## Next Steps - [SQL Console](/platform/features/sql-console) - Write and manage SQL queries - [Understanding Platform Views](/platform/advanced-topics/understanding-platform-views) - How the platform creates queryable views - [Reports](/platform/features/reports) - Build reports from snapshot queries --- Source: https://www.cloudquery.io/docs/platform/features/sql-console # SQL Console The SQL Console lets you run ClickHouse SQL queries directly against your synced cloud data. Use it for ad-hoc analysis, cross-table joins, aggregations, and any query that goes beyond the [Asset Inventory](/platform/features/asset-inventory) search syntax. The SQL Console is read-only. You can query data but cannot modify it, or create tables or views. ## Table Browser The left sidebar lists all available tables, organized into two sections: - **Asset Inventory Tables**: normalized tables that provide a consistent schema across all cloud resources, regardless of the source provider - **Integration Tables**: the raw tables synced from each [integration](/platform/core-concepts/integrations), reflecting the exact schema from the provider's API Expand any table to see its columns and data types. Click a column name to insert it into the query editor. ## Running a Query Queries use [ClickHouse SQL syntax](https://clickhouse.com/docs/en/sql-reference). Type your query in the editor and press the **Run Query** button or use the keyboard shortcut: | Action | Shortcut | | --------- | -------------------------- | | Run query | `Ctrl+Enter` / `Cmd+Enter` | The editor provides autocomplete for table names and column names as you type. ### Example: Find Unassigned EC2 Images This query joins `aws_ec2_images` with `aws_ec2_instances` to find AMIs not attached to any instance: ```sql SELECT img.image_id, inst.image_id as ec2_image, -- should all be NULL to show unassigned img.name, img.creation_date, img.state, img.tags FROM aws_ec2_images img LEFT JOIN aws_ec2_instances inst ON inst.image_id = img.image_id WHERE inst.image_id IS NULL -- AMIs not associated with any EC2 instance AND img.state = 'available' AND img.creation_date <= NOW() - INTERVAL 30 DAY ORDER BY img.creation_date ASC; ``` Results appear in a table below the editor.
    ![SQL Console displaying query results in a table below the query editor after running a query](/images/platform/sql-console-results.png) SQL Query Results
    If a query returns more than 100 rows, the results are automatically paged. ## Inspecting Results Click any cell in the results table to open the **Cell Inspection** sidebar. This is useful for exploring complex JSON objects or copying individual values.
    ![Cell Inspection view with a complex JSON object](/images/platform/sql-console-cell.png)
    Cell Inspection view with a complex JSON object
    ## Understanding the Data Integration tables map directly to the provider's API endpoints. The data is not transformed. To understand a table's schema and where its data comes from, look it up on [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source). For example, the [AWS EC2 Instances table](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/tables/aws_ec2_instances) documentation shows the columns, types, and the underlying AWS API. For query ideas and worked examples, see the [tutorials on the CloudQuery blog](https://www.cloudquery.io/blog/category/tutorials). ## Saved Queries Save frequently used queries by clicking **Save** after writing a query. Saved queries can be: - Accessed from the **Saved queries** panel in the top right of the SQL Console - Tagged for organization - Shared with your team - Managed via the [REST API](https://platform-multi-tenant-api-docs.cloudquery.io/#tag/queries) - Configured with [alerts](/platform/features/alerts) to trigger notifications when they return results You do not need to save a query before running it. The editor works for ad-hoc queries too. ## Query Examples For practical examples of security, compliance, and FinOps queries, see the [query examples](/platform/features/sql-console/query-examples) section. ## Related Features - [Asset Inventory](/platform/features/asset-inventory): browse and search resources without writing SQL - [Historical Snapshots](/platform/features/sql-console/historical-snapshots): query point-in-time snapshots of your tables for trend analysis - [Reports](/platform/features/reports): turn SQL queries into persistent visualizations and dashboards - [Policies](/platform/features/policies): use SQL queries as detective controls that evaluate after every sync - [Alerts](/platform/features/alerts): trigger notifications when a saved query returns results - [AI Assistant](/platform/features/ai-assistant): ask questions about your infrastructure in plain language and get direct answers or SQL from the Platform Home page - [AI Query Writer](/platform/features/sql-console/ai-query-writer): describe what you want in natural language and get a SQL query ## Next Steps - [Historical Snapshots](/platform/features/sql-console/historical-snapshots) - Query point-in-time snapshots of your data - [AI Query Writer](/platform/features/sql-console/ai-query-writer) - Generate SQL queries using natural language - [Reports](/platform/features/reports) - Build reports from your saved queries - [Query Examples](/platform/features/sql-console/query-examples) - Security, compliance, and cost queries --- Source: https://www.cloudquery.io/docs/platform/features/sql-console/query-examples/compliance-focused-queries # Compliance-Focused Queries ## Verify Cloud Compliance with SQL CloudQuery lets you verify compliance with standards like CIS, ISO 27001, and NIST by querying your cloud asset inventory directly. ### Compliance Queries in Action Use these queries to **maintain compliance and avoid costly violations.**
    Find untagged resources (AWS, GCP, Azure) **Why it matters:** Lack of tagging makes compliance tracking and cost allocation difficult. ```sql SELECT cloud, account, name, region, resource_type FROM cloud_assets WHERE tags = '{}' OR tags IS NULL; ```
    Identify inconsistent tagging formats (AWS, GCP, Azure) **Why it matters:** Tag inconsistency prevents automated cost allocation and policy enforcement. ```sql SELECT cloud, account, name, region, resource_type FROM cloud_assets WHERE tags LIKE '%Environment%' OR tags LIKE '%ENV%' OR tags LIKE '%Env%' OR tags LIKE '%env%' OR tags LIKE '%ENVIRONMENT%'; ```
    Identify resources not tagged according to a governance policy (AWS, GCP, Azure) **Why it matters:** Proper tagging helps track resources for compliance, auditing, and cost allocation. ```sql SELECT cloud, account, name, region, resource_type, tags FROM cloud_assets WHERE tags NOT LIKE '%cost_center%' OR tags NOT LIKE '%owner%' OR tags NOT LIKE '%environment%'; ```
    Identify resources running in unauthorized regions (AWS, GCP, Azure) **Why it matters:** Regulatory restrictions prevent companies from deploying workloads in unauthorized regions. ```sql SELECT cloud, account, name, region, resource_type FROM cloud_assets WHERE region NOT IN ('us-east-1', 'us-west-1', 'eu-west-1'); ```
    Find AWS RDS instances lacking automated backups (AWS) **Why it matters:** Without backups, organizations risk permanent data loss. ```sql SELECT db_instance_arn, backup_retention_period, region FROM aws_rds_instances WHERE backup_retention_period = 0; ```
    Identify databases without SSL encryption (AWS) **Why it matters:** Unsecured database connections expose data to attacks. ```sql SELECT * FROM aws_rds_instances WHERE empty(ca_certificate_identifier); ```
    List RDS instances not using encryption at rest (AWS) **Why it matters:** Unencrypted databases violate compliance and security policies. ```sql SELECT * FROM aws_rds_instances WHERE storage_encrypted=false; ```
    Find IAM users without MFA enabled (AWS) **Why it matters:** Lack of MFA increases the risk of account compromise. ```sql SELECT u.* FROM aws_iam_users AS u LEFT JOIN aws_iam_mfa_devices AS m ON u.user_name = m.user_name WHERE m.user_name IS NULL; ```
    ## More query examples See the [query examples overview](./) for security and FinOps-focused queries. ## Next Steps - [Security-Focused Queries](/platform/features/sql-console/query-examples/security-focused-queries) - Security queries - [FinOps-Focused Queries](/platform/features/sql-console/query-examples/finops-focused-queries) - Cost optimization queries - [Policies](/platform/features/policies) - Automate compliance checks with policy rules - [Reports](/platform/features/reports) - Generate compliance reports --- Source: https://www.cloudquery.io/docs/platform/features/sql-console/query-examples/finops-focused-queries # FinOps-Focused Queries ## Reduce Cloud Costs with SQL CloudQuery lets you query your cloud asset inventory to identify cost inefficiencies and reduce spending. ### FinOps Queries in Action Use these queries to **cut costs and improve cloud efficiency.**
    Identify resources without a "cost center" tag (AWS, GCP, Azure) **Why it matters:** Without a `cost_center` tag, it's difficult to allocate cloud expenses correctly. ```sql SELECT cloud, account, name, region, resource_type, tags FROM cloud_assets WHERE tags NOT LIKE '%cost_center%'; ```
    List unattached (orphaned) storage volumes (AWS, GCP, Azure) **Why it matters:** Unused storage volumes drive up cloud costs unnecessarily. ```sql WITH unattached_disks AS ( SELECT _cq_id FROM gcp_compute_disks WHERE length(users) = 0 UNION DISTINCT SELECT _cq_id FROM azure_compute_disks WHERE managed_by is null UNION DISTINCT SELECT _cq_id FROM aws_ec2_ebs_volumes WHERE attachments='[]' ) SELECT cloud, account, region, name, resource_type_label, tags FROM cloud_assets a JOIN unattached_disks d ON d._cq_id = a._cq_id ORDER BY cloud, account, region, resource_type_label, name ```
    Identify expired reserved instances (AWS) **Why it matters:** Missing out on renewing reserved instances leads to higher on-demand costs. ```sql SELECT arn, fixed_price, instance_count, instance_type FROM aws_ec2_reserved_instances WHERE toDate(end) < now(); ```
    Find GCP BigQuery tables without partitioning enabled (GCP) **Why it matters:** Tables without partitioning lead to expensive queries due to full table scans. ```sql SELECT friendly_name, project_id, location FROM gcp_bigquery_tables WHERE num_partitions = 0; ```
    List EC2 instances that are not using graviton processors (AWS) **Why it matters:** Graviton-based instances provide better performance at a lower cost. ```sql SELECT arn, instance_type, region FROM aws_ec2_instances WHERE instance_type NOT LIKE 't4g%' AND instance_type NOT LIKE 'm6g%' AND instance_type NOT LIKE 'c6g%' AND instance_type NOT LIKE 'r6g%'; -- List of Graviton-based instance types ```
    Detect unused elastic IPs (AWS) **Why it matters:** Unused Elastic IPs incur unnecessary costs. ```sql SELECT public_ip, instance_id FROM aws_ec2_eips WHERE instance_id IS NULL; ```
    List old snapshots that can be deleted (AWS) **Why it matters:** Keeping old snapshots that are no longer needed increases storage costs. ```sql SELECT arn, allocated_storage, engine FROM aws_rds_db_snapshots WHERE instance_create_time < NOW() - INTERVAL 180 DAY; -- Older than 6 months ```
    ## More query examples See the [query examples overview](./) for security and compliance-focused queries. ## Next Steps - [Security-Focused Queries](/platform/features/sql-console/query-examples/security-focused-queries) - Security and compliance queries - [Compliance-Focused Queries](/platform/features/sql-console/query-examples/compliance-focused-queries) - Compliance monitoring queries - [Reports](/platform/features/reports) - Build cost reports from your FinOps queries - [AWS CUR Integration](/platform/integration-guides/setting-up-an-awscur-integration) - Add AWS billing data for cost analysis --- Source: https://www.cloudquery.io/docs/platform/features/sql-console/query-examples # Query Examples CloudQuery transforms raw cloud asset inventory into structured data that you can query with SQL. This enables real-time visibility into your infrastructure for security, compliance, and cost optimization. The following query examples cover different focus areas: * [**Security-focused queries**](/platform/features/sql-console/query-examples/security-focused-queries): Identify misconfigurations and security risks before they become breaches. * [**Compliance-focused queries**](/platform/features/sql-console/query-examples/compliance-focused-queries): Verify that cloud resources meet regulatory and governance requirements. * [**FinOps-focused queries**](/platform/features/sql-console/query-examples/finops-focused-queries): Reduce cloud spending and eliminate waste. Each section provides ready-to-use SQL queries you can run in the [SQL Console](/platform/features/sql-console). --- Source: https://www.cloudquery.io/docs/platform/features/sql-console/query-examples/security-focused-queries # Security-Focused Queries ## Secure Your Cloud with SQL Queries CloudQuery lets you query your cloud asset inventory directly to identify vulnerabilities and misconfigurations. The [Insights](/platform/features/insights) feature surfaces many of these security findings automatically from supported sources. ### Security Queries in Action The following queries help you identify and address common security risks.
    Find public S3 buckets (AWS) **Why it matters:** Public S3 buckets can expose sensitive data and lead to breaches. ```sql SELECT name, JSONExtractBool(policy_status, 'IsPublic') as is_public FROM aws_s3_buckets WHERE is_public = TRUE ```
    Detect AWS lambda functions using deprecated runtimes (AWS) **Why it matters:** Deprecated runtimes introduce security risks and compatibility issues. ```sql SELECT arn, region, tags, JSONExtractString(configuration, 'Runtime') as runtime FROM aws_lambda_functions WHERE runtime in ('nodejs10.x', 'python2.7', 'dotnetcore2.1', 'ruby2.5', 'go1.x') ```
    Identify expired SSL certificates (AWS) **Why it matters:** Expired SSL certificates can lead to service disruptions and security vulnerabilities. ```sql SELECT account_id, arn, domain_name, not_after AS expiry_date, status, NOW() AS current_time, not_after < NOW() AS is_expired FROM aws_acm_certificates ORDER BY not_after ASC; ```
    List EC2 instances authorizing SSH from anywhere (AWS) **Why it matters:** Exposed SSH ports allow unauthorized access and increase security risks. ```sql SELECT e.arn as instance_arn, e._cq_name as name, e.region as instance_region, e.account_id as account_id, arrayJoin(JSONExtractArrayRaw(assumeNotNull(security_groups))) AS security_group, JSONExtractString(security_group, 'GroupId') AS instance_group_id, arrayJoin(JSONExtractArrayRaw(assumeNotNull(s.ip_permissions))) as ip_permission, JSONExtractInt(ip_permission, 'FromPort') AS from_port, JSONExtractInt(ip_permission, 'ToPort') AS to_port, JSONExtractArrayRaw(ip_permission, 'IpRanges') AS ip_ranges, JSONExtractArrayRaw(ip_permission, 'Ipv6Ranges') AS ipv6_ranges, arrayJoin(empty(ip_ranges) ? [''] : ip_ranges) as ip_range, arrayJoin(empty(ipv6_ranges) ? [''] : ipv6_ranges) as ipv6_range, JSONExtractString(ip_range, 'CidrIp') as cidr, JSONExtractString(ipv6_range, 'CidrIp') as ipv6_cidr FROM aws_ec2_instances as e JOIN aws_ec2_security_groups as s ON instance_group_id = s.group_id WHERE from_port = 22 AND to_port = 22 AND (cidr = '0.0.0.0/0' OR ipv6_cidr = '::/0') ```
    Find IAM users without MFA enabled (AWS) **Why it matters:** Lack of MFA increases the risk of account compromise. ```sql SELECT u.* FROM aws_iam_users AS u LEFT JOIN aws_iam_mfa_devices AS m ON u.user_name = m.user_name WHERE m.user_name IS NULL; ```
    ## More query examples See the [query examples overview](./) for compliance and FinOps-focused queries. ## Next Steps - [Compliance-Focused Queries](/platform/features/sql-console/query-examples/compliance-focused-queries) - Compliance monitoring queries - [FinOps-Focused Queries](/platform/features/sql-console/query-examples/finops-focused-queries) - Cost optimization queries - [Policies](/platform/features/policies) - Turn security queries into automated compliance rules - [Alerts](/platform/features/alerts) - Get notified when security conditions are detected --- Source: https://www.cloudquery.io/docs/platform/integration-guides/aws-onboarding-wizard # AWS Onboarding Wizard The AWS onboarding wizard creates a **unified AWS integration** that consolidates asset inventory, cost and usage reporting, and additional metadata into a single setup flow. One [CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/) deployment provisions all required IAM roles and OIDC trust, and CloudQuery automatically creates separate syncs for each feature you enable. If you need to define the IAM roles and policies manually, see [AWS (Manual Setup)](/docs/platform/integration-guides/setting-up-an-aws-integration) and [AWS Cost & Usage](/docs/platform/integration-guides/setting-up-an-awscur-integration). ## Prerequisites - A CloudQuery Platform account with admin access - AWS console access with permissions to create [CloudFormation stacks](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/) and [IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) - If using multi-account mode: access to the [AWS Organizations](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_ous.html) management account - If enabling Cost Metrics: access to [AWS Billing and Cost Management](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-what-is.html) The CloudFormation stack creates IAM roles with [ReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ReadOnlyAccess.html) permissions. Review the stack template before deploying to confirm the permissions meet your organization's security requirements. ## Step 1: Select configuration mode 1. Navigate to **Data Pipelines** → **Integrations**. 2. Click **Create Integration** and select **AWS**. 3. Choose how to configure what CloudQuery syncs from your AWS account. ![AWS Setup Wizard — configuration mode selector](/images/platform/integration-guides/aws-onboarding-wizard/aws-1.png) - **All features**: enables all three features automatically — Asset Inventory, Cost Metrics, and Additional Metadata. Recommended for most users. - **Individual features**: lets you choose which features to enable from the list below. - **Manual services and tables configuration**: lets you select individual AWS services or specific tables to sync. Does not set up Cost Metrics. If you selected **Individual features**, choose from the following: | Feature | What it syncs | Sync created | | ----------------------- | -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | **Asset Inventory** | 301 core AWS resource tables (EC2, IAM, RDS, S3, Lambda, etc.) plus CloudTrail write events | Two syncs: `{integration name} - Asset Inventory` and `{integration name} - CloudTrail` | | **Cost Metrics** | AWS Cost and Usage Report (CUR 2.0) data via BCM Data Exports | One sync: `{integration name} - Cost and Usage` using the `cloudquery/awscur` plugin | | **Additional Metadata** | 68 supplementary tables for snapshots, policies, attachments, and account credentials — used by Reports and Insights | One sync: `{integration name} - Additional Metadata` | If you selected **Manual services and tables configuration**, choose the AWS services or individual tables you want to sync. Cost Metrics are not available in this mode. **All features** is recommended for most users. It enables the Asset Inventory view, cost analysis, and all Reports and Insights functionality out of the box. ## Step 2: Choose authentication method and account scope Select how CloudQuery authenticates with AWS, and whether you are syncing a single account or an entire organization. ![Authentication method + account scope selection screen, showing Guided (CloudFormation) vs Manual options alongside Single Account vs Multiple Accounts options](/images/platform/integration-guides/aws-onboarding-wizard/aws-2.png) **Authentication method:** - **Guided**: CloudQuery generates a CloudFormation template that provisions all required IAM roles and OIDC trust automatically based on the features selected in the previous step. Recommended for most users. - **Manual**: Provide your own IAM role ARNs. Use this if your organization manages IAM roles through infrastructure-as-code (Terraform, CDK) or restricts CloudFormation usage. See [AWS (Manual Setup)](/docs/platform/integration-guides/setting-up-an-aws-integration) for instructions on how to create the required IAM roles. **Account scope:** - **Multiple accounts**: for [AWS Organizations](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_ous.html) with multiple member accounts. CloudQuery discovers your organizational structure and provisions roles across accounts automatically. - **Single account**: for a standalone AWS account. ## Step 3: Deploy the CloudFormation stack (guided only) If you chose **Guided**, a single CloudFormation stack provisions everything required for your selected features. 1. First, in a new browser tab, make sure you are logged in to AWS Console with the account that will be deploying the CloudFormation stack. If you selected to sync from **multiple accounts** and want to sync an organization, log in either with a root account or a delegated admin account. 2. If you are syncing **multiple accounts**, your management account must have trusted access enabled between CloudFormation and AWS Organizations. You can enable it via AWS Organizations → Services → CloudFormation StackSets or by running: ``` aws organizations enable-aws-service-access --service-principal member.org.stacksets.cloudformation.amazonaws.com ``` 3. Click **Open AWS console**. This opens the AWS CloudFormation console with a pre-configured stack template. 4. Review the stack parameters. The stack pre-fills the OIDC trust relationship parameters (audience, issuer URL, subject) for CloudQuery. 5. Check the **I acknowledge that AWS CloudFormation might create IAM resources with custom names** checkbox and click **Create stack**. Depending on the features you selected, the stack creates: - An IAM management role (for organization-wide account discovery, if multi-account) - IAM sync roles used by CloudQuery when running syncs - An OIDC identity provider (trust relationship with CloudQuery Platform) - CUR-specific IAM roles and S3 bucket access (if Cost Metrics is enabled) CloudQuery Platform polls for the stack deployment status and updates the wizard automatically when the stack finishes deploying. If you chose **Manual**, enter the IAM role ARN(s) you have provisioned in your AWS account and click **Continue**. ## Step 4: Select organizational units (multi-account only) If you chose **Multiple accounts**, the wizard displays your AWS organization structure as a tree after the CloudFormation stack deploys: 1. Expand organizational units to see child OUs and accounts. 2. Select the organizational units you want CloudQuery to sync from. 3. Click **Submit** to provision IAM roles for the selected OUs. CloudQuery creates member roles in each account within the selected organizational units and displays provisioning status. For **Single account** mode this step is skipped automatically. ## Step 5: Verify CUR access (Cost Metrics only) If you enabled **Cost Metrics**, the wizard verifies that CloudQuery can access the S3 bucket where your CUR 2.0 Parquet export is stored. AWS delivers the first CUR data within 24 hours of creating a new export. If verification succeeds but no data appears after the first sync, wait for AWS to deliver the initial export and re-run the sync. ## Step 6: Test connections CloudQuery runs a connection test for each enabled feature to validate AWS connectivity before creating the syncs. | Feature | Plugin used | | ------------------- | --------------------------------------------------- | | Asset Inventory | `cloudquery/aws` | | Additional Metadata | `cloudquery/aws` (shared test with Asset Inventory) | | Cost Metrics | `cloudquery/awscur` | If any test fails, the wizard shows an error with a suggested fix. Correct the issue and click **Retry**. ## Step 7: Specify the schedule and destination You can specify how often the integration data gets refreshed from the cloud provider. We recommend scheduling syncs to run daily to keep your asset inventory and insights up to date. If you choose to sync to an external destination (other than CloudQuery), the synced data will not be visible in CloudQuery directly. Review your selected configuration and click **Create** to finalize. CloudQuery will start syncing the data. ## Verify the integration After your first syncs complete, browse your AWS resources in the [Asset Inventory](/platform/features/asset-inventory) under the **Compute**, **Storage**, **Identity**, and other categories. You can also open the [SQL Console](/platform/features/sql-console) and run these queries to confirm data arrived: ```sql -- Count synced EC2 instances SELECT count(*) FROM aws_ec2_instances ``` ```sql -- List synced AWS accounts and regions SELECT DISTINCT account_id, region FROM aws_ec2_instances ``` ```sql -- View S3 buckets across accounts SELECT account_id, name, region FROM aws_s3_buckets LIMIT 10 ``` ## Troubleshooting | Issue | Cause | Fix | | -------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CloudFormation stack fails to create | Insufficient permissions | Verify your AWS user has `cloudformation:CreateStack` and `iam:CreateRole` permissions. Check the **Events** tab in the [CloudFormation console](https://console.aws.amazon.com/cloudformation/) for specific error details. | | Stack stuck in `CREATE_IN_PROGRESS` | Large organization or slow webhook | Wait up to 10 minutes. If still in progress, check the CloudFormation console. | | Provisioning fails for member accounts | StackSet deployment issues | Verify your management account has the [CloudFormation StackSets service-linked role](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-prereqs.html). Check that target OUs contain accounts. | | CUR verification fails | S3 bucket inaccessible or export not yet created | Check that the CUR 2.0 Parquet export exists in [AWS Billing > Data Exports](https://us-east-1.console.aws.amazon.com/costmanagement/home#/bcm-data-exports). The automated setup creates the export — wait up to 24 hours for AWS to deliver the first data. | | No CUR data after sync | CUR export not yet delivered | AWS delivers the first CUR export within 24 hours. Wait and re-run the sync. | | OIDC trust error | Trust relationship misconfigured | The wizard auto-configures OIDC. If it fails, check that your CloudQuery Platform deployment has a valid OIDC issuer. See [AWS OIDC documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html). | | Asset Inventory sync has no data | No services selected or role assumption failing | Check the sync logs. Verify the IAM role ARN in the sync source config matches the role deployed by CloudFormation. | | Metadata sync missing tables | Tables managed by the platform | The metadata sync table list is managed centrally. Individual tables cannot be customized in the unified onboarding flow. | ## Re-entering an existing setup If you want to modify any of the existing configuration to add or remove synced tables, navigate to **Data Pipelines** → **Integrations** and edit the individual integration definitions. ## Next steps - Browse synced resources in the [Asset Inventory](/platform/features/asset-inventory) - Run advanced queries in the [SQL Console](/platform/features/sql-console) - See the [AWS integration documentation](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs) for full configuration options and table reference ## Related resources - [AWS CloudFormation documentation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/) - [AWS IAM OIDC identity providers](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html) - [AWS Organizations and organizational units](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_ous.html) - [AWS CUR 2.0 documentation](https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html) - [CloudQuery AWS integration on Hub](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs) --- Source: https://www.cloudquery.io/docs/platform/integration-guides/general-destination-setup-guide # General Destination Setup Guide ## Prerequisites - A CloudQuery Platform account with admin access - Connection credentials for your destination (hostname, port, username, password, or connection string) CloudQuery Platform includes a default ClickHouse database destination. All platform features (including [SQL Console](/platform/features/sql-console) and [Asset Inventory](/platform/features/asset-inventory)) depend on this destination. If you add another destination, keep the default ClickHouse destination as your primary. Additional destinations should serve as auxiliary targets. ## Creating the destination Go to **Data Pipelines** → **Destinations** and click **Create Destination**:
    ![CloudQuery Platform Destinations page with the Create Destination button highlighted](/images/platform/destinations-1.png)
    Search for the name of your destination. In the example below, the search is for PostgreSQL. Click the destination card in the search results.
    ![Search for destination](/images/platform/destinations-2.png)
    On the next screen, enter a **Destination Name** to identify the destination later (if unsure, use the name of the destination).
    ![Destination setup form with a text field for entering a name to identify the destination](/images/platform/destinations-3.png)
    Fill in the required configuration fields for the destination. Most destinations present a guided form with fields for connection details and credentials. Some destinations use a YAML specification field instead of a guided form. For those, store sensitive values as secrets in the **Secrets** section and reference them using placeholders such as **`${MY_SECRET_VALUE}`**. See the destination's documentation for the required format. ## Testing your connection To save the destination, click the **Test and Save** button. The platform validates your configuration by attempting to connect to the destination. During the test: - A progress bar shows the test is running (tests can take up to 60 seconds) - If the test **succeeds**, the destination is saved - If the test **fails**, an error message and logs are displayed below the form ### Common test connection failures | Issue | Cause | Fix | | --------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | Connection refused | Destination is not reachable | Verify the hostname, port, and network connectivity from the platform to the destination | | Authentication error | Invalid credentials | Check the username, password, or connection string in the **Secrets** section | | Permission denied | User lacks write permissions | Verify the database user has permissions to create tables and insert data | | SSL/TLS error | Certificate or encryption mismatch | Check SSL mode settings in the destination configuration and verify the destination's TLS configuration | | Invalid configuration | Syntax error or missing required field | Check that all required fields are filled in correctly; refer to the destination's documentation for the expected format | You can re-run the test after making changes without leaving the page. ## After your first sync Once a sync completes, verify data is arriving at your destination. If you are using the default ClickHouse destination, open the [SQL Console](/platform/features/sql-console) and run a query to confirm tables were created: ```sql SHOW TABLES ``` You can also browse synced resources in the [Asset Inventory](/platform/features/asset-inventory). ## Next steps - [Set up a sync](/platform/syncs/setting-up-a-sync) to schedule when your data is fetched and from which [integration](general-integration-setup-guide) - Browse synced resources in the [Asset Inventory](/platform/features/asset-inventory) - Run advanced queries in the [SQL Console](/platform/features/sql-console) - Find destination-specific configuration options in the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source) --- Source: https://www.cloudquery.io/docs/platform/integration-guides/general-integration-setup-guide # General Integration Setup Guide ## Prerequisites - A CloudQuery Platform account with admin access ## Creating the integration Go to **Data Pipelines** → **Integrations** and click **Create Integration**:
    ![CloudQuery Platform Integrations page with the Create Integration button](/images/platform/integration-1.png)
    Search for the name of your integration. In the example below, the search is for Microsoft Entra ID (Azure AD):
    ![Search for integration](/images/platform/integration-2.png)
    On the next screen, enter an **Integration Name** to identify the integration later (if unsure, use the name of the integration).
    ![Integration setup form with a text field for entering a name to identify the integration](/images/platform/integration-3.png)
    Fill in the required configuration fields for the integration. Most integrations present a guided form with fields for authentication credentials and configuration options. Some integrations use a YAML specification field instead of a guided form. For those, store sensitive values as secrets in the **Secrets** section and reference them using placeholders such as **`${MY_SECRET_VALUE}`**. See the integration's documentation for the required format. ## Testing your connection When you're ready, click **Test and Continue**. The platform creates a test connection job that validates your configuration by attempting to authenticate with the source and fetch a small amount of data. During the test: - A progress bar shows the test is running (tests can take up to 60 seconds) - If the test **succeeds**, the integration is saved and ready to use in a sync - If the test **fails**, an error message is displayed with details about what went wrong ### Common test connection failures | Issue | Cause | Fix | | --------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | Authentication error | Invalid credentials or expired tokens | Verify the secrets in the **Secrets** section match your provider's credentials | | Permission denied | Role or service account lacks required permissions | Check that the IAM role, service account, or service principal has the correct access policies | | Connection timeout | Network connectivity issue | Verify the platform can reach the provider's API endpoints (check firewall rules, VPC configuration, or proxy settings) | | Invalid configuration | Syntax error or missing required field | Check that all required fields are filled in correctly; refer to the integration's documentation for the expected format | You can re-run the test after making changes without leaving the page. ## Select a destination Pick the available destinations from the dropdown. Keep **CloudQuery** if you want to use the features of CloudQuery. If you want t If you want to sync to the CloudQuery default destination or to another existing destination, leave the **CloudQuery** destination selected.
    ![CloudQuery Platform destination selection dropdown](/images/platform/syncs-2.png)
    ## Configure sync schedule You can choose to run syncs Daily, Weekly, Monthly, or set No Schedule to trigger runs manually. Preset schedules run at midnight UTC. The scheduled time is shown below the dropdown.
    ![Sync schedule configuration with Daily, Weekly, Monthly, and No Schedule options](/images/platform/syncs-3.png)
    If you want to run syncs at a different frequency or at a different time, use the **Advanced** option and enter a cron expression. The actual frequency and time of the sync will be explained below the input.
    ![Advanced sync schedule configuration using a cron expression for custom frequency and timing](/images/platform/syncs-4.png)
    ## After your first sync Once you [create a sync](/platform/syncs/setting-up-a-sync) and it completes, verify your data arrived by opening the [SQL Console](/platform/features/sql-console). Run a query against one of the tables your integration syncs, for example: ```sql SELECT count(*) FROM ``` You can find the full list of tables for each integration in the [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source). Browse your synced resources visually in the [Asset Inventory](/platform/features/asset-inventory), which provides a unified view across all connected integrations. ## Next steps - [Set up a sync](/platform/syncs/setting-up-a-sync) to schedule when your data is fetched and to which destinations - Browse synced resources in the [Asset Inventory](/platform/features/asset-inventory) - Run advanced queries in the [SQL Console](/platform/features/sql-console) - See provider-specific guides for detailed setup instructions: - [AWS (Guided Setup)](aws-onboarding-wizard) | [AWS (Manual Setup)](setting-up-an-aws-integration) - [GCP](setting-up-a-gcp-integration) - [Azure](setting-up-an-azure-integration) - [GitHub](setting-up-a-github-integration) - [Kubernetes](setting-up-a-k8s-integration) --- Source: https://www.cloudquery.io/docs/platform/integration-guides/overview # Integration Guides These guides walk you through connecting your cloud providers and services to CloudQuery Platform. Each guide covers authentication setup, configuration, and verification: everything you need to start syncing data into the [Asset Inventory](/platform/features/asset-inventory). ## How integrations work The end-to-end flow for getting data into CloudQuery Platform is: 1. **Create an integration**: authenticate with your cloud provider and configure which resources to sync 2. **Create a sync**: schedule how often data is fetched and to which [destination](/platform/integration-guides/general-destination-setup-guide) 3. **Browse and query**: view resources in the [Asset Inventory](/platform/features/asset-inventory) or run SQL in the [SQL Console](/platform/features/sql-console) ## Choose your integration | Provider | Authentication method | Automated setup | Best for | Guide | | -------------- | ----------------------------- | --------------------------- | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **AWS** | Cross-account IAM role (OIDC) | Yes (CloudFormation wizard) | First-time AWS users, multi-account orgs | [Guided setup](/platform/integration-guides/aws-onboarding-wizard) | | **AWS** | Cross-account IAM role (IRSA) | No | Custom IAM policies, IaC-managed roles | [Manual setup](/platform/integration-guides/setting-up-an-aws-integration) | | **AWS CUR** | Cross-account IAM role | Yes (CloudFormation) | Cost and usage data from AWS Billing | [Guide](/platform/integration-guides/setting-up-an-awscur-integration) | | **GCP** | Service Account (JSON key) | No | GCP projects, orgs, or folders | [Guide](/platform/integration-guides/setting-up-a-gcp-integration) | | **Azure** | Service Principal | No | Azure subscriptions or management groups | [Guide](/platform/integration-guides/setting-up-an-azure-integration) | | **GitHub** | PAT or GitHub App | No | Repositories, issues, PRs, org data | [Guide](/platform/integration-guides/setting-up-a-github-integration) | | **Kubernetes** | Cloud provider credentials | No | EKS, AKS, or GKE clusters | [EKS](/platform/integration-guides/setting-up-a-k8s-integration/aws-eks) · [AKS](/platform/integration-guides/setting-up-a-k8s-integration/azure-aks) · [GKE](/platform/integration-guides/setting-up-a-k8s-integration/gcp-gke) | For providers not listed above, see the [General Integration Setup Guide](/platform/integration-guides/general-integration-setup-guide). CloudQuery supports [hundreds of integrations](https://www.cloudquery.io/hub/plugins/source); the guides above cover the most common cloud providers with provider-specific authentication steps. ## Recommended starting path If you're setting up CloudQuery Platform for the first time: 1. **AWS users:** Start with the [AWS Guided Setup](/platform/integration-guides/aws-onboarding-wizard). The CloudFormation wizard handles IAM role creation automatically, with no CLI commands needed. 2. **GCP users:** Follow the [GCP guide](/platform/integration-guides/setting-up-a-gcp-integration). You need a service account with the Viewer role and a JSON key file. 3. **Azure users:** Follow the [Azure guide](/platform/integration-guides/setting-up-an-azure-integration). You need a service principal created via the Azure CLI. 4. **Multi-cloud:** Set up each provider separately using the guides above, then create syncs for each. All resources appear together in the [Asset Inventory](/platform/features/asset-inventory). ## After setup Once your integration is configured and your first sync completes: - **Asset Inventory**: Browse all synced resources in one place at [Asset Inventory](/platform/features/asset-inventory). Filter by provider, resource type, region, tags, and more. - **SQL Console**: Run [ClickHouse SQL queries](/platform/features/sql-console) against your synced data for security analysis, compliance checks, or cost optimization. - **Policies**: Set up automated [policies](/platform/features/policies) to continuously evaluate your cloud resources against best practices. - **Alerts**: Configure [alerts](/platform/features/alerts) to get notified when queries return results that need attention. ## General guides These guides cover setup steps that apply to any integration or destination, regardless of the specific provider: - [General Integration Setup Guide](/platform/integration-guides/general-integration-setup-guide): step-by-step walkthrough for configuring any integration - [General Destination Setup Guide](/platform/integration-guides/general-destination-setup-guide): how to add destinations beyond the default ClickHouse database --- Source: https://www.cloudquery.io/docs/platform/integration-guides/setting-up-a-gcp-integration # Setting up a GCP Integration CloudQuery Platform authenticates with GCP through [Service Accounts](https://cloud.google.com/iam/docs/service-accounts-create) using JSON key files. The GCP integration supports additional authentication methods (including [Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) and [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation)) when used with the [CloudQuery CLI](/cli). CloudQuery Platform does not yet support these methods through the UI. See the [GCP integration documentation](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs) for the full list of authentication options. ## Prerequisites - A CloudQuery Platform account with admin access - A [GCP project](https://cloud.google.com/resource-manager/docs/creating-managing-projects) with permissions to create service accounts and manage IAM - Access to the [GCP Console](https://console.cloud.google.com/) ## Step 1: Set up a service account CloudQuery uses a service account to read resources from your GCP environment. Follow these steps to set up a new service account with read-only access: 1. Open the [GCP Service Accounts page](https://console.cloud.google.com/iam-admin/serviceaccounts) 2. Select the project to create the service account in (you can assign access to other projects later) 3. Click **Create Service Account** 4. Enter the details: 1. Service account display name, e.g. `CloudQuery Readonly` 2. Service account ID, e.g. `cloudquery-readonly` 3. A description, e.g. `Service account for CloudQuery to fetch resources in GCP` 4. Click **Create and Continue**
    ![Setting up a service account](/images/platform/setting-up-a-gcp-integration-1.png)
    5. Under **Basic**, select the [`Viewer`](https://cloud.google.com/iam/docs/understanding-roles#basic) role for the service account.
    ![Selecting the Viewer role](/images/platform/setting-up-a-gcp-integration-2.png)
    6. Click **Continue** and **Done**. 7. In the service accounts list, click on the new service account, then go to the **Keys** tab. Click **Add Key** → **Create New Key**. 8. Select **JSON** and click **Create**. This downloads a JSON key file to your computer. You need this file when configuring the integration in CloudQuery Platform.
    ![Getting a private key](/images/platform/setting-up-a-gcp-integration-3.png)
    Store the JSON key file securely. It contains credentials that grant access to your GCP resources. You will upload the contents to CloudQuery Platform in Step 2, after which you can delete the local file. ### Optional: Assign organization or folder-wide access To sync resources across all your GCP projects, grant the [Viewer role](https://cloud.google.com/iam/docs/understanding-roles#basic) to the service account at the [organization or folder level](https://cloud.google.com/resource-manager/docs/access-control-org): 1. In the GCP Console project selection screen, select your top-level organization (or folder) 2. Go to **IAM and Admin** → **IAM**, and click **Grant Access** 3. Paste the email address of the service account you created above in the **New Principals** text box. Assign the **Viewer** role. 4. Click **Save**
    ![Assign Organization or folder-wide access to the Service Account](/images/platform/setting-up-a-gcp-integration-4.png)
    ### Optional: Assign access to individual projects You can also grant access to specific projects if you don't need organization-wide access: 1. In the GCP Console project selection screen, select the relevant project 2. Go to **IAM and Admin** → **IAM**, and click **Grant Access** 3. Paste the email address of the service account in the **New Principals** text box. Assign the **Viewer** role. 4. Click **Save** ## Step 2: Create the integration 1. In CloudQuery Platform, go to **Data Pipelines** → **Integrations**. Click **Create Integration** and type **GCP** to find the GCP integration.
    ![CloudQuery Platform Create Integration page with GCP selected from the integration search results](/images/platform/setting-up-a-gcp-integration-5.png)
    2. Choose a name for your integration (e.g. `GCP`). 3. Upload your service account key file using the **Upload JSON File** button, or paste the contents of the JSON key file directly into the field provided. 4. By default, **Full auto discovery** is enabled, which syncs all projects the service account has access to. To sync specific projects, disable auto discovery and enter your project IDs, organization IDs, or folder IDs. 5. Click **Continue** to select the GCP services you want to sync. 6. Click **Test and Continue** to verify the setup. After a successful test connection, you can safely delete the JSON key file from your local disk. The credentials are stored securely in CloudQuery Platform. ## What gets synced The GCP integration can sync hundreds of tables across GCP services. Some of the most commonly used tables include: | Category | Tables | Description | | ---------- | -------------------------------------------------------------------------- | ------------------------------ | | Compute | `gcp_compute_instances`, `gcp_compute_disks` | VM instances, persistent disks | | Storage | `gcp_storage_buckets` | Cloud Storage buckets | | Networking | `gcp_compute_networks`, `gcp_compute_firewalls`, `gcp_compute_subnetworks` | VPCs, firewall rules, subnets | | Containers | `gcp_container_clusters` | GKE clusters | | Identity | `gcp_iam_service_accounts` | IAM service accounts | See the [full GCP table list](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs) for all available tables. ## Verify the integration After your first [sync](/platform/syncs/setting-up-a-sync) completes, open the [SQL Console](/platform/features/sql-console) and run these queries to confirm data arrived: ```sql -- Count synced compute instances SELECT count(*) FROM gcp_compute_instances ``` ```sql -- List synced GCP projects SELECT DISTINCT project_id FROM gcp_compute_instances ``` ```sql -- View storage buckets SELECT project_id, name, location FROM gcp_storage_buckets LIMIT 10 ``` You can also browse your GCP resources in the [Asset Inventory](/platform/features/asset-inventory) under the **Compute**, **Storage**, **Networking**, and other categories. ## Troubleshooting | Issue | Cause | Fix | | -------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Invalid JSON key | Malformed or incomplete JSON | Verify you copied the entire contents of the JSON key file, including the opening `{` and closing `}`. Re-download the key from GCP if needed. | | Permission denied | Service account lacks Viewer role | Verify the service account has the [Viewer role](https://cloud.google.com/iam/docs/understanding-roles#basic) on the project, folder, or organization you want to sync. | | Project not found | Service account not granted access | The service account can only access projects where it has been granted IAM permissions. Grant the Viewer role on each project, or at the organization/folder level. | | API not enabled | Required GCP API is disabled | Some resources require specific APIs to be enabled in the project (e.g., Compute Engine API, Cloud Storage API). Enable them in the [GCP API Library](https://console.cloud.google.com/apis/library). | | No data for some resources | Services not selected | Check that the relevant services are selected in the **Select services** step. | ## Next steps - [Set up a sync](/platform/syncs/setting-up-a-sync) to schedule when your GCP data is fetched - Browse synced resources in the [Asset Inventory](/platform/features/asset-inventory) - Run advanced queries in the [SQL Console](/platform/features/sql-console) - See the [GCP integration documentation](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs) for full configuration options and table reference ## Related resources - [GCP Service Accounts documentation](https://cloud.google.com/iam/docs/service-accounts-create) - [GCP IAM roles overview](https://cloud.google.com/iam/docs/understanding-roles#basic) - [GCP Organization-level IAM](https://cloud.google.com/resource-manager/docs/access-control-org) - [GCP Service Account keys](https://cloud.google.com/iam/docs/keys-create-delete) - [CloudQuery GCP integration on Hub](https://www.cloudquery.io/hub/plugins/source/cloudquery/gcp/latest/docs) --- Source: https://www.cloudquery.io/docs/platform/integration-guides/setting-up-a-github-integration # Setting up a GitHub Integration The GitHub integration supports two authentication methods: Personal Access Token (PAT) and GitHub App. Choose the method that best aligns with your organization's security requirements. ## Prerequisites - A CloudQuery Platform account with admin access - A GitHub account with access to the repositories or organizations you want to sync - For GitHub App authentication: organization admin access to create and install apps GitHub Apps have higher [rate limits](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/rate-limits-for-github-apps) than personal access tokens. For organizations with many repositories, GitHub App authentication is recommended. ## Personal Access Token authentication ### Generate a token 1. Go to your GitHub account **Settings** 2. Navigate to **Developer settings** → **Personal access tokens** → **Tokens (classic)** 3. Click **Generate new token** and select **Generate new token (classic)** 4. Name your token (e.g. `cloudquery-readonly`) 5. Set an expiration period 6. Select the following scopes based on the data you want to sync: | Scope | Required for | | ------------------ | --------------------------------------------------- | | `repo` (read-only) | Repository metadata, issues, pull requests, commits | | `read:org` | Organization members, teams, settings | | `read:user` | User profile information | | `read:project` | GitHub Projects data | | `read:packages` | GitHub Packages metadata | 7. Click **Generate token** and copy the token value. For detailed instructions, see the [GitHub personal access token documentation](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens). ### Create the integration with PAT 1. In CloudQuery Platform, go to **Data Pipelines** → **Integrations**. Click **Create Integration** and type **GitHub** to find the GitHub integration.
    ![Find GitHub Integration](/images/platform/gh-integration-1.png)
    2. Choose a name for your integration. Under **Select authentication type**, choose **Access Token**, then enter your personal access token in the **Access Token** field. 3. Under **Configuration**, select **Organizations** or **Repositories** and enter the relevant names. 4. Click **Continue** to select the tables you want to sync. 5. Click **Test and Continue** to verify the setup. ## GitHub App authentication ### Create a GitHub App GitHub App authentication provides higher rate limits and fine-grained permissions. For detailed setup instructions, see the [GitHub App registration documentation](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app). 1. Go to your **Organization Settings** → **Developer settings** → **GitHub Apps** → **New GitHub App** 2. Configure the app: - Set a name (e.g. `CloudQuery Integration`) - Set the homepage URL to your organization's URL - Deselect **Webhook** (not needed) - Under **Permissions**, grant read-only access to the resources you want to sync (e.g., **Repository**: Contents, Issues, Pull requests, Metadata; **Organization**: Members) 3. Click **Create GitHub App** 4. On the app settings page, note the **App ID** 5. Scroll down to **Private keys** and click **Generate a private key**. This downloads a `.pem` file. 6. Go to **Install App** in the left sidebar and install it on your organization. Note the **Installation ID** from the URL (e.g., `https://github.com/settings/installations/`) For a full list of permissions required per resource type, see the [GitHub App permissions reference](https://docs.github.com/en/rest/overview/permissions-required-for-github-apps). ### Create the integration with GitHub App 1. In CloudQuery Platform, go to **Data Pipelines** → **Integrations**. Click **Create Integration** and type **GitHub** to find the GitHub integration. 2. Choose a name for your integration. **Github app authentication** is selected by default. Enter your **App ID**, **Installation ID**, and **Private Key** in the fields provided. Paste the entire contents of the `.pem` file into the **Private Key** field. 3. Under **Configuration**, select **Organizations** or **Repositories** and enter the relevant names. 4. Click **Continue** to select the tables you want to sync. 5. Click **Test and Continue** to verify the setup. ## What gets synced The GitHub integration syncs repository, organization, and security data. Some of the most commonly used tables include: | Category | Tables | Description | | ------------ | -------------------------------------------------------------- | ------------------------------------------ | | Repositories | `github_repositories` | Repository metadata, settings, visibility | | Issues & PRs | `github_issues`, `github_pull_requests` | Issues, pull requests, and their metadata | | CI/CD | `github_workflow_runs` | GitHub Actions workflow run history | | Security | `github_code_scanning_alerts`, `github_secret_scanning_alerts` | Code scanning and secret scanning findings | | Organization | `github_organization_members_with_role` | Org members and their roles | See the [full GitHub table list](https://www.cloudquery.io/hub/plugins/source/cloudquery/github/latest/docs) for all available tables. ## Verify the integration After your first [sync](/platform/syncs/setting-up-a-sync) completes, open the [SQL Console](/platform/features/sql-console) and run these queries to confirm data arrived: ```sql -- Count synced repositories SELECT count(*) FROM github_repositories ``` ```sql -- List synced organizations SELECT DISTINCT org FROM github_repositories ``` ```sql -- View recent issues SELECT repository_full_name, title, state, created_at FROM github_issues ORDER BY created_at DESC LIMIT 10 ``` ```sql -- Check pull request activity SELECT repository_full_name, title, state, created_at FROM github_pull_requests ORDER BY created_at DESC LIMIT 10 ``` You can also browse your GitHub resources in the [Asset Inventory](/platform/features/asset-inventory). ## Troubleshooting | Issue | Cause | Fix | | ------------------- | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `401 Unauthorized` | Invalid or expired token | Generate a new PAT or rotate the GitHub App private key. Verify the secret value in CloudQuery Platform matches. | | `403 Forbidden` | Insufficient token scopes | Check that the PAT has the required scopes (see the scopes table above). For GitHub Apps, verify the app has the necessary [permissions](https://docs.github.com/en/rest/overview/permissions-required-for-github-apps). | | Rate limit exceeded | Too many API requests | GitHub Apps have higher [rate limits](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/rate-limits-for-github-apps) than PATs. Consider switching to GitHub App authentication. You can also reduce the number of tables or repositories synced. | | `404 Not Found` | Repository or organization not accessible | Verify the **Repositories** or **Organizations** values in the form match existing repositories/organizations that the token or app has access to. | | No data after sync | Empty repositories or organizations list | Verify that at least one **Repository** is entered in `owner/repo` format, or switch to **Organizations** and enter at least one organization name. | ## Next steps - [Set up a sync](/platform/syncs/setting-up-a-sync) to schedule when your GitHub data is fetched - Browse synced resources in the [Asset Inventory](/platform/features/asset-inventory) - Run advanced queries in the [SQL Console](/platform/features/sql-console) - See the [GitHub integration documentation](https://www.cloudquery.io/hub/plugins/source/cloudquery/github/latest/docs) for full configuration options and table reference ## Related resources - [GitHub personal access token documentation](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) - [GitHub App registration documentation](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app) - [GitHub App rate limits](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/rate-limits-for-github-apps) - [GitHub App permissions reference](https://docs.github.com/en/rest/overview/permissions-required-for-github-apps) - [CloudQuery GitHub integration on Hub](https://www.cloudquery.io/hub/plugins/source/cloudquery/github/latest/docs) --- Source: https://www.cloudquery.io/docs/platform/integration-guides/setting-up-a-k8s-integration/aws-eks # AWS EKS CloudQuery Platform supports integration with Amazon Elastic Kubernetes Service (EKS). To sync EKS clusters, configure cross-account access and create an [access entry](https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html) for each cluster. ## Prerequisites - A CloudQuery Platform account with admin access - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) installed and configured - An existing [AWS integration](/platform/integration-guides/setting-up-an-aws-integration) configured in CloudQuery Platform (this creates the `cross-account-readonly-role` referenced below) - One or more EKS clusters running in your AWS account The [API server endpoint](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html) of your EKS cluster must be publicly available for the CloudQuery Kubernetes integration to sync resources. Private-only endpoints are not supported. Before starting, set this environment variable: ```bash export TARGET_ACCOUNT_ID="" ``` ## Configure EKS cluster access In the target AWS account, create an IAM [access entry](https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html) and assign an [access policy](https://docs.aws.amazon.com/eks/latest/userguide/access-policies.html) to your EKS cluster. 1. Create an IAM access entry for your EKS cluster: ```bash aws eks create-access-entry \ --cluster-name \ --principal-arn arn:aws:iam::${TARGET_ACCOUNT_ID}:role/cross-account-readonly-role \ --type STANDARD ``` 2. Choose one of the following policies based on the level of access required: **Full cluster view (recommended):** Allows CloudQuery to sync all cluster resources. ```bash aws eks associate-access-policy \ --cluster-name \ --principal-arn arn:aws:iam::${TARGET_ACCOUNT_ID}:role/cross-account-readonly-role \ --access-scope type=cluster \ --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSAdminViewPolicy ```
    Restricted view If you choose this policy, most core cluster resources such as Nodes, RBAC Roles, and Secrets will not be synced. If you don't need access to these resources, you can use a more restrictive view policy: ```bash aws eks associate-access-policy \ --cluster-name \ --principal-arn arn:aws:iam::${TARGET_ACCOUNT_ID}:role/cross-account-readonly-role \ --access-scope type=cluster \ --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy ```
    ## Creating the K8s integration 1. Navigate to **Data Pipelines** → **Integrations** in CloudQuery Platform. 2. Click **Create Integration** and select **K8s**. 3. Select **Amazon Web Services (AWS EKS)** from the **Cloud provider** dropdown. ### Cluster configuration For each EKS cluster, fill in the following fields: - **Cluster name**: the name of your EKS cluster. - **Region**: the AWS region where the cluster is hosted (e.g. `us-east-1`). - **Role ARN**: the principal ARN used to assign policies during the [configure EKS cluster access](#configure-eks-cluster-access) step. - **External ID**: the `EXTERNAL_ID` from the [AWS integration setup guide](/platform/integration-guides/setting-up-an-aws-integration). To sync multiple EKS clusters, click **Add cluster** and repeat the configuration for each cluster. To add clusters from a different cloud provider (Azure or GCP), create a separate K8s integration. 4. Click **Test and Continue** to verify access. ## What gets synced The Kubernetes integration syncs cluster resources across all standard Kubernetes API groups. Some of the most commonly used tables include: | Category | Tables | Description | | ------------- | ----------------------------------------------------------------- | ------------------------------- | | Workloads | `k8s_core_pods`, `k8s_apps_deployments`, `k8s_apps_stateful_sets` | Pods, Deployments, StatefulSets | | Networking | `k8s_core_services`, `k8s_networking_ingresses` | Services, Ingresses | | Configuration | `k8s_core_config_maps`, `k8s_core_secrets` | ConfigMaps, Secrets | | Cluster | `k8s_core_nodes`, `k8s_core_namespaces` | Nodes, Namespaces | | RBAC | `k8s_rbac_roles`, `k8s_rbac_cluster_roles` | Roles, ClusterRoles | See the [full K8s table list](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) for all available tables. ## Verify the integration After your first [sync](/platform/syncs/setting-up-a-sync) completes, open the [SQL Console](/platform/features/sql-console) and run these queries to confirm Kubernetes data arrived: ```sql -- Count synced pods SELECT count(*) FROM k8s_core_pods ``` ```sql -- List namespaces SELECT DISTINCT namespace FROM k8s_core_pods ``` ```sql -- View deployments SELECT namespace, name FROM k8s_apps_deployments LIMIT 10 ``` ```sql -- Check nodes SELECT name FROM k8s_core_nodes ``` You can also browse your Kubernetes resources in the [Asset Inventory](/platform/features/asset-inventory) under the **Containers** category. ## Troubleshooting | Issue | Cause | Fix | | ---------------------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AccessDeniedException` | Access entry not created or wrong principal ARN | Verify the access entry exists with `aws eks list-access-entries --cluster-name `. The principal ARN must match the `cross-account-readonly-role`. | | Cluster unreachable | Private API server endpoint | The EKS cluster must have a [public endpoint](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html) enabled. Check the cluster's networking configuration in the AWS console. | | Missing resources (no Nodes, RBAC) | Restrictive access policy | The `AmazonEKSViewPolicy` does not grant access to Nodes, RBAC, or Secrets. Switch to `AmazonEKSAdminViewPolicy` for full read-only access. | | No data after sync | Access policy not associated | Verify the access policy is associated with `aws eks list-associated-access-policies --cluster-name --principal-arn `. | ## Next steps - [Set up a sync](/platform/syncs/setting-up-a-sync) to schedule when your cluster data is fetched - Browse synced resources in the [Asset Inventory](/platform/features/asset-inventory) - Run advanced queries in the [SQL Console](/platform/features/sql-console) - See the [K8s integration documentation](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) for full table reference ## Related resources - [EKS access entries documentation](https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html) - [EKS access policies documentation](https://docs.aws.amazon.com/eks/latest/userguide/access-policies.html) - [EKS cluster endpoint access](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html) - [CloudQuery K8s integration on Hub](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) --- Source: https://www.cloudquery.io/docs/platform/integration-guides/setting-up-a-k8s-integration/azure-aks # Azure AKS CloudQuery Platform supports integration with Azure Kubernetes Service (AKS). To sync AKS clusters, assign the [Azure Kubernetes Service Cluster User Role](https://learn.microsoft.com/en-us/azure/aks/manage-azure-rbac) to your service principal. ## Prerequisites - A CloudQuery Platform account with admin access - [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) installed and configured - An existing [Azure integration](/platform/integration-guides/setting-up-an-azure-integration) configured in CloudQuery Platform (this creates the service principal referenced below) - One or more AKS clusters running in your Azure subscription ## Configure AKS cluster access Assign the `Azure Kubernetes Service Cluster User Role` to the service principal created in the [Azure integration setup guide](/platform/integration-guides/setting-up-an-azure-integration). Using the Azure CLI: ```bash az role assignment create \ --assignee "" \ --role "Azure Kubernetes Service Cluster User Role" \ --scope "/subscriptions/" ``` - ``: the `appId` from the service principal you created in the Azure integration setup guide. - ``: the Azure subscription where your AKS cluster is deployed. For more details on AKS access control, see the [AKS identity and access management documentation](https://learn.microsoft.com/en-us/azure/aks/concepts-identity). ## Creating the K8s integration 1. Navigate to **Data Pipelines** → **Integrations** in CloudQuery Platform. 2. Click **Create Integration** and select **K8s**. 3. Select **Microsoft Azure (Azure AKS)** from the **Cloud provider** dropdown. ### Cluster configuration For each AKS cluster, fill in the following fields: - **Cluster name**: the name of your AKS cluster. - **Client ID**: the `appId` from your service principal. - **Tenant ID**: the `tenant` from your service principal. - **Client Secret**: the `password` from your service principal. - **Subscription ID**: the Azure subscription ID where the cluster is deployed. - **Resource group name**: the Azure resource group containing the AKS cluster. To sync multiple AKS clusters, click **Add cluster** and repeat the configuration for each cluster. To add clusters from a different cloud provider (AWS or GCP), create a separate K8s integration. 4. Click **Test and Continue** to verify access. ## What gets synced The Kubernetes integration syncs cluster resources across all standard Kubernetes API groups. Some of the most commonly used tables include: | Category | Tables | Description | | ------------- | ----------------------------------------------------------------- | ------------------------------- | | Workloads | `k8s_core_pods`, `k8s_apps_deployments`, `k8s_apps_stateful_sets` | Pods, Deployments, StatefulSets | | Networking | `k8s_core_services`, `k8s_networking_ingresses` | Services, Ingresses | | Configuration | `k8s_core_config_maps`, `k8s_core_secrets` | ConfigMaps, Secrets | | Cluster | `k8s_core_nodes`, `k8s_core_namespaces` | Nodes, Namespaces | | RBAC | `k8s_rbac_roles`, `k8s_rbac_cluster_roles` | Roles, ClusterRoles | See the [full K8s table list](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) for all available tables. ## Verify the integration After your first [sync](/platform/syncs/setting-up-a-sync) completes, open the [SQL Console](/platform/features/sql-console) and run these queries to confirm Kubernetes data arrived: ```sql -- Count synced pods SELECT count(*) FROM k8s_core_pods ``` ```sql -- List namespaces SELECT DISTINCT namespace FROM k8s_core_pods ``` ```sql -- View deployments SELECT namespace, name FROM k8s_apps_deployments LIMIT 10 ``` ```sql -- Check nodes SELECT name FROM k8s_core_nodes ``` You can also browse your Kubernetes resources in the [Asset Inventory](/platform/features/asset-inventory) under the **Containers** category. ## Troubleshooting | Issue | Cause | Fix | | --------------------------------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Authentication error | Invalid service principal credentials | Verify the **Client ID**, **Client Secret**, and **Tenant ID** match the service principal output. If the secret expired, [create a new one](https://learn.microsoft.com/en-us/cli/azure/azure-cli-sp-tutorial-3). | | `Azure Kubernetes Service Cluster User Role` not assigned | Role assignment missing | Run `az role assignment list --assignee ` to verify the role is assigned. Re-run the `az role assignment create` command if needed. | | Cluster not found | Wrong subscription or resource group | Verify the **Subscription ID** and **Resource group name** match the AKS cluster's location in the Azure portal. | | No data after sync | Cluster not reachable | Verify the AKS cluster is running and the API server is accessible. Check the cluster status in the Azure portal. | ## Next steps - [Set up a sync](/platform/syncs/setting-up-a-sync) to schedule when your cluster data is fetched - Browse synced resources in the [Asset Inventory](/platform/features/asset-inventory) - Run advanced queries in the [SQL Console](/platform/features/sql-console) - See the [K8s integration documentation](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) for full table reference ## Related resources - [AKS identity and access management](https://learn.microsoft.com/en-us/azure/aks/concepts-identity) - [Azure RBAC for AKS](https://learn.microsoft.com/en-us/azure/aks/manage-azure-rbac) - [Azure CLI installation guide](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) - [CloudQuery K8s integration on Hub](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) --- Source: https://www.cloudquery.io/docs/platform/integration-guides/setting-up-a-k8s-integration/gcp-gke # GCP GKE CloudQuery Platform supports integration with Google Kubernetes Engine (GKE). To sync GKE clusters, configure a service account with the [Kubernetes Engine Viewer](https://cloud.google.com/kubernetes-engine/docs/how-to/iam#predefined) role. ## Prerequisites - A CloudQuery Platform account with admin access - An existing [GCP integration](/platform/integration-guides/setting-up-a-gcp-integration) configured in CloudQuery Platform (this creates the service account referenced below) - One or more GKE clusters running in your GCP project - The [Kubernetes Engine Viewer](https://cloud.google.com/kubernetes-engine/docs/how-to/iam#predefined) role assigned to the service account To assign the Kubernetes Engine Viewer role: 1. Open the [GCP IAM page](https://console.cloud.google.com/iam-admin/iam) 2. Find the service account you created in the [GCP integration setup guide](/platform/integration-guides/setting-up-a-gcp-integration) 3. Click **Edit** and add the **Kubernetes Engine Viewer** role 4. Click **Save** You also need the JSON key file for the service account. If you didn't save it from the GCP integration setup, [create a new key](https://cloud.google.com/iam/docs/keys-create-delete) from the service account's **Keys** tab. ## Creating the K8s integration 1. Navigate to **Data Pipelines** → **Integrations** in CloudQuery Platform. 2. Click **Create Integration** and select **K8s**. 3. Select **Google Cloud Platform (GCP GKE)** from the **Cloud provider** dropdown. ### Cluster configuration For each GKE cluster, fill in the following fields: - **Cluster name**: the name of your GKE cluster as shown in the GCP console. - **GCP Project ID**: the GCP project ID that contains the cluster. - **Location (Region)**: the region or zone hosting the cluster (e.g. `us-central1` or `us-central1-a`). - **Service Account Key JSON**: the full contents of the service account JSON key file. To sync multiple GKE clusters, click **Add cluster** and repeat the configuration for each cluster. To add clusters from a different cloud provider (AWS or Azure), create a separate K8s integration. 4. Click **Test and Continue** to verify access. ## Optional: adding permissions to read cluster secrets By default, the `Kubernetes Engine Viewer` role does not allow reading cluster secrets. To sync secrets, either: - Assign the `Kubernetes Engine Admin` role to the service account, or - Create a [custom role](https://cloud.google.com/iam/docs/creating-custom-roles) with the `container.secrets.list` permission ## What gets synced The Kubernetes integration syncs cluster resources across all standard Kubernetes API groups. Some of the most commonly used tables include: | Category | Tables | Description | | ------------- | ----------------------------------------------------------------- | ------------------------------- | | Workloads | `k8s_core_pods`, `k8s_apps_deployments`, `k8s_apps_stateful_sets` | Pods, Deployments, StatefulSets | | Networking | `k8s_core_services`, `k8s_networking_ingresses` | Services, Ingresses | | Configuration | `k8s_core_config_maps`, `k8s_core_secrets` | ConfigMaps, Secrets | | Cluster | `k8s_core_nodes`, `k8s_core_namespaces` | Nodes, Namespaces | | RBAC | `k8s_rbac_roles`, `k8s_rbac_cluster_roles` | Roles, ClusterRoles | See the [full K8s table list](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) for all available tables. ## Verify the integration After your first [sync](/platform/syncs/setting-up-a-sync) completes, open the [SQL Console](/platform/features/sql-console) and run these queries to confirm Kubernetes data arrived: ```sql -- Count synced pods SELECT count(*) FROM k8s_core_pods ``` ```sql -- List namespaces SELECT DISTINCT namespace FROM k8s_core_pods ``` ```sql -- View deployments SELECT namespace, name FROM k8s_apps_deployments LIMIT 10 ``` ```sql -- Check nodes SELECT name FROM k8s_core_nodes ``` You can also browse your Kubernetes resources in the [Asset Inventory](/platform/features/asset-inventory) under the **Containers** category. ## Troubleshooting | Issue | Cause | Fix | | -------------------- | ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Permission denied | Missing Kubernetes Engine Viewer role | Verify the service account has the [Kubernetes Engine Viewer](https://cloud.google.com/kubernetes-engine/docs/how-to/iam#predefined) role on the project containing the cluster. | | Invalid JSON key | Malformed or expired key | Verify the full JSON key was pasted correctly. If the key has been deleted, [create a new one](https://cloud.google.com/iam/docs/keys-create-delete) from the GCP console. | | Cluster not found | Wrong project ID or location | Verify the **GCP Project ID** and **Location** match the cluster's settings in the GCP console. For zonal clusters, use the full zone (e.g. `us-central1-a`), not the region. | | Missing secrets data | Insufficient permissions | The Kubernetes Engine Viewer role cannot read secrets. Assign the Kubernetes Engine Admin role or a custom role with `container.secrets.list`. | ## Next steps - [Set up a sync](/platform/syncs/setting-up-a-sync) to schedule when your cluster data is fetched - Browse synced resources in the [Asset Inventory](/platform/features/asset-inventory) - Run advanced queries in the [SQL Console](/platform/features/sql-console) - See the [K8s integration documentation](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) for full table reference ## Related resources - [GKE IAM documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/iam) - [GKE predefined IAM roles](https://cloud.google.com/kubernetes-engine/docs/how-to/iam#predefined) - [GCP Service Account keys](https://cloud.google.com/iam/docs/keys-create-delete) - [CloudQuery K8s integration on Hub](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) --- Source: https://www.cloudquery.io/docs/platform/integration-guides/setting-up-a-k8s-integration # Setting up a K8s Integration The Kubernetes integration syncs cluster resources (including pods, deployments, services, namespaces, nodes, RBAC roles, config maps, and more) into CloudQuery Platform. This gives you visibility into your Kubernetes infrastructure alongside your cloud resources. Before setting up a K8s integration, you need an existing cloud provider integration configured for the account where your cluster runs. See the [AWS](/platform/integration-guides/setting-up-an-aws-integration), [Azure](/platform/integration-guides/setting-up-an-azure-integration), or [GCP](/platform/integration-guides/setting-up-a-gcp-integration) integration guides. ## Choose your cloud provider - [AWS EKS integration guide](/platform/integration-guides/setting-up-a-k8s-integration/aws-eks): for Amazon Elastic Kubernetes Service clusters - [Azure AKS integration guide](/platform/integration-guides/setting-up-a-k8s-integration/azure-aks): for Azure Kubernetes Service clusters - [GCP GKE integration guide](/platform/integration-guides/setting-up-a-k8s-integration/gcp-gke): for Google Kubernetes Engine clusters ## Next steps - [Set up a sync](/platform/syncs/setting-up-a-sync) to schedule when your Kubernetes data is fetched - Browse synced resources in the [Asset Inventory](/platform/features/asset-inventory) under the **Containers** category - Run advanced queries in the [SQL Console](/platform/features/sql-console) - See the [K8s integration documentation](https://www.cloudquery.io/hub/plugins/source/cloudquery/k8s/latest/docs) for full configuration options and table reference --- Source: https://www.cloudquery.io/docs/platform/integration-guides/setting-up-an-aws-integration # Setting up an AWS Integration CloudQuery Platform authenticates with AWS through cross-account IAM roles using [IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (IRSA). This guide walks through the manual setup process. Use this if your organization manages IAM roles through infrastructure-as-code (Terraform, CDK) or restricts CloudFormation usage. For an automated approach using CloudFormation, see the [AWS Onboarding Wizard](aws-onboarding-wizard). The AWS accounts involved are: - **CloudQuery Account:** The AWS account where CloudQuery Platform is deployed. This account hosts the IAM role that CloudQuery uses to assume roles in other accounts. - **Your Account:** The AWS account you want to sync resources from. You create a role here that allows the CloudQuery account's role to assume it and read resources. ## Prerequisites - A CloudQuery Platform account with admin access - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) installed and configured with credentials that have IAM administrative permissions - Your AWS account ID - **`EXTERNAL_ID`**: Generate this yourself (e.g., by running `uuidgen`). An external ID is recommended by [AWS best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html) to provide an additional verification layer when assuming roles in a third-party account. It can be any alphanumeric string between 2 and 1224 characters. ## Step 1: Select configuration mode 1. Navigate to **Data Pipelines** → **Integrations**. 2. Click **Create Integration** and select **AWS**. 3. Choose how to configure what CloudQuery syncs from your AWS account. ![AWS Setup Wizard — configuration mode selector](/images/platform/integration-guides/aws-onboarding-wizard/aws-1.png) - **All features**: enables all three features automatically — Asset Inventory, Cost Metrics, and Additional Metadata. Recommended for most users. - **Individual features**: lets you choose which features to enable from the list below. - **Manual services and tables configuration**: lets you select individual AWS services or specific tables to sync. Does not set up Cost Metrics. If you selected **Individual features**, choose from the following: | Feature | What it syncs | Sync created | | ----------------------- | -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | **Asset Inventory** | 301 core AWS resource tables (EC2, IAM, RDS, S3, Lambda, etc.) plus CloudTrail write events | Two syncs: `{integration name} - Asset Inventory` and `{integration name} - CloudTrail` | | **Cost Metrics** | AWS Cost and Usage Report (CUR 2.0) data via BCM Data Exports | One sync: `{integration name} - Cost and Usage` using the `cloudquery/awscur` plugin | | **Additional Metadata** | 68 supplementary tables for snapshots, policies, attachments, and account credentials — used by Reports and Insights | One sync: `{integration name} - Additional Metadata` | If you selected **Manual services and tables configuration**, choose the AWS services or individual tables you want to sync. Cost Metrics are not available in this mode. **All features** is recommended for most users. It enables the Asset Inventory view, cost analysis, and all Reports and Insights functionality out of the box. ## Step 2: Choose authentication method and account scope Select **Manual** authentication method. This will allow you to provide your own IAM role ARNs. ![AWS Setup Wizard — manual setup](/images/platform/integration-guides/aws-onboarding-wizard/aws-3.png) ### IAM role and permissions Deploy the read-only role and Cost & Usage export with one CloudFormation stack instead of running the commands in the other tab. The stack creates a role that trusts the role used by CloudQuery. Replace the `` in the command below with a value you generate (e.g. `uuidgen`). Replace the `` with the role ARN displayed in the side panel documentation on the integration UI page in CloudQuery Platform. ```shell aws s3 cp s3://cq-cloud-cloudformation-templates-us-east-1/onboarding/unified/self-service-template.json self-service-template.json && \ aws cloudformation deploy \ --stack-name cloudquery-readonly \ --template-file self-service-template.json \ --capabilities CAPABILITY_NAMED_IAM \ --region us-east-1 \ --parameter-overrides \ CloudQueryRoleArn= \ ExternalId= \ EnableAssetInventory=true \ EnableCostAndUsage=true ``` When the stack reaches CREATE_COMPLETE, read its outputs: ```shell aws cloudformation describe-stacks \ --stack-name cloudquery-readonly \ --region us-east-1 \ --query "Stacks[0].Outputs" --output table ``` Copy **RoleArn** and **CurReportName** from the outputs into the **Role ARN** and **CUR report name** fields, and enter the same **External ID** you used in the command. Cost & Usage is single-account, so deploy the stack in that one account only. 1. Create the trust relationship for the cross-account role: Replace `` with the CloudQuery AWS Account ID found in the side panel documentation on the integration UI page in CloudQuery Platform. Replace `` with your generated external ID. ```shell cat >third-party-trust.json <" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } EOF ``` 2. Create the cross-account role and attach the ReadOnly policy: ```shell aws iam create-role \ --role-name cross-account-readonly-role \ --assume-role-policy-document file://third-party-trust.json aws iam attach-role-policy \ --role-name cross-account-readonly-role \ --policy-arn="arn:aws:iam::aws:policy/ReadOnlyAccess" ``` The trust policy above only authorises who may assume the role. The ReadOnlyAccess attachment is a separate, identity-based policy that grants what the role can read once assumed. Never put service actions like s3:GetObject or bcm-data-exports:\* into the trust policy - AWS rejects it with "AssumeRole policy may only specify STS AssumeRole actions". Cost & Usage uses AWS BCM Data Exports (the CUR 2.0 service launched in late 2023). The AWS-managed ReadOnlyAccess policy does not grant bcm-data-exports:\*, so the connection test fails with AccessDeniedException on bcm-data-exports:ListExports unless you attach an additional permissions policy. Attach this inline policy to the cross-account role (NOT to the trust policy): ```shell cat >cur-extra-permissions.json <:role/cross-account-readonly-role`). Replace `` with your AWS account ID. **External ID** - The same external ID you used in the trust relationship when creating the role. Cost & Usage Reports are single-account, so you provide a single role ARN / external ID pair. 3. Click **Test and Continue** to verify the setup. 4. Specify the schedule and destination You can specify how often the integration data gets refreshed from the cloud provider. We recommend scheduling syncs to run daily to keep your asset inventory and insights up to date. If you choose to sync to an external destination (other than CloudQuery), the synced data will not be visible in CloudQuery directly. Review your selected configuration and click **Create** to finalize. CloudQuery will start syncing the data. ## What gets synced The AWS integration can sync over 500 tables across all major AWS services. Some of the most commonly used tables include: | Category | Tables | Description | | ---------- | ------------------------------------------------------------ | ------------------------------- | | Compute | `aws_ec2_instances`, `aws_lambda_functions` | EC2 instances, Lambda functions | | Storage | `aws_s3_buckets`, `aws_ebs_volumes` | S3 buckets, EBS volumes | | Networking | `aws_ec2_vpcs`, `aws_ec2_security_groups`, `aws_ec2_subnets` | VPCs, security groups, subnets | | Identity | `aws_iam_roles`, `aws_iam_users`, `aws_iam_policies` | IAM roles, users, policies | | Databases | `aws_rds_instances`, `aws_rds_clusters` | RDS instances and clusters | See the [full AWS table list](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs) for all available tables. ## Verify the integration After your first syncs complete, browse your AWS resources in the [Asset Inventory](/platform/features/asset-inventory) under the **Compute**, **Storage**, **Identity**, and other categories. You can also open the [SQL Console](/platform/features/sql-console) and run these queries to confirm data arrived: ```sql -- Count synced EC2 instances SELECT count(*) FROM aws_ec2_instances ``` ```sql -- List synced AWS accounts and regions SELECT DISTINCT account_id, region FROM aws_ec2_instances ``` ```sql -- View S3 buckets across accounts SELECT account_id, name, region FROM aws_s3_buckets LIMIT 10 ``` ## Troubleshooting | Issue | Cause | Fix | | --------------------------------- | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AccessDenied` when assuming role | Trust relationship misconfigured | Verify the `Principal` ARN in the trust policy matches the CloudQuery account role exactly. Check that `` and `` are correct. | | External ID mismatch | `EXTERNAL_ID` in trust policy differs from the secret | Ensure the `EXTERNAL_ID` value in the IAM trust policy matches the secret value in CloudQuery Platform exactly | | `InvalidIdentityToken` | IRSA configuration issue | Verify the CloudQuery Platform deployment has a valid OIDC provider. See [AWS IRSA documentation](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). | | No data after sync | Tables list is empty or incorrect | In the YAML configuration, check the `tables` field. Use `["*"]` to sync everything, or specify individual table names from the [AWS tables list](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs). | | Partial data (missing accounts) | Member roles not deployed | For multi-account setups, verify the CloudFormation StackSet deployed roles to all target accounts. Run `aws cloudformation describe-stacks` to check. | ## Next steps - [Set up a sync](/platform/syncs/setting-up-a-sync) to schedule when your AWS data is fetched - Browse synced resources in the [Asset Inventory](/platform/features/asset-inventory) - Run advanced queries in the [SQL Console](/platform/features/sql-console) - See the [AWS integration documentation](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs) for full configuration options and table reference ## Related resources - [AWS IAM cross-account access](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html): AWS documentation on external ID best practices - [AWS ReadOnlyAccess policy reference](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/ReadOnlyAccess.html): full list of permissions granted - [AWS CLI installation guide](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) - [AWS STS AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html): API reference for role assumption - [CloudQuery AWS integration on Hub](https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs): full integration documentation --- Source: https://www.cloudquery.io/docs/platform/integration-guides/setting-up-an-awscur-integration # Setting up an AWS Cost and Usage Integration If you are already setting up an AWS integration, you can enable **Cost Metrics** as part of the [AWS Onboarding Wizard](aws-onboarding-wizard) instead of creating a separate integration. The wizard provisions the required CUR roles and S3 access in the same CloudFormation deployment. Use this guide to set up a standalone CUR integration, or if you already have an existing CUR export and need to configure access manually. The AWS Cost and Usage Report (CUR) integration syncs [CUR 2.0](https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html) Parquet data from S3 into your destination. ## Prerequisites - A CloudQuery Platform account with admin access - An AWS account with access to [AWS Billing and Cost Management](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-what-is.html) - For manual setup: [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) installed and configured This integration supports [CUR 2.0](https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html) with **Parquet** format only. Legacy CUR format is not supported. If you have an existing legacy CUR, you need to [create a new CUR 2.0 data export](https://docs.aws.amazon.com/cur/latest/userguide/dataexports-create-standard.html). The initial sync may take significant time depending on the volume of your cost and usage data. Subsequent syncs are incremental and faster. CloudQuery Platform offers two setup methods: - **Automated setup**: CloudQuery provisions an S3 bucket, CUR 2.0 Parquet export, and a scoped read-only IAM role in your AWS account via CloudFormation. - **Manual setup**: You provide access to an existing CUR 2.0 Parquet export by creating a cross-account IAM role yourself. ## Creating the integration 1. Navigate to **Data Pipelines** → **Integrations** in CloudQuery Platform. 2. Click **Create Integration** and select **AWS Cost and Usage**. 3. Choose your setup method: **Automated setup** or **Manual setup**. ### Automated setup Automated setup creates all required AWS resources through a [CloudFormation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/) stack. 1. Click **Open AWS Console**. This opens the AWS CloudFormation console with a pre-configured stack that creates: - An S3 bucket for CUR data - A CUR 2.0 Parquet export configuration - A scoped read-only IAM role that grants CloudQuery access to the bucket 2. In the AWS CloudFormation console, review the stack parameters and click **Create stack**. 3. Return to CloudQuery Platform. The setup screen polls for progress and shows the current status: - **Setting up**: Waiting for the CloudFormation stack to be applied. - **Finalizing setup**: Stack applied. CloudQuery is validating access and finalizing configuration. - **Setup complete**: CloudQuery populates the Role ARN and report name automatically. 4. Click **Test and Continue** to verify access. If setup fails, click **Reset** to start over. Check the CloudFormation stack events in the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/) for error details. ### Manual setup Use manual setup if you already have a CUR 2.0 Parquet export configured in your AWS account. #### Prerequisites for manual setup - A [CUR 2.0 data export](https://docs.aws.amazon.com/cur/latest/userguide/dataexports-create-standard.html) with **Parquet** file format, delivering to an S3 bucket. If you don't have one, create it in the [AWS Billing > Data Exports](https://us-east-1.console.aws.amazon.com/costmanagement/home#/bcm-data-exports) console. - Your **Tenant ID**, visible in the setup guide panel when configuring the integration. - An **External ID**: any alphanumeric string between 2 and 1224 characters. Generating a UUID (e.g. `uuidgen`) is recommended per [AWS best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html). #### IAM role and permissions 1. Create an IAM policy granting read access to your CUR S3 bucket. Replace `` with the name of the S3 bucket where your CUR data is stored. ```bash cat >bucket-policy.json <" }, { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": "arn:aws:s3:::/*" } ] } EOF ``` 2. Create the trust relationship for the cross-account role. Replace `` with the CloudQuery AWS Account ID found in the side panel documentation on the plugin UI page in CloudQuery Platform. Replace `` with your tenant ID from the "Setup guide" section when configuring the integration. Replace `` with your generated external ID. ```bash cat >third-party-trust.json <:role/syncs--role" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } EOF ``` 3. Create the cross-account role and attach the S3 read policy: ```bash aws iam create-role --role-name cross-account-readonly-role-cost-usage \ --assume-role-policy-document file://third-party-trust.json aws iam put-role-policy --role-name cross-account-readonly-role-cost-usage \ --policy-name ReadS3Policy --policy-document file://bucket-policy.json ``` #### Configure the integration Back in CloudQuery Platform: 1. Enter the **Role ARN** of the IAM role you created (e.g. `arn:aws:iam:::role/cross-account-readonly-role-cost-usage`). 2. Enter the **External ID** you used in the trust relationship. 3. Under **Reports**, enter the report name for your CUR export. You can find this in the [AWS Billing > Data Exports](https://us-east-1.console.aws.amazon.com/costmanagement/home#/bcm-data-exports) console under your export's settings. Add additional reports with the **Add report** button if needed. 4. Click **Test and Continue** to verify access. ## What gets synced The AWS CUR integration syncs a single table: | Table | Description | | ------------------- | ------------------------------------------------------------------------------------------------------------------------- | | `awscur_reports_v2` | Line-item cost and usage data from your CUR 2.0 Parquet export, including service, usage type, pricing, and resource tags | Each row represents a line item from your cost report. See the [AWS CUR table documentation](https://www.cloudquery.io/hub/plugins/source/cloudquery/awscur/latest/docs) for the full schema. ## Verify the integration After your first [sync](/platform/syncs/setting-up-a-sync) completes, open the [SQL Console](/platform/features/sql-console) and run these queries to confirm cost data arrived: ```sql -- Count synced cost records SELECT count(*) FROM awscur_reports_v2 ``` ```sql -- Preview cost data SELECT * FROM awscur_reports_v2 LIMIT 5 ``` ## Troubleshooting | Issue | Cause | Fix | | -------------------------- | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CloudFormation stack fails | Insufficient permissions | Verify your AWS user has `cloudformation:CreateStack`, `s3:CreateBucket`, and `iam:CreateRole` permissions. Check the **Events** tab in the [CloudFormation console](https://console.aws.amazon.com/cloudformation/). | | No data after sync | CUR export not yet generated | AWS delivers CUR data within 24 hours of creating a new export. Wait for the first delivery, then re-run the sync. | | `AccessDenied` on S3 | Bucket policy mismatch | Verify the S3 bucket name in the IAM policy matches the actual bucket. Check that both `s3:ListBucket` (on the bucket) and `s3:GetObject` (on bucket contents) are granted. | | External ID mismatch | Trust policy doesn't match secret | Ensure the `EXTERNAL_ID` in the IAM trust policy matches the value entered in CloudQuery Platform exactly. | | Legacy CUR format error | Not using CUR 2.0 Parquet | This integration requires CUR 2.0 with Parquet format. [Create a new data export](https://docs.aws.amazon.com/cur/latest/userguide/dataexports-create-standard.html) with the correct format. | ## Next steps - [Set up a sync](/platform/syncs/setting-up-a-sync) to schedule when your CUR data syncs - Browse cost data in the [SQL Console](/platform/features/sql-console) - View synced resources in the [Asset Inventory](/platform/features/asset-inventory) - See the [AWS CUR integration documentation](https://www.cloudquery.io/hub/plugins/source/cloudquery/awscur/latest/docs) for full configuration reference and table schema ## Related resources - [AWS CUR 2.0 documentation](https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur.html) - [Creating a CUR 2.0 data export](https://docs.aws.amazon.com/cur/latest/userguide/dataexports-create-standard.html) - [AWS Billing and Cost Management](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/billing-what-is.html) - [AWS IAM cross-account access best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html) - [CloudQuery AWS CUR integration on Hub](https://www.cloudquery.io/hub/plugins/source/cloudquery/awscur/latest/docs) --- Source: https://www.cloudquery.io/docs/platform/integration-guides/setting-up-an-azure-integration # Setting up an Azure Integration The Azure integration uses a [service principal](https://learn.microsoft.com/en-us/cli/azure/azure-cli-sp-tutorial-1) with the [Reader role](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#reader) for authentication. This is the recommended method for production deployments. ## Prerequisites - A CloudQuery Platform account with admin access - [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) installed - An Azure account with permissions to create service principals and assign roles - Your Azure subscription ID (find it in the [Azure Portal subscriptions page](https://portal.azure.com/#view/Microsoft_Azure_Billing/SubscriptionsBlade)) ## Set up a service principal Service principal secrets expire after 1 year by default. Set a calendar reminder to rotate the secret before it expires, or use the `--years` flag with `az ad sp create-for-rbac` to set a custom expiration. ### Syncing from a single subscription 1. Open your terminal and log in to Azure: ```bash az login ``` 2. Register the security provider and create a service principal with [Reader access](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#reader): ```bash # Register the security provider az provider register --namespace 'Microsoft.Security' # Create a service principal and grant Reader access az ad sp create-for-rbac --name cloudquery-sp \ --scopes /subscriptions/ --role Reader ``` The command outputs credentials in this format: ```json { "appId": "YOUR_AZURE_CLIENT_ID", "displayName": "cloudquery-sp", "password": "YOUR_AZURE_CLIENT_SECRET", "tenant": "YOUR_AZURE_TENANT_ID" } ``` 3. Save these credentials; you need them when configuring the integration. ### Syncing from multiple subscriptions There are two approaches for multi-subscription setups: **1. Management group level access (recommended)** Scoping the service principal at the [management group](https://learn.microsoft.com/en-us/azure/governance/management-groups/overview) level allows automatic discovery of all subscriptions under the specified group(s), including subscriptions added later. ```bash # Register the security provider az provider register --namespace 'Microsoft.Security' # Create service principal with Management Group access az ad sp create-for-rbac --name cloudquery-sp-root-1 \ --scopes /providers/Microsoft.Management/managementGroups/ \ --role Reader ``` **2. Specific subscriptions access** To limit access to specific subscriptions, list them explicitly. This command grants access to all subscriptions you can currently access: ```bash # Register the security provider az provider register --namespace 'Microsoft.Security' # Create service principal with access to specific subscriptions az ad sp create-for-rbac --name cloudquery-sp \ --scopes $(az account subscription list --query "[].id" -o tsv --only-show-errors | xargs) \ --role Reader ``` With the specific subscriptions approach, the service principal does not automatically get access to subscriptions added later. Run the command again to include new subscriptions. ## Configure the integration 1. In CloudQuery Platform, go to **Data Pipelines** → **Integrations**. Click **Create Integration** and type **Azure** to find the Azure integration.
    ![Find Azure integration](/images/platform/azure-1.png)
    2. Choose a name for your integration (e.g. `Azure`). Enter the service principal credentials in the fields provided: | Field | Value | | ------------------------------ | -------------------------------------------- | | **Azure AD Tenant ID** | `tenant` from the service principal output | | **Service Principal App ID** | `appId` from the service principal output | | **Service Principal Password** | `password` from the service principal output | 3. Under **Configure resources to sync**, choose which subscriptions to include or exclude. 4. Click **Continue** to select the Azure services you want to sync. 5. Click **Test and Continue** to verify the configuration. ## What gets synced The Azure integration can sync hundreds of tables across Azure services. Some of the most commonly used tables include: | Category | Tables | Description | | ---------- | ----------------------------------------------------------------- | ----------------------------------------- | | Compute | `azure_compute_virtual_machines`, `azure_compute_skus` | VMs, compute SKUs | | Storage | `azure_storage_accounts` | Storage accounts | | Networking | `azure_network_virtual_networks`, `azure_network_security_groups` | Virtual networks, network security groups | | Databases | `azure_sql_servers`, `azure_cosmosdb_accounts` | SQL servers, Cosmos DB | | Security | `azure_keyvault_vault_keys`, `azure_security_assessments` | Key Vault keys, security assessments | See the [full Azure table list](https://www.cloudquery.io/hub/plugins/source/cloudquery/azure/latest/docs) for all available tables. ## Verify the integration After your first [sync](/platform/syncs/setting-up-a-sync) completes, open the [SQL Console](/platform/features/sql-console) and run these queries to confirm data arrived: ```sql -- Count synced virtual machines SELECT count(*) FROM azure_compute_virtual_machines ``` ```sql -- List synced subscriptions SELECT DISTINCT subscription_id FROM azure_compute_virtual_machines ``` ```sql -- View storage accounts SELECT subscription_id, name, location FROM azure_storage_accounts LIMIT 10 ``` You can also browse your Azure resources in the [Asset Inventory](/platform/features/asset-inventory) under the **Compute**, **Storage**, **Networking**, and other categories. ## Troubleshooting | Issue | Cause | Fix | | -------------------------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Authentication error | Invalid or expired service principal secret | Verify the `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, and `AZURE_TENANT_ID` match the service principal output. If the secret has expired, [create a new one](https://learn.microsoft.com/en-us/cli/azure/azure-cli-sp-tutorial-3). | | Permission denied | Service principal lacks Reader role | Verify the service principal has the [Reader role](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#reader) on the target subscription, management group, or resource group. | | `Microsoft.Security` provider not registered | Security provider registration required | Run `az provider register --namespace 'Microsoft.Security'` and wait for registration to complete. Check status with `az provider show --namespace 'Microsoft.Security' --query "registrationState"`. | | Missing subscriptions in data | Service principal scope too narrow | For multi-subscription setups, verify the service principal has access to all target subscriptions. Use management group scoping for automatic discovery. | | No data after sync | No services selected | Check that at least one service is selected in the **Select services** step. | ## Next steps - [Set up a sync](/platform/syncs/setting-up-a-sync) to schedule when your Azure data is fetched - Browse synced resources in the [Asset Inventory](/platform/features/asset-inventory) - Run advanced queries in the [SQL Console](/platform/features/sql-console) - See the [Azure integration documentation](https://www.cloudquery.io/hub/plugins/source/cloudquery/azure/latest/docs) for full configuration options and table reference ## Related resources - [Azure CLI installation guide](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) - [Azure Service Principal tutorial](https://learn.microsoft.com/en-us/cli/azure/azure-cli-sp-tutorial-1) - [Azure Management Groups overview](https://learn.microsoft.com/en-us/azure/governance/management-groups/overview) - [Azure RBAC built-in roles](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#reader) - [Azure resource providers and types](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/resource-providers-and-types) - [CloudQuery Azure integration on Hub](https://www.cloudquery.io/hub/plugins/source/cloudquery/azure/latest/docs) --- Source: https://www.cloudquery.io/docs/platform/introduction/faq # Platform FAQ ## Getting Started ### How do I get access to CloudQuery Platform? Sign up at [CloudQuery.io](https://cloud.cloudquery.io/auth/platform-trial) to start a trial. You will need a business email address. See [Creating a New Account](/platform/quickstart/creating-a-new-account) for the full walkthrough. ### What's the difference between CloudQuery Platform and CloudQuery CLI? - **CloudQuery Platform**: Managed service with a web UI, built-in asset inventory, SQL console, reports, policies, automations, AI-powered analysis, and SSO. Data is synced and stored in a managed ClickHouse database. - **CloudQuery CLI**: Command-line tool that runs on your infrastructure. You manage the syncs, choose your own destination database, and handle scheduling yourself. Platform is the primary product for teams that want visibility, governance, and automation out of the box. CLI is for teams that need full control over the pipeline and destination. ### Does CloudQuery Platform support Single Sign-On (SSO)? Yes. CloudQuery Platform supports SSO through SAML-based identity providers. See [Enabling Single Sign-On](/platform/production-deployment/enabling-single-sign-on-sso) for setup instructions. ## Syncs and Integrations ### Which integrations does CloudQuery Platform support? CloudQuery supports 100+ source integrations including AWS, Google Cloud, Azure, DigitalOcean, GitHub, GitLab, Slack, Jira, Okta, and many more. Browse the full list at [CloudQuery Hub](https://www.cloudquery.io/hub/plugins/source). ### How often does CloudQuery sync my data? You control the schedule. Syncs can run daily, weekly, monthly, on a custom cron schedule, or on-demand. Preset schedules run at midnight UTC. See [Setting up a Sync](/platform/syncs/setting-up-a-sync) for configuration details. ### Where does my synced data go? CloudQuery Platform includes a default managed ClickHouse database. All platform features (Asset Inventory, SQL Console, Reports, Policies) run against this database. You can also configure additional destinations if you need data in your own systems. ### Can I sync data from multiple cloud accounts or regions? Yes. You can set up separate integrations for each account, project, or subscription. Each integration configuration should have a unique name to avoid data conflicts. See the [Integration Guides](/platform/integration-guides) for provider-specific setup. ## Features ### What are Policies? Policies are SQL-based detective controls that continuously evaluate your cloud infrastructure. You write a SQL query that defines what a violation looks like, and CloudQuery runs it after every sync. Each row returned is a violation. Unlike IaC scanners that only catch issues at deployment time, Policies detect problems in resources that already exist, including ones created through the console or by tools that have since drifted. Policies can be organized into groups (CIS, SOC 2, HIPAA) and trigger notifications to Slack or webhook destinations. See [Policies](/platform/features/policies) for details. ### What can I do with Reports? Reports give you a visual overview of your cloud resources, security posture, compliance status, and cost optimization opportunities. They're defined as YAML code, so you can build from scratch or customize one of the built-in templates. Each report widget is backed by a SQL query. You can click through to the SQL Console from any visualization to explore the underlying data. See [Reports](/platform/features/reports) for details. ### What is the Asset Inventory? The Asset Inventory is a unified, normalized view of all your cloud assets across every connected integration. Resources from AWS, GCP, Azure, and other providers appear in a single searchable inventory with consistent schemas, organized into categories like Compute, Storage, Networking, Databases, and Identity. You can filter by any attribute, save filters for reuse, and inspect individual resources including their related resources. See [Asset Inventory](/platform/features/asset-inventory) for the full feature overview. ### What is the SQL Console? The SQL Console lets you run ClickHouse SQL queries directly against your synced data. You can query raw integration tables, the unified Asset Inventory tables, or join across both. Queries can be saved, shared with your team, and managed via the [REST API](https://platform-multi-tenant-api-docs.cloudquery.io/). The SQL Console is read-only. You can't modify the underlying data. See [SQL Console](/platform/features/sql-console) for details. ### What AI features does CloudQuery offer? CloudQuery Platform includes three AI-powered features: - **AI Assistant**: A conversational agent on the Platform Home page. Ask questions about your cloud infrastructure in plain language and get direct answers or executable SQL. Supports a data privacy mode (schema-only, default) and a full access mode where it can execute queries on your behalf. See [AI Assistant](/platform/features/ai-assistant) for details. - **AI Query Writer**: Built into the SQL Console. Describe what you want in natural language, and the AI generates the SQL. Always operates in schema-only mode. See [AI Query Writer](/platform/features/sql-console/ai-query-writer) for details. - **MCP Server**: Connects CloudQuery to Claude Desktop, Cursor, VS Code, or any MCP-compatible tool. In Platform mode, it can execute queries and reason about your data. See [CloudQuery MCP Server](/platform/features/mcp-server) for setup instructions. ### What's the difference between the AI Assistant and the AI Query Writer? The [AI Assistant](/platform/features/ai-assistant) lives on the Platform Home page and is a full conversational agent. It can answer questions directly, reason across multiple tables, and, when Full Access mode is enabled by an admin, execute queries and return results in the chat. It also supports [Custom Context](/platform/features/ai-assistant/custom-context) to carry organization-specific knowledge into conversations. The [AI Query Writer](/platform/features/sql-console/ai-query-writer) is a focused tool inside the SQL Console. It always operates in schema-only mode and always returns a SQL query for you to review and run. Use it when you're already working in the SQL Console and want help authoring a specific query. ### Does the AI Assistant have access to my data? It depends on your organization's setting. By default, the AI Assistant operates in **Data Privacy mode**: it can only see table names and column definitions, and it returns SQL queries for you to run manually; it never reads row data. Organization administrators can enable **Full Access mode** in **Organization Settings → Platform Settings**. In this mode, the assistant executes queries on your behalf and returns results in the chat. Only the rows returned by those queries are processed. In either mode, the assistant is powered by Claude (Anthropic) on AWS Bedrock. AWS Bedrock does not use customer data to train or improve models. See [AI Assistant](/platform/features/ai-assistant) for the full breakdown. ### How do notifications work? CloudQuery Platform supports two notification destination types: **Slack** (native OAuth integration with channel selection) and **Webhook** (HTTP POST to any endpoint, including PagerDuty, Opsgenie, or custom services). Notifications are triggered by policy violations and alerts. Configure destinations in Organization Settings, then attach them to individual policies or alerts. See [Notification Destinations](/platform/features/notification-destinations) for setup details. ## Data and Security ### Does CloudQuery access my application data? No. CloudQuery integrations access metadata and configuration data from cloud provider APIs (resource inventories, IAM configurations, network settings). Your application data, file contents, and database records are not accessed. ### Where is my data stored on CloudQuery Platform? Your synced data is stored in a managed ClickHouse database that is part of your CloudQuery Platform installation. The data is accessible through the Asset Inventory, SQL Console, Reports, and Policies features. ### Can I delete synced data? Yes. The [Data Management](/platform/production-deployment/data-management) page lets you view all synced integrations and their tables, and selectively delete data. Deletions remove the data from ClickHouse, including any views and historical snapshots built on those tables. If the integration is still active, deleted tables repopulate on the next sync. ### Does the AI Query Writer have access to my data? No. The AI model can access table names and their schemas to generate accurate SQL, and it can test queries for syntax errors. It does not retrieve or see actual data from your CloudQuery database. ## Getting Help ### Where can I get help with CloudQuery Platform? - **Community Forum**: [community.cloudquery.io](https://community.cloudquery.io) for questions and discussions - **GitHub Issues**: [github.com/cloudquery/cloudquery](https://github.com/cloudquery/cloudquery) for bug reports and feature requests - **Platform Docs**: You're here. Browse the sidebar for guides on integrations, syncs, reports, and more - **API Reference**: [Platform API Reference](https://platform-multi-tenant-api-docs.cloudquery.io/) for programmatic access - **CLI Docs**: For CloudQuery CLI topics, see the [CLI Documentation](/cli) and [CLI FAQ](/cli/getting-support/faq) ### How do I report a bug? 1. Check if the issue already exists on [GitHub](https://github.com/cloudquery/cloudquery/issues) 2. If not, create a new issue with: - Steps to reproduce the problem - Expected vs actual behavior - Screenshots of the Platform UI if relevant - Your browser and OS version --- Source: https://www.cloudquery.io/docs/platform/introduction/getting-help # Getting Help There are several ways to get help for any CloudQuery-related issues or questions: ## Service Status Check real-time platform status, ongoing incidents, and scheduled maintenance windows: [CloudQuery Status Page →](https://status.cloudquery.io/) ## Community Support - **Community Forum** - Join the [CloudQuery Community](https://community.cloudquery.io/) to browse previous threads or start a new discussion. ## Bug Reports and Feature Requests - **GitHub Issues** - Check out previous issues at [github.com/cloudquery/cloudquery](https://github.com/cloudquery/cloudquery) and if none address your problem, open a new one. ## FAQ Check our [Platform FAQ](./faq) for answers to common questions about setup, syncs, policies, reports, AI features, and data security. ## Documentation - **Platform Docs** - You are here! Browse the sidebar for guides on integrations, syncs, reports, and more. - **CLI Docs** - For CloudQuery CLI, SDK, or integration development, see the [CLI Documentation](/cli). - **Platform API** - For programmatic access, see the [Platform API](/platform/reference/api-reference) guide, or jump straight to the [Interactive API Reference ↗](https://platform-multi-tenant-api-docs.cloudquery.io/). ## Next Steps - [Platform Quickstart](/platform/quickstart/creating-a-new-account) - Create your account and get started - [FAQ](/platform/introduction/faq) - Frequently asked questions about CloudQuery Platform - [Troubleshooting](/platform/reference/troubleshooting) - Common issues and solutions for CloudQuery Platform --- Source: https://www.cloudquery.io/docs/platform/introduction # CloudQuery Documentation CloudQuery is a cloud infrastructure platform that gives platform engineering and cloud operations teams a queryable data layer for visibility, governance, and automation. It syncs configuration data from cloud providers, SaaS applications, and APIs into a normalized, SQL-queryable data layer. You connect [sources](https://www.cloudquery.io/hub/plugins/source) (AWS, GCP, Azure, GitHub, Okta, and hundreds more) and send data to [destinations](https://www.cloudquery.io/hub/plugins/destination) (PostgreSQL, BigQuery, Snowflake, S3, and others). Once synced, you can query your infrastructure with SQL to answer questions like: What resources exist across all accounts? Which ones are misconfigured? Where are we overspending? ## Common Use Cases - **Cloud asset inventory** - A complete, queryable view of every resource across all cloud accounts. - **Security and compliance** - Continuous monitoring of infrastructure configuration against security and compliance requirements. - **Cloud governance** - Policy enforcement across your cloud estate, catching misconfigurations and violations regardless of how resources were created. - **FinOps** - Correlate resource configuration with cost data to identify waste and attribute spending to teams. ## Get Started - [Platform Quickstart](/platform/quickstart/creating-a-new-account) - Sign up and connect your first cloud account - [CLI Getting Started](/cli/getting-started) - Install the CLI and run your first sync - [Platform vs CLI](/platform/introduction/platform-vs-cli) - Understand the differences and decide which path is right for your team ## Platform Features Managed service with a web UI, built-in asset inventory, policies, automations, and AI-powered analysis. CloudQuery Platform Asset Inventory interface displaying cloud resources with attribute-based search and filtering - **[Cloud Connection Setup](/platform/integration-guides/general-integration-setup-guide)** - Set up and schedule syncs from AWS, GCP, Azure and dozens of other cloud and SaaS products, all from the UI. - **[Asset Inventory](/platform/features/asset-inventory)** - Explore your cloud assets. - **[SQL Console](/platform/features/sql-console)** - Query your raw cloud data with SQL, save queries, and share them with your team. - **[Reports](/platform/features/reports) and [Alerts](/platform/features/alerts)** - Build reports from built-in templates or custom SQL, and configure alerts to notify your team when conditions are met. - **[Policies](/platform/features/policies)** - Define SQL-based policies to continuously monitor your cloud for security, compliance, and operational issues. - **[Insights](/platform/features/insights)** - Explore security, compliance, and cost findings across your cloud infrastructure with severity-based triaging and saved filters. - **[AI Assistant](/platform/features/ai-assistant)** - Ask questions about your cloud infrastructure in plain language and get direct answers or executable SQL. Includes [Custom Context](/platform/features/ai-assistant/custom-context) to share organization-specific knowledge with the assistant. - **[AI Query Writer](/platform/features/sql-console/ai-query-writer)** - Generate SQL from natural language descriptions directly inside the SQL Console. - **[Platform API](https://platform-multi-tenant-api-docs.cloudquery.io/)** - Access all Platform features programmatically, including querying cloud data, managing syncs, and configuring policies. - **[CloudQuery Cloud API](https://api-docs.cloudquery.io/)** - Manage your cloudquery.io account programmatically: teams, plugins, addons, and usage. - **[Sync to Your Own Data Warehouse](/platform/integration-guides/general-destination-setup-guide)** - Export data to PostgreSQL, BigQuery, S3, Snowflake, and many other destinations beyond the built-in ClickHouse database. ## Architecture Overview ``` +------------------------------------------------+ | CloudQuery Platform | | | | +---------+ +---------+ +--------------+ | | | Queries | | Reports | | Automations | | | | & AI | | & Alerts| | & Policies | | | +----+----+ +----+----+ +------+-------+ | | | | | | | +-------------+-------------+ | | | | | +---------v---------+ | | | ClickHouse | | | | (data backend) | | | +---------^---------+ | | | | | +---------+---------+ | | | Integrations | | | +---+--------+---+-+ | +------------------------------------------------+ | | | v v v AWS GCP Azure ... ``` - **[Integrations](/platform/core-concepts/integrations)** connect to your cloud providers and sync asset data into the platform. - **ClickHouse** serves as the data backend, storing all synced cloud asset data and views. - **Platform features** - queries, reports, alerts, policies, and automations - all operate on top of this data layer. ## CLI Documentation Self-hosted, code-first tool you run on your own infrastructure. Full control over configuration, scheduling, and data storage. - [Core Concepts](/cli/core-concepts/integrations) - Integrations, syncs, and destinations - [Configuration](/cli/core-concepts/configuration) - Configure your syncs - [Available Integrations](/cli/integrations) - Browse source and destination integrations - [Managing CloudQuery](/cli/managing-cloudquery/deployments/overview) - Deployment, monitoring, and upgrades - [CLI Reference](/cli/cli-reference/cloudquery) - Command reference ## Next Steps - **[Quickstart](/platform/quickstart/creating-a-new-account)** - Create your account and set up your first integration - **[Integration Guides](/platform/integration-guides/general-integration-setup-guide)** - Step-by-step setup for AWS, GCP, Azure, GitHub, Kubernetes, and more - **[Query Examples](/platform/features/sql-console/query-examples)** - Real-world query examples for security, compliance, and cost optimization - Browse [step-by-step tutorials](https://www.cloudquery.io/blog/category/tutorials) - Join the [CloudQuery Community](https://community.cloudquery.io/) - **[Getting Help](/platform/introduction/getting-help)** - Support and additional resources --- Source: https://www.cloudquery.io/docs/platform/introduction/platform-vs-cli # Platform vs CLI CloudQuery offers two paths to get value from your cloud data: the **Platform** (managed service) and the **CLI** (self-managed). Both share the same integration ecosystem with 70+ sources and destinations. The right choice depends on your team's needs and priorities. ## Comparison | | CloudQuery Platform | CloudQuery CLI | | --------------------------------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------ | | **Deployment** | Managed service | Self-hosted on your infrastructure | | **Interface** | Web UI + REST API | YAML configuration + CLI commands | | **Setup time** | Minutes (sign up + connect) | Minutes (install + configure) | | **Best for** | Teams who want fast time-to-value with less ops overhead | Teams with infra expertise who want full control | | **Data storage** | Built-in ClickHouse + optional export to any destination | Your choice: PostgreSQL, BigQuery, Snowflake, and more | | **Scheduling** | Built-in scheduling via UI | Your orchestrator (cron, Airflow, etc.) | | **Querying data** | Built-in SQL console + AI Assistant + AI Query Writer | SQL in your chosen destination database | | **[Reports & visualization](/platform/features/reports)** | [Built-in reports](/platform/features/reports/built-in-report-templates) with 40+ templates | DIY with external BI tools (Grafana, Metabase, etc.) | | **Customization** | Platform features + ability to export to any destination | Full: custom integrations, transforms, SDK | | **Cost** | Platform pricing (free tier available) | Infrastructure costs only | ### Platform-Exclusive Features The Platform includes capabilities beyond data syncing that aren't part of the CLI: - **[Asset inventory](/platform/features/asset-inventory)**: Search and explore cloud assets with a built-in UI, saved searches, and custom columns - **[Policies](/platform/features/policies)**: Customizable compliance rules across all your cloud providers - **[Automations](/platform/features/alerts)**: Event-driven triggers that respond to resource changes and policy violations - **[Alerts](/platform/features/alerts)**: SQL-based alerting with webhook, Slack, and Jira notifications - **[AI Assistant](/platform/features/ai-assistant)**: Ask questions about your cloud in plain language and get direct answers or SQL, with optional full data access mode - **[AI Query Writer](/platform/features/sql-console/ai-query-writer)**: Generate SQL from natural language descriptions directly inside the SQL Console ## Choose Platform If… - You want **faster onboarding** with minimal infrastructure setup - You prefer a **web UI** for managing syncs, schedules, and configurations - You want **built-in features** like asset inventory, SQL console, saved searches, alerts, policies, automations, reports, and AI features (AI Assistant and AI Query Writer) - Your team is smaller or doesn't have **dedicated infrastructure resources** - You want **managed scheduling, monitoring, and notifications** without maintaining orchestration tooling [Get started with the Platform →](/platform/quickstart/creating-a-new-account) ## Choose CLI If… - You want **maximum control** over your data pipeline and infrastructure - You already have **orchestration tooling** (Airflow, cron, CI/CD) in place - You need to integrate CloudQuery into **existing CI/CD workflows** - You prefer a **code-first, config-as-code** approach - You have **strict data residency** or compliance requirements that demand full self-hosting - You want to build **custom integrations** using the SDK [Get started with the CLI →](/cli/getting-started) ## Can I use both? Yes. Many teams use both the Platform and CLI together: - Use **Platform** for day-to-day cloud visibility, asset inventory, and alerting - Use **CLI** for custom data pipelines, CI/CD-integrated syncs, or advanced transformations - **Export Platform data** to the same destinations the CLI supports, keeping everything in one place The Platform and CLI share the same integration ecosystem, so your source and destination configurations translate between the two. ## Next Steps - [Platform Getting Started Guide](/platform/quickstart/creating-a-new-account): Sign up and connect your first cloud account - [CLI Getting Started Guide](/cli/getting-started): Install the CLI and complete your first sync - [Core Concepts](/platform/core-concepts/integrations): Understand how integrations, syncs, and destinations work --- Source: https://www.cloudquery.io/docs/platform/production-deployment/account-profile # Account Profile The Account Profile page lets you update your personal account settings, including your display name and platform usage tracking preferences. ## Accessing Your Profile 1. In the sidebar, click your user icon and select **Account settings**. 2. You are on the **Profile** tab by default. ## Editable Fields ### Full Name Your display name as shown across the platform. Enter your name and click **Save** to update. **Restrictions:** - Must start with a letter - Can contain letters, spaces, hyphens, and apostrophes - Maximum 255 characters If you log in via SSO, your name is managed by your identity provider and cannot be edited here. ### Email Address Your email address is displayed but cannot be changed. It is set during account creation or provided by your SSO identity provider. ### Platform Usage Statistics Toggle **Enable platform usage statistics collection** to control whether anonymous usage data is collected. This setting is per-user. ## SSO Users If you authenticate through an SSO provider (Google, Microsoft, Okta, or another SAML provider), your name is managed by the identity provider. The Full Name field is read-only. You can still change your usage tracking preference. ## Next Steps - [Multi-Factor Authentication](/platform/production-deployment/user-management/multi-factor-authentication) - Secure your account with MFA - [Platform Settings](/platform/production-deployment/platform-settings) - Configure platform-wide settings --- Source: https://www.cloudquery.io/docs/platform/production-deployment/api-keys # API Keys API keys provide programmatic access to the CloudQuery Platform API. Use them to automate workflows, integrate with CI/CD pipelines, or access platform features from scripts and external tools. ## Prerequisites You need admin permissions to create and manage API keys. ## Creating an API key 1. In the sidebar, click your user icon and select **Organization settings**. 2. Navigate to the **API Keys** section. 3. Click **Create API Key**. 4. Enter a **Name** to identify the key's purpose (e.g., "CI/CD Pipeline" or "Terraform Integration"). 5. Set an **Expiration date**. Preset options are 7 days, 30 days, and 1 year. You can also pick a custom date. 6. Assign a **Role** to control what the key can access: | Role | Description | | --------------- | ------------------------------------------------- | | `admin:write` | Full admin access: manage syncs, users, settings | | `admin:read` | Read-only admin access | | `general:write` | Read and write access to non-admin features | | `general:read` | Read-only access to non-admin features | | `ci` | CI/CD pipeline access for syncs and deployments | | `schema-only` | Access to table schemas only | 7. Click **Create**. The API key value is displayed only once after creation. Copy it immediately and store it securely; you will not be able to view it again. ## Using an API key Include the key in the `Authorization` header of API requests: ```bash curl -H "Authorization: Bearer " \ https:///api/plugins ``` API keys are also used for plugin downloads from the CloudQuery Hub. ## Managing API keys The API keys list shows: | Field | Description | | -------------- | ---------------------------------------------------- | | **Name** | The identifier you assigned when creating the key | | **Created by** | The email of the user who created the key | | **Created at** | When the key was created | | **Expires at** | When the key will stop working | | **Last used** | The last date the key was used (accurate to the day) | | **Status** | Whether the key is active or expired | To delete a key, use the action menu on the key's row. Deleted keys stop working immediately. ## Best practices - Create separate keys for different use cases (CI/CD, monitoring, automation) so you can rotate or revoke them independently. - Set expiration dates. Avoid creating keys that never expire. - Store keys in a secrets manager (e.g., AWS Secrets Manager, HashiCorp Vault) rather than in plaintext configuration files. ## Next Steps - [CloudQuery MCP Server](/platform/features/mcp-server) - Use API keys with the MCP server - [Audit Log](/platform/production-deployment/audit-log) - Track API key usage - [Troubleshooting](/platform/reference/troubleshooting#api-issues) - Diagnose API authentication and authorization errors --- Source: https://www.cloudquery.io/docs/platform/production-deployment/audit-log # Audit Log The Audit Log records actions performed by users and the system across your CloudQuery Platform installation. Use it to track who did what, when, and from where, for security reviews, compliance requirements, and troubleshooting. ## Accessing the audit log 1. In the sidebar, click your user icon and select **Organization settings**. 2. Navigate to the **Audit Log** section. ## What is logged Each audit log entry includes: | Field | Description | | -------------- | ----------------------------------------------------------------------------- | | **User** | The user who performed the action | | **IP address** | The IP address the action originated from | | **Team** | The team associated with the event (if applicable) | | **Event type** | The type of action performed (e.g., user login, sync created, policy updated) | | **Entity** | The specific resource or object that was affected | | **Status** | Whether the action succeeded or failed | | **Timestamp** | When the event occurred | | **Details** | Additional context about the event | ## Filtering and searching Use the search bar and filters to narrow down audit log entries: - **Search**: full-text search across user name, team name, event type, entity name, and IP address - **User**: filter by a specific user - **Team**: filter by team name - **Event type**: filter by the type of action - **Entity**: filter by the affected resource name - **IP address**: filter by source IP - **Time range**: filter by start and end time ## Use cases - **Security investigation**: identify who modified a sync configuration or changed user permissions - **Compliance**: demonstrate access controls and change tracking for auditors - **Troubleshooting**: correlate user actions with system behavior to diagnose issues - **Access review**: verify that API keys and user accounts are being used as expected ## Programmatic access Audit log entries can be queried via the Platform API for compliance automation and SIEM integrations. See the [Platform API Reference](/platform/reference/api-reference) (`audit-logs` section) for endpoint details including time-range filtering and pagination. ## Next Steps - [User Management](/platform/production-deployment/user-management) - Manage the users shown in audit logs - [API Keys](/platform/production-deployment/api-keys) - Manage API keys that appear in audit entries - [Platform Settings](/platform/production-deployment/platform-settings) - Configure platform-wide settings --- Source: https://www.cloudquery.io/docs/platform/production-deployment/data-management # Data Management The Data Management page lets you view all synced integrations and their associated tables, and selectively delete data. Use this when you need to remove stale data from decommissioned integrations or clear specific tables. ## Accessing Data Management 1. In the sidebar, click your user icon and select **Organization settings**. 2. Navigate to the **Data management** tab. ## Viewing Synced Data The page displays a paginated table of all integrations with: | Column | Description | |--------|-------------| | **Integration name** | The name of the sync integration | | **Source plugin** | The plugin used (e.g., `cloudquery/aws`) | | **Table count** | Number of tables synced by this integration | Expand an integration to see its individual tables. ## Deleting Table Data Select one or more tables across one or more integrations, then click **Delete selected**. A confirmation dialog appears showing how many tables and integrations are affected. The dialog warns that **this action cannot be undone and permanently removes the data**. If the integration still exists and is active, the deleted tables will repopulate during the next sync. If the integration has been removed, the data is permanently gone. Tables with a pending deletion are marked accordingly in the list until the deletion completes. ### What Gets Deleted When you delete table data, the platform removes the synced rows from the underlying ClickHouse storage. This affects: - The raw data tables - Any views built on top of the deleted tables - Historical snapshot data for those tables The integration configuration itself is not affected. Only the synced data is removed. ## Next Steps - [Syncs](/platform/core-concepts/syncs) - Understand how data flows into the platform - [Understanding Platform Views](/platform/advanced-topics/understanding-platform-views) - How the platform organizes synced data - [Performance Tuning](/platform/advanced-topics/performance-tuning) - Optimize data management --- Source: https://www.cloudquery.io/docs/platform/production-deployment/enabling-single-sign-on-sso/certificate-rollover # SSO Certificate Rollover When your SAML signing certificate is approaching expiry, you can rotate it using the **Certificate management** section on the **Settings / Single sign-on** page. The rollover process works by generating a new certificate and broadcasting it alongside the current one in the platform's SAML Metadata endpoint, giving your Identity Provider time to start trusting it before the switch. Begin the process well in advance of the expiry date to leave enough time for your IdP to be updated and tested. In most SAML setups the rotation can be completed without any interruption to SSO logins. The IdP only needs to know the SP certificate in two situations, both of which are optional features that are off by default in most setups: - The SP signs its AuthnRequests and the IdP is configured to verify those signatures. - The IdP encrypts its SAML assertions and uses the SP certificate to do so. If either of these applies to your setup, update your IdP to trust the new certificate before promoting it to prevent interruption to SSO logins. The **Certificate management** section is only visible to users with the **Admin** role. Users with the **Admin:Read** role cannot create or promote certificates. ## Step 1: Create the rollover certificate Navigate to **Settings / Single sign-on** and scroll down to the **Certificate management** section. You will see the fingerprint and expiry date of your currently active certificate. Click **Create rollover certificate** to generate a new certificate.
    ![Certificate management section showing the active certificate and the Create rollover certificate button](/images/platform/sso-cert-rollover-1.png) The Certificate management section before a rollover certificate exists
    ## Step 2: Trust the new certificate in your Identity Provider Once the rollover certificate has been created, its fingerprint and expiry date are shown alongside the active certificate. From this point on, the rollover certificate is also included in the platform's SAML Metadata endpoint.
    ![Certificate management section showing both the active and rollover certificates](/images/platform/sso-cert-rollover-2.png) Both the active certificate and the rollover certificate are now listed
    You can use **Download rollover certificate** to obtain the certificate file and explicitly upload it to your Identity Provider (IdP) if required. In typical SAML setups, the IdP does **not** need to explicitly trust or upload the Service Provider's certificate. However, some IDPs require this step; check your provider's documentation if you are unsure. ## Step 3: Promote the rollover certificate to active Once your IdP is configured to trust the new certificate, click **Promote to active**. A confirmation dialog will warn you that the current active certificate will be removed and replaced by the rollover certificate.
    ![Promote rollover certificate confirmation dialog](/images/platform/sso-cert-rollover-3.png) The confirmation dialog before promoting the rollover certificate
    Only proceed once you have confirmed your IdP is configured to trust the rollover certificate. If it is not, users may be locked out of SSO. Click **Promote to active** in the dialog to complete the rotation. The rollover certificate becomes the new active certificate and the previous one is removed. ## Step 4: Verify the rotation After promoting, confirm that the **Certificate management** section now shows the new certificate fingerprint and expiry date.
    ![Certificate management section after a successful rollover, showing the new active certificate](/images/platform/sso-cert-rollover-4.png) The new certificate is now active; a fresh rollover can be created at any time
    Finally, verify that SSO login still works as expected. If you are currently logged in via SSO, test in a separate or incognito browser session to avoid accidental lockout. ## Next Steps - [SSO Overview](/platform/production-deployment/enabling-single-sign-on-sso) - Single sign-on configuration options - [User Management](/platform/production-deployment/user-management) - Manage users and roles --- Source: https://www.cloudquery.io/docs/platform/production-deployment/enabling-single-sign-on-sso # Enabling Single Sign-on (SSO) Setting up Single Sign-on (SSO) allows your users to log into the CloudQuery Platform using their company credentials, such as those provided by Okta, Google, Microsoft, or other providers. To set up SSO for your Platform installation, see one of the specific guides below: - [Single Sign-on with Google](enabling-single-sign-on-sso/single-sign-on-with-google) - [Single Sign-on with Microsoft](enabling-single-sign-on-sso/single-sign-on-with-microsoft) - [Single Sign-on with Okta](enabling-single-sign-on-sso/single-sign-on-with-okta) If your provider isn't listed, the steps are similar. Choose one of the guides to use as an example, but adjust as necessary for your application. If you get stuck at any point, contact us for help. When the basic setup is complete, see how to [map groups to user roles](enabling-single-sign-on-sso/map-groups-to-roles). To rotate your SAML signing certificate, see [SSO Certificate Rollover](enabling-single-sign-on-sso/certificate-rollover). ## Programmatic configuration SAML SSO can be configured programmatically via the Platform API, which is useful for IaC-managed deployments or automated certificate rotation. See the [Platform API Reference](/platform/reference/api-reference) (`admin` section) for the SAML configuration and certificate rollover endpoints. ## Next Steps - [SSO with Google](/platform/production-deployment/enabling-single-sign-on-sso/single-sign-on-with-google) - Configure Google Workspace SSO - [SSO with Microsoft](/platform/production-deployment/enabling-single-sign-on-sso/single-sign-on-with-microsoft) - Configure Microsoft Entra ID SSO - [SSO with Okta](/platform/production-deployment/enabling-single-sign-on-sso/single-sign-on-with-okta) - Configure Okta SSO - [Map Groups to Roles](/platform/production-deployment/enabling-single-sign-on-sso/map-groups-to-roles) - Assign roles based on identity provider groups - [Troubleshooting](/platform/reference/troubleshooting#sso-issues) - Diagnose common SSO login and role mapping issues --- Source: https://www.cloudquery.io/docs/platform/production-deployment/enabling-single-sign-on-sso/map-groups-to-roles # Map Groups to User Roles on Platform CloudQuery Platform supports [user roles](../user-management/workspace-roles-overview) that specify what activities users can perform in the application. There are additional data access roles that specify what data the users can see. You can map each group from your SSO identity provider to a set of roles on CloudQuery Platform so the Platform roles are updated automatically for each user. For example, a user who is a member of the `test-team` group in your Google workspace automatically receives the `admin:read` role when they log in. To set up the mapping between groups and user roles, navigate to **Organization settings > Single sign-on** and scroll down to the **Role Mapping** section. ## Default Mapping The first section provides an option to set the default user roles for all users who are not a member of any group on your SSO identity provider. We recommend you leave this empty or assign a restrictive role. ![Default Role Mapping](/images/platform/default-roles.png) ## Custom Group Mapping This section enables mapping of groups from your SSO identity provider to roles in CloudQuery Platform. ![Role Mapping](/images/platform/role-mapping.png) In the left column, put the group name from the SSO Identity Provider. In the right column, select roles to assign to the members of the group. You can select multiple roles as long as they are of the same type (built-in feature roles, or data access roles). Roles are additive, not restrictive. This means that if a user has `Admin:Read` and `General:Read` role assigned via group memberships, they will have the permissions of `Admin:Read`. See also [Limiting Access to Data](../user-management/limiting-access-to-data.mdx) as **Workspace Roles** override **Data Access Roles**. ## Next Steps - [Workspace Roles Overview](/platform/production-deployment/user-management/workspace-roles-overview) - Understand available roles - [Limiting Access to Data](/platform/production-deployment/user-management/limiting-access-to-data) - Restrict data access by role - [Certificate Rollover](/platform/production-deployment/enabling-single-sign-on-sso/certificate-rollover) - Manage SSO certificate updates --- Source: https://www.cloudquery.io/docs/platform/production-deployment/enabling-single-sign-on-sso/single-sign-on-with-google # Single Sign-On with Google ## Step 1: Set the Base URL 1. In the Platform sidebar, click on your user, then click **Organization settings**. Switch to the **Single sign-on** tab. 2. In the **Base URL** field, enter the HTTPS URL for your platform installation and click **Submit**. This will generally match the value in your browser. It should be the domain or subdomain you host CloudQuery platform on, like https://cloudquery.example.com:

    ![Setting the base URL in the CloudQuery admin panel](/images/platform/sso-google-1.png)

    ## Step 2: Create a SAML app in Google Admin 1. In a new tab, open [https://admin.google.com/](https://admin.google.com/) 2. Click **Apps** → **Web and mobile apps** → **Add app** → **Add custom SAML app**

    ![Creating a SAML app in Google Admin](/images/platform/sso-google-2.png)

    ## Step 3: Complete App Details 1. In the App Name field, enter a name to identify your application with. `CloudQuery` is a good choice in most cases. 2. (Optional) Enter a description 3. (Optional) Provide an App icon for your users. You can use this icon: Google CloudQuery icon (Right-click on the image → Save Image As… → Save to your drive. Then upload it in the Google interface.) 4. Click **Continue**

    ![Google CloudQuery app details](/images/platform/sso-google-4.png)

    ## Step 4: Download & Upload Metadata 1. On the next page, click **Download Metadata**:

    ![Google Admin SAML app setup page with the Download Metadata button highlighted](/images/platform/sso-google-5.png)

    This will download a `GoogleIDPMetadata.xml` file onto your drive. Click **Continue**. 2. Upload the XML metadata file in the CloudQuery admin panel by clicking **Upload metadata file**:

    ![CloudQuery Platform SSO admin panel with the Upload metadata file button for importing Google IdP metadata](/images/platform/sso-google-6.png)

    ## Step 5: Enter ACS URL and Entity ID Back in the Google Admin interface, enter the value for ACS URL and Entity ID. These values can be copy-pasted from the CloudQuery Platform Admin page:

    ![Copying ACS URL and Entity ID](/images/platform/sso-google-7.png)

    Copy these values into the highlighted fields:

    ![Entering ACS URL and Entity ID](/images/platform/sso-google-8.png)

    When done, click **Continue** on the Google page. ## Step 6: Set Attribute Mappings Next, enter some basic attribute mapping information: 1. `First name` → `first_name` 2. `Last name` → `last_name` 3. `Primary email` → `email`

    ![Setting attribute mappings](/images/platform/sso-google-9.png)

    ## Step 7: Configure Group Membership On the same screen, configure group membership so that CloudQuery Platform can assign roles based on your Google Workspace groups. 1. In the **Group membership** section of the Google SAML app, add the groups you want to map to CloudQuery Platform roles. Set the **App attribute** to the name you want to use as the group claim (e.g., `groups`).

    ![Setting group membership](/images/platform/sso-google-10.png)

    2. In the CloudQuery Platform SSO settings, set the **Group attribute** field to the same value as the **App attribute** you configured in Google (e.g., `groups`).

    ![Setting the group attribute in CloudQuery Platform](/images/platform/sso-google-11.png)

    3. Click **Continue** in the Google UI. You can map multiple groups to different roles in CloudQuery Platform. For the full configuration, including default roles for users not in any group and multiple group-to-role mappings, see [Map Groups to User Roles](/platform/production-deployment/enabling-single-sign-on-sso/map-groups-to-roles). The **Group attribute** value in CloudQuery must exactly match the **App attribute** value in Google. This is case-sensitive. ## Step 8: Enable User Access Now, click on the User access section. The entire User access block is clickable

    ![Google Admin SAML app overview page with the User access section to click for enabling access](/images/platform/sso-google-12.png)

    Select **ON for everyone**. Then click **SAVE**.

    ![Google Admin User access settings with ON for everyone selected to enable SSO for all organization users](/images/platform/sso-google-13.png)

    Though not covered in this guide, you can also specify which users in your organization should have access by only turning it on for certain groups. ## Step 9: Save and Test Click **Save and enable** on the CloudQuery admin page:

    ![Saving and enabling user access](/images/platform/sso-google-14.png)

    On the Google Admin page, click **TEST SAML LOGIN**.

    ![Google Admin SAML app page with the TEST SAML LOGIN button to verify the SSO configuration](/images/platform/sso-google-15.png)

    If everything is set up correctly, you should now be logged into CloudQuery Platform with your Google account. ## Next Steps - [Map Groups to Roles](/platform/production-deployment/enabling-single-sign-on-sso/map-groups-to-roles) - Map Google groups to platform roles - [Certificate Rollover](/platform/production-deployment/enabling-single-sign-on-sso/certificate-rollover) - Manage certificate updates - [User Management](/platform/production-deployment/user-management) - Manage users and roles --- Source: https://www.cloudquery.io/docs/platform/production-deployment/enabling-single-sign-on-sso/single-sign-on-with-microsoft # Single Sign-On with Microsoft ## Step 1: Set the Base URL 1. In the CloudQuery Platform sidebar, click on your user profile, then select **Organization settings**. 2. Switch to the **Single Sign-On** tab. 3. In the **Base URL** field, enter the HTTPS URL for your platform installation and click **Submit**. - This should be the domain or subdomain where you host CloudQuery Platform, e.g., `https://cloudquery.example.com`.
    ![Configuration of your domain name in CloudQuery platform](/images/platform/sso-ms-1.png) Configuration of your domain name in CloudQuery platform
    ## Step 2: Register an Application in Microsoft Entra ID 1. In a new tab, navigate to [Microsoft Entra ID (Azure AD)](https://portal.azure.com/). 2. Click **Enterprise Applications** → **New Application**. 3. Click **Create your own application**. 4. Enter a name for the application, such as **CloudQuery**, and select **Integrate any other application you don’t find in the gallery (Non-gallery)**. 5. Click **Create**.
    ![Creating a new enterprise application](/images/platform/sso-ms-2.png) Creating a new enterprise application
    ## Step 3: Configure SAML-based SSO 1. Inside the newly created application, navigate to **Single sign-on** under **Manage** section. 2. Select **SAML** as the sign-in method.
    ![Setup of SAML protocol](/images/platform/sso-ms-3.png) Setup of SAML protocol
    1. Click **Edit** under **Basic SAML Configuration**. 2. Enter the following details: - **Identifier (Entity ID)**: Copy this value from the CloudQuery Admin panel. - **Reply URL (ACS URL)**: Copy this value from the CloudQuery Admin panel. 3. Click **Save**.
    ![SAML configuration with values from CloudQuery admin page](/images/platform/sso-ms-4.png) SAML configuration with values from CloudQuery admin page
    ## Step 4: Download & Upload Metadata 1. Scroll down to the **SAML Certificates** section. 2. Click **Download** next to **Federation Metadata XML**. - This will download a file named `MicrosoftIDPMetadata.xml`.
    ![Download of Federation Metadata XML file](/images/platform/sso-ms-5.png) Download of Federation Metadata XML file
    In the CloudQuery Admin panel, click Upload metadata file and upload the `MicrosoftIDPMetadata.xml` file as shown in the figure below:
    ![Uploading federation metadata XML file](/images/platform/sso-ms-6.png) Uploading federation metadata XML file
    ## Step 5: Configure User Attributes & Claims 1. Click **Edit** in the **Attributes & Claims** section. 2. Add the following mappings: - **Given name** → `first_name` - **Surname** → `last_name` - **Email address** → `email` 3. Click **Save**.
    ![Configuration of attributes](/images/platform/sso-ms-7.png) Configuration of attributes
    ## Step 6: Assign Users and Groups 1. In the **Users and groups** section, click **Add user/group**. 2. Select the users or groups that should have access to CloudQuery. 3. (Optional) To map groups to specific CloudQuery Platform roles, create Microsoft Entra ID Groups for each role level and assign users accordingly. 4. In the CloudQuery Platform SSO settings, set the **Group attribute** field and configure role mappings. See [Map Groups to User Roles](/platform/production-deployment/enabling-single-sign-on-sso/map-groups-to-roles) for full configuration. ## Step 7: Enable User Access 1. Navigate to **Enterprise Applications** → **CloudQuery**. 2. Click **Properties**. 3. Set **Enabled for users to sign in?** to **Yes**. 4. Click **Save**.
    ![Enablement of sign-in](/images/platform/sso-ms-8.png) Enablement of sign-in
    ## Step 8: Save and Test 1. In the CloudQuery Admin panel, click **Save and enable**. 2. In the Microsoft Entra ID portal, click **Test SAML login**. 3. If everything is configured correctly, you can log into CloudQuery Platform with your Microsoft account. ## Next Steps - [Map Groups to Roles](/platform/production-deployment/enabling-single-sign-on-sso/map-groups-to-roles) - Map Entra ID groups to platform roles - [Certificate Rollover](/platform/production-deployment/enabling-single-sign-on-sso/certificate-rollover) - Manage certificate updates - [User Management](/platform/production-deployment/user-management) - Manage users and roles --- Source: https://www.cloudquery.io/docs/platform/production-deployment/enabling-single-sign-on-sso/single-sign-on-with-okta # Single Sign-On with Okta ## Step 1: Set the Base URL 1. In the CloudQuery Platform sidebar, click on your user profile, then select **Organization settings**. 2. Switch to the **Single Sign-On** tab. 3. In the **Base URL** field, enter the HTTPS URL for your platform installation and click **Submit**. - This should be the domain or subdomain where you host CloudQuery Platform, e.g., `https://cloudquery.example.com`.
    ![Configuring the platform base URL in CloudQuery](/images/platform/sso-okta-1.png) Configuring the platform base URL in CloudQuery
    ## Step 2: Create a SAML Application in Okta 1. In a new tab, log in to your Okta Admin Console. 2. Navigate to **Applications** → **Applications**. 3. Click **Create App Integration**. 4. Select **SAML 2.0** as the sign-in method and click **Next**.
    ![Creating a new SAML 2.0 application](/images/platform/sso-okta-2.png) Creating a new SAML 2.0 application
    ## Step 3: Configure SAML Settings In the **General Settings** section, enter `CloudQuery` as the App name, upload a logo (optionally) and click **Next**.
    ![CloudQuery application details](/images/platform/sso-okta-3.png) CloudQuery application details
    Under **SAML Settings**, enter the following: - **Single sign-on URL (ACS URL)**: Copy this value from the CloudQuery Admin panel. - **Audience URI (Entity ID)**: Copy this value from the CloudQuery Admin panel. - **Name ID Format**: Select `EmailAddress`.
    ![SAML settings with values coming from CloudQuery SSO page](/images/platform/sso-okta-4.png) SAML settings with values coming from CloudQuery SSO page
    Scroll down to **Attribute Statements** and add the following: - **`first_name`** → `user.firstName` - **`last_name`** → `user.lastName` - **email** → `user.email`
    ![Mapping CloudQuery attributes with Okta attributes](/images/platform/sso-okta-5.png) Mapping CloudQuery attributes with Okta attributes
    Then, click on **Next**. ## Step 4: Assign Users and Groups 1. In the **Assignments** section, select **Skip group assignment for now** or assign users as needed. 2. Click **Finish**. 3. Navigate to the **Assignments** tab of the CloudQuery application in Okta. 4. Click **Assign** → **Assign to People** or **Assign to Groups** and select users or groups. ## Step 5: Configure Metadata and Sign-Out URL In the newly created application, go to the **Sign On** tab. Then, under **Settings**, find the **Identity Provider metadata** link and copy the **Metadata URL**.
    ![Metadata URL and Sign out URL values to be copied](/images/platform/sso-okta-6.png) Metadata URL and Sign out URL values to be copied
    In the CloudQuery Admin panel, enter the **Metadata URL** instead of uploading an XML file. Then, locate the **Sign-Out URL** in Okta and copy it. In the CloudQuery Admin panel, enter the **Sign-Out URL** to ensure proper logout functionality.
    ![Configuration of Metadata URL and Sign out URL](/images/platform/sso-okta-7.png) Configuration of Metadata URL and Sign out URL
    ## Step 6: Enable and Test SSO 1. In the CloudQuery Admin panel, click **Save and enable**. 2. In the Okta Admin Console, click **Sign On** → **Test Sign In**. 3. If everything is configured correctly, you can log into CloudQuery Platform with your Okta credentials. ## Next Steps - [Map Groups to Roles](/platform/production-deployment/enabling-single-sign-on-sso/map-groups-to-roles) - Map Okta groups to platform roles - [Certificate Rollover](/platform/production-deployment/enabling-single-sign-on-sso/certificate-rollover) - Manage certificate updates - [User Management](/platform/production-deployment/user-management) - Manage users and roles --- Source: https://www.cloudquery.io/docs/platform/production-deployment/platform-settings # Platform Settings The Platform Settings page provides configuration options and system information for your CloudQuery Platform installation. You need an Admin role to access this page. ## Accessing Platform Settings 1. In the sidebar, click your user icon and select **Organization settings**. 2. Navigate to the **Platform settings** tab. ## Outbound IP Addresses This section displays the public IP addresses used by CloudQuery Platform for outbound connections. If your destinations (databases, APIs, or other endpoints) use IP-based allow lists, add these addresses to permit sync traffic. Click **Copy All** to copy the full list to your clipboard. ## Multi-factor Authentication Enforcement Toggle **Enforce Multi-factor authentication for all users** to require MFA across your organization. When you enable enforcement: - All users without MFA configured are logged out immediately - Those users must set up MFA before they can log in again When you disable enforcement, a confirmation dialog appears. Disabling does not remove existing MFA configurations; users who have already set up MFA keep it active. For individual user MFA setup and management, see [Multi-factor Authentication](/platform/production-deployment/user-management/multi-factor-authentication). ## Platform Version This section displays the current platform backend version and UI version. Use this information when contacting support or checking compatibility with plugin versions. For version history and upgrade details, see [Platform Updates](/platform/production-deployment/platform-updates). ## Next Steps - [User Management](/platform/production-deployment/user-management) - Manage users and roles - [Account Profile](/platform/production-deployment/account-profile) - Configure your personal profile settings --- Source: https://www.cloudquery.io/docs/platform/production-deployment/platform-updates # Platform Updates The Platform Updates page shows the version history of your CloudQuery Platform installation. Each entry represents a release with its version name and release date. ## Accessing platform updates 1. In the sidebar, click your user icon and select **Organization settings**. 2. Navigate to the **Platform updates** section. ## Version history The updates page displays a paginated table with: | Column | Description | | --------------- | ----------------------------- | | **Date** | When the version was released | | **New version** | The version identifier | Click any version row to open a detail drawer showing what changed between that version and the previous one. ## Upgrade details The detail drawer shows: - **Version transition**: the previous version and the new version - **Release date**: when the new version was released - **Per-integration upgrades**: for each source or destination integration that was upgraded: - Integration path and kind (source or destination) - Previous version and new version - Affected syncs that use this integration - Link to view the changelog ## Next Steps - [Platform Settings](/platform/production-deployment/platform-settings) - Configure platform settings - [Data Management](/platform/production-deployment/data-management) - Manage data after updates - [Getting Help](/platform/introduction/getting-help) - Contact support for update issues --- Source: https://www.cloudquery.io/docs/platform/production-deployment/resource-ownership # Resource Ownership Resource Ownership lets you define tag keys that represent ownership in your organization. Once configured, these tags appear as filters in Insights and are reflected in resource detail views. ## Accessing Resource Ownership 1. In the sidebar, click your user icon and select **Organization settings**. 2. Navigate to the **Resource ownership** tab. ## Configuring Ownership Tags The page displays a list of tag keys that your organization uses to mark asset ownership. Common examples include `org`, `business_unit`, `team`, and `owner`. To configure: 1. Enter a tag key name in the **Add tag** field. 2. Click **Add another tag key** to add more tags. 3. Arrange the tags in the order you want them displayed in resource details. 4. Click **Save changes**. ### Rules - Tag names must be unique (case-insensitive). - Empty tags are filtered out automatically. - The order of tags is preserved and reflected in resource detail views. ## How Ownership Tags Are Used Once configured, ownership tags appear as additional filter dimensions in the Insights feature. For example, if you define `team` as an ownership tag, you can filter insights by the `team` tag value to see findings for a specific team's resources. Ownership tags do not change how data is synced or stored. They add a filtering layer on top of existing tag data from your cloud providers. ## Next Steps - [Limiting Access to Data](/platform/production-deployment/user-management/limiting-access-to-data) - Control data visibility - [User Management](/platform/production-deployment/user-management) - Manage users and assign roles - [Workspace Roles Overview](/platform/production-deployment/user-management/workspace-roles-overview) - Understand role permissions --- Source: https://www.cloudquery.io/docs/platform/production-deployment/usage # Usage The Usage page shows resource consumption across your CloudQuery Platform installation, broken down by integration. Use it to monitor sync activity, track usage trends, and understand cost drivers. ## Accessing usage data Navigate to **Usage** in the sidebar. This page is available to users with an Admin role. The page displays usage data for the selected time period. ## Time period Select between: - **This month**: usage from the start of the current calendar month to today - **Last month**: usage for the previous calendar month Usage data is aggregated daily. ## What is displayed The usage page shows two views: **Usage graph**: a bar chart showing daily paid rows over the selected period, broken down by integration. Each integration is a separate color in the chart. **Integration table**: a paginated table listing each source and destination integration with its total row count and price category for the period. ### Tracked metrics The Usage page displays **paid rows**: the number of rows synced during the period, broken down by integration. Additional metrics (vCPU seconds, vRAM byte-seconds, network egress) are tracked by the platform API but are not displayed in the Usage page UI. Access these via the [Platform API](https://platform-multi-tenant-api-docs.cloudquery.io/). ## Next Steps - [Data Management](/platform/production-deployment/data-management) - Manage data storage - [Performance Tuning](/platform/advanced-topics/performance-tuning) - Optimize resource usage - [Platform Settings](/platform/production-deployment/platform-settings) - Configure platform settings --- Source: https://www.cloudquery.io/docs/platform/production-deployment/user-management # User Management Grant users access to CloudQuery Platform through the user management interface, or externally using a single sign-on [identity provider](/docs/platform/production-deployment/enabling-single-sign-on-sso). You can grant users access at the Platform level and at the Workspace level. At the moment, only one Workspace called **default** is supported. All users should be added to that Workspace. ## Adding a single sign-on user If you have configured the [single sign-on integration](/docs/platform/production-deployment/enabling-single-sign-on-sso), all configured users can log in and receive the assigned roles to the **default** Workspace automatically. ## Adding a non-SSO user to CloudQuery Platform To add a user, you need to have the **Admin:Write** role on the Platform. Click your profile badge in the left bottom corner and select **Organization settings** from the popup menu. Switch to the **Users** tab.
    ![Users tab in Organization settings](/images/platform/user-management-1.png)
    Click the **Add user** button. Specify the name, email, and password.

    ![Add user form with fields for name, email, password, and role selection including workspace and data access roles](/images/platform/user-management-2.png)

    For roles, select either a built-in [Workspace Role](/docs/platform/production-deployment/user-management/workspace-roles-overview), or check the **Limit data access** and select a [Data Access Role](/docs/platform/production-deployment/user-management/limiting-access-to-data). Make sure the user is **Enabled** and click **Save**. The platform adds the user to the **default** Workspace with [**General:Read**](/docs/platform/production-deployment/user-management/workspace-roles-overview) role. To change it, head to [Changing non-SSO user's Workspace role](#changing-non-sso-users-workspace-role). ## Changing non-SSO user's Workspace role In the users administration page, scroll to the right and click the kebab menu button (the three dots). Select Edit. Select new roles for the user in the popup. ## Programmatic access Users can be provisioned and managed via the Platform API, which is useful for automated onboarding workflows. See the [Platform API Reference](/platform/reference/api-reference) (`users` and `admin` sections) for endpoint details. ## Next Steps - [Workspace Roles Overview](/platform/production-deployment/user-management/workspace-roles-overview) - Understand roles and permissions - [Limiting Access to Data](/platform/production-deployment/user-management/limiting-access-to-data) - Restrict data access for teams - [Multi-Factor Authentication](/platform/production-deployment/user-management/multi-factor-authentication) - Enable MFA for users - [Single Sign-On](/platform/production-deployment/enabling-single-sign-on-sso) - Configure SSO for automated provisioning --- Source: https://www.cloudquery.io/docs/platform/production-deployment/user-management/limiting-access-to-data # Limiting Access to Data You can limit what data individual users can see and query using **data permissions** and **custom roles**. Once set, users with custom roles will be able to perform the activities of users with the [`General:Read`](./workspace-roles-overview#generalread) permission, but they only see data their permissions allow. This includes all parts of the Platform, including the Asset Inventory, SQL Console, and Reports. This feature applies only to data synced to CloudQuery Platform. If you sync to an external database, you need to manage the data access yourself. ## Data access controls overview Data access controls consist of: - **Data permissions**: individual permissions granting access to a defined set of data in the CloudQuery Platform database. - **Custom roles**: containers for individual data permissions. ### Data Permissions Data permissions grant access to a subset of data synced to CloudQuery Platform. You define data permissions dynamically using SQL as a set of `_cq_platform_id` values each permission grants access to. For example, a data permission defined as ```sql SELECT _cq_platform_id FROM cloud_assets WHERE resource_type = 'aws_ec2_instances' ``` grants access to data about all EC2 instances, but nothing else. ### Custom roles Custom roles are a collection of one or more data permissions. They function as a special case of the [GENERAL:READ](./workspace-roles-overview#generalread) role. Users with custom roles can perform all `General:Read` activities, but only see data granted by their assigned permissions. Assign roles in [User Management](./), or automatically through an [SSO identity provider](../enabling-single-sign-on-sso/map-groups-to-roles). A user can have multiple custom roles assigned. However, for the roles to have an effect, the user must not have any of the built-in roles assigned.

    ![data access role definition example](/images/platform/data-access-role-diagram.png)

    For a custom role to have any effect, the user must not have any built-in role assigned. Custom roles automatically inherit all permissions from the `General:Read` role. ## Configuring Data Access from scratch ### Planning When planning data access restrictions, consider: - What is the primary criteria you want to distinguish by? Is it available on all assets? For example, if you decide to limit access by region, not all assets actually have a region available and you may need to include them in the data access permissions in another way. - What granularity do you require for the data access permissions? You can have 1-1 mapping of permissions to roles, or you can make them more granular and then use various combinations of permissions in custom roles. - Are you going to need to restrict access to some specific data? Data permissions are inclusive, meaning you cannot restrict access to specific assets with one permission once it is granted with another one for the same role. ### Defining a data permission Data permissions rely on the `_cq_platform_id` of assets, which are deterministic unique identifiers of all resources in the Platform database. The Permissions are defined as SQL queries returning the set of the identifiers in the `_cq_platform_id` column. For most of the use cases, you should be able to rely on the `cloud_assets` view to define the permissions, but you can combine it with arbitrary tables available on the platform. For example, to create a Data Access Permission that grants access only to AWS EC2 Instances in the `us-east-1` region, you can specify the permission either as ```sql SELECT _cq_platform_id, region FROM aws_ec2_instances WHERE region='us-east-1' ``` or ```sql select _cq_platform_id from cloud_assets where region = 'us-east-1' and resource_type='aws_ec2_instances' ``` To create a new data permission, navigate to **Organization settings** > **Data permissions**. Click **Add data permission** and fill in the name and description.

    ![Data Access Permission details](/images/platform/dap-1.png)

    Click **Open SQL Console** to start defining the SQL query for this permission. The SQL Console will open in its data permission definition mode (identified by the **Add to data permission** button in the top right). Write the query and execute it. The Console will warn you if the resulting table does not use the `_cq_platform_id` column. You cannot name an arbitrary column `_cq_platform_id`, you need to use the column from the database. You should see the results of the query in the bottom part of the screen. These are the assets the permission will grant the access to.

    ![SQL Console with a data access permission definition](/images/platform/dap-2.png)

    When ready, click **Add to data permission** and you'll be taken back to the data permission screen. As confirmation, you will see the count of resources this permission grants access to. To change the query, click **Edit Query**.

    ![Data Access Permission details](/images/platform/dap-3.png)

    Confirm creation of the data permission by clicking **Create data permission**. Next, assign the permission to a role. ### Defining a custom role To create a new custom role, navigate to **Organization settings** > **Roles**. Click **Create Role** and fill in the name and description.

    ![Data Access Role details](/images/platform/dap-4.png)

    Click **Add data permission** and select the permission to add to this role from the panel. You can select and add multiple permissions at the same time. Click **Add selected data permissions** and then finish creation of the role with **Create**. Next, you need to assign the role to a user. If your users log in using Single sign-on, you will need to [Map Groups to User Roles ](../enabling-single-sign-on-sso/map-groups-to-roles.mdx). Otherwise, see [Changing Non-SSO User's Workspace Role](./#changing-non-sso-users-workspace-role). ## Next Steps - [Workspace Roles Overview](/platform/production-deployment/user-management/workspace-roles-overview) - Understand available roles - [Resource Ownership](/platform/production-deployment/resource-ownership) - Control who owns integrations and syncs - [Policies](/platform/features/policies) - Combine data access controls with compliance policies --- Source: https://www.cloudquery.io/docs/platform/production-deployment/user-management/multi-factor-authentication # Multi-factor Authentication Multi-factor authentication (MFA) protects CloudQuery Platform accounts with a second verification step. Individual users can enable it for themselves, or Organization Administrators can enforce it for all users. Both standard Platform users and SSO users can use multi-factor authentication. ### Prerequisites - Active CloudQuery Platform account - Access to a mobile device or authenticator app - Admin permissions (for organization-wide enforcement) ## For Organization Administrators ### View MFA adoption across organization To see the status of individual user's MFA, navigate to the **Organization settings**. In the Users tab, the **MFA Status** column shows each user's status. ### **Enforcing MFA Organization-Wide** Enforcing multi-factor authentication for all users will terminate all active sessions and will force the users to enable the multi-factor authentication before they can continue using the Platform. To enforce multi-factor authentication for all users, you need to have the [Admin](workspace-roles-overview) role. Navigate to the **Organization settings**. In the **Platform settings** tab scroll down to the **Multi-factor Authentication** section and click the **Enforce Multi-factor authentication for all users** toggle to enforce the policy.
    ![Enforcing MFA Organization-Wide](/images/platform/mfa-1.png)
    ### Reset a user's MFA If a user loses access to their authenticator, Admin can reset their multi-factor authentication from the Organization settings. In the Users tab, click the actions menu on the right, and select **Reset MFA**. ## For End Users ### **Configuring your multi-factor authentication app** To set up multi-factor authentication for your user account, you will need an app on your phone or in your password manager that supports TOTP protocol, such as Google Authenticator, Microsoft Authenticator, or 1Password plugin in your browser. If you are not prompted to set up MFA by the Platform, you can opt in from **Account Settings**. Switch to the **Multi-factor Authentication** tab. You will see a QR code and an alternative text representation. Scan the QR code with your authenticator app or enter the alternative code manually. Your authenticator will show a 6-digit code that will change every 30 seconds. Enter the code in the input below the QR code and click the **Set Up MFA** button. A confirmation screen shows that MFA is now active.
    ![Multi-factor authentication confirmation screen showing MFA has been successfully enabled](/images/platform/mfa-2.png)
    ### If the verification code is not accepted If the verification code is not accepted, try reloading the page and scanning the QR code again. Make sure the date and time on your device are correct. ### If you lose your authenticator If you lose your authenticator, contact your organization administrator to reset the multi-factor authentication status and set it up again on a new device. ## Next Steps - [Single Sign-On](/platform/production-deployment/enabling-single-sign-on-sso) - Configure SSO for stronger authentication - [User Management](/platform/production-deployment/user-management) - Manage users and security settings - [Audit Log](/platform/production-deployment/audit-log) - Monitor authentication events --- Source: https://www.cloudquery.io/docs/platform/production-deployment/user-management/workspace-roles-overview # Workspace Roles Overview CloudQuery Platform controls access through **built-in feature roles**, which decide what a user can do, and **data access roles**, which decide which rows of synced data a user can see. This page describes the built-in feature roles. For data access roles, see [Limiting Access to Data](/platform/production-deployment/user-management/limiting-access-to-data). There are four general-purpose roles (General:Read, General:Write, Admin:Read, Admin:Write) and two special-purpose roles (CI and Schema Only). Roles are additive, not restrictive. If a user is assigned both `Admin:Read` and `General:Read`, they receive the combined permissions of both. To restrict a user, assign only the most limited role they need. ## Permissions at a glance The four general-purpose roles form a gradient: `General:Read` and `Admin:Read` are read-only; `General:Write` adds operational write access (integrations, syncs, policies); `Admin:Write` adds administration (users, roles, settings, billing, API keys, public reports, and SSO). | Capability | General:Read | General:Write | Admin:Read | Admin:Write | | --- | :---: | :---: | :---: | :---: | | View Asset Inventory data | Yes | Yes | Yes | Yes | | Run queries in the SQL Console | Yes | Yes | Yes | Yes | | Save personal SQL queries and filters¹ | Yes | Yes | Yes | Yes | | Save shared (public) queries and filters | No | Yes | No | Yes | | Use the AI Assistant | Yes | Yes | Yes | Yes | | View Insights, Policies, and Reports | Yes | Yes | Yes | Yes | | View integrations, destinations, syncs, and users | Yes | Yes | Yes | Yes | | Create, update, and delete integrations, destinations, and syncs | No | Yes | No | Yes | | Create and manage Policies, Alerts, and custom columns | No | Yes | No | Yes | | Silence and unsilence Insights³ | No | Yes | No | Yes | | Manage users, roles, and data permissions | No | No | No | Yes | | Manage platform settings and billing | No | No | No | Yes | | Manage notification destinations | No | No | No | Yes | | Create and manage own private reports² | Yes | Yes | Yes | Yes | | Create and manage public reports | No | No | No | Yes | | Create Workspace API keys | No | No | No | Yes | | Configure SSO / SAML | No | No | No | Yes | ¹ Read roles (`General:Read`, `Admin:Read`) can create and edit only their **own private** queries and filters. Creating or modifying shared (public) ones requires write access. ² Any role can create reports, but only as **private** reports visible to and managed by their creator. Creating or managing **public** reports requires `Admin:Write`. ³ Insight silences are **workspace-wide**: silencing or unsilencing an insight changes it for everyone in the workspace, not just the acting user. A silence can be indefinite or a timed snooze that re-surfaces automatically. All roles can view silenced insights and list the active silences (and how their resources were selected); only write roles can create, change, or remove them. Data visibility is enforced separately. Even when a role grants access to a feature, the rows a user sees are filtered by their assigned [data access roles](/platform/production-deployment/user-management/limiting-access-to-data). ## Built-in Roles ### General:Read The most restricted built-in role. A user with this role can: * View data in the Asset Inventory * Create and save their own search filters in the Asset Inventory * Query data in the SQL Console * Create and save their own SQL queries in the SQL Console * Create their own private reports * Use the AI Assistant * View Insights, Policies, and Reports * View the list of integrations, destinations, and syncs and their status A `General:Read` user can edit and delete their own saved queries, filters, and reports, but cannot create or modify shared (public) queries or public reports. ### General:Write In addition to General:Read capabilities, a user with this role can: * Create, update, and delete integrations, destinations, and syncs * Create and manage Policies, Alerts, and custom columns * Silence and unsilence Insights (workspace-wide) * Save shared (public) SQL queries and search filters A `General:Write` user cannot perform administrative actions: managing users, roles, or data permissions; changing platform settings or billing; managing notification destinations; creating Workspace API keys; managing public reports; or configuring SSO. Those require `Admin:Write`. ### Admin:Read A read-only role. It has the same view access as `General:Read`, including viewing workspace users and Usage in Organization settings, and cannot change any configuration. It is intended for auditors and stakeholders who need visibility without write access. ### Admin:Write The full administrative role. In addition to all `General:Write` capabilities, a user with this role can: * Add, update, and delete users * Manage roles and data permissions * Manage platform settings and billing * Manage notification destinations * Create and manage public reports * Create Workspace API keys with any role level * Configure SSO / SAML ## Additional Role Types Beyond the four general-purpose roles, the platform supports these role types: * **CI**: The default role for API keys used in automation such as CI/CD pipelines. A CI key can view plugin and integration metadata, manage sync sources, destinations, and syncs, and trigger and monitor sync runs. It has no access to Asset Inventory data, the SQL Console, Reports, or administration features. * **Schema Only**: A read-only role that grants access to the schema of Asset Inventory tables (the list of tables and their column definitions) without granting access to the underlying asset data. It is intended for tools and integrations that need to understand the data model but must not read the synced data. * **Custom**: Roles created through [data access permissions](/platform/production-deployment/user-management/limiting-access-to-data). Custom roles combine data permissions to restrict which data a user can see while granting General:Read-level feature access. For a custom role to take effect, the user must not be assigned any built-in role. For more on custom roles and data permissions, see [Limiting Access to Data](/platform/production-deployment/user-management/limiting-access-to-data). ## Programmatic access Roles and permissions can be managed via the Platform API. See the [Platform API Reference](/platform/reference/api-reference) (`rbac` section) for endpoint details on listing, creating, and assigning roles programmatically. The AI Assistant is only available to signed-in users in the dashboard. It cannot be used with a Workspace API key, regardless of the key's role. ## Next Steps - [Limiting Access to Data](/platform/production-deployment/user-management/limiting-access-to-data) - Restrict what data roles can see - [Map Groups to Roles](/platform/production-deployment/enabling-single-sign-on-sso/map-groups-to-roles) - Automate role assignment with SSO groups - [User Management](/platform/production-deployment/user-management) - Invite and manage users --- Source: https://www.cloudquery.io/docs/platform/quickstart/creating-a-new-account # Creating a New Account ## Sign up Sign up at [cloudquery.io](https://cloud.cloudquery.io/auth/platform-trial). After your email is validated, CloudQuery will provision a new workspace for you. This should not take more than a minute. ## Next steps Set up your first [sync](./your-first-sync.md). You can also: - Learn about [Integrations](../core-concepts/integrations), [Syncs](../core-concepts/syncs), and other core concepts - Configure [Single Sign-On](../production-deployment/enabling-single-sign-on-sso) for your team --- Source: https://www.cloudquery.io/docs/platform/quickstart/your-first-sync # Your First Sync Set up your first sync and start querying cloud resources in CloudQuery Platform. ## Prerequisites - A CloudQuery Platform account ([create one](/platform/quickstart/creating-a-new-account) if you don't have one) - Credentials for at least one cloud provider (AWS, GCP, or Azure) ## Step 1: Set up an integration An integration connects CloudQuery Platform to a cloud provider. Choose your provider and follow the corresponding guide: - [AWS Integration](/platform/integration-guides/setting-up-an-aws-integration): requires an IAM role with cross-account access - [GCP Integration](/platform/integration-guides/setting-up-a-gcp-integration): requires a service account with appropriate roles - [Azure Integration](/platform/integration-guides/setting-up-an-azure-integration): requires a service principal with Reader access If you want to start with a different provider, see the [General Integration Setup Guide](/platform/integration-guides/general-integration-setup-guide) for instructions that apply to any supported source. ## Step 2: Create a sync A sync connects your integration to a destination and runs on a schedule. 1. Navigate to **Data Pipelines** → **Integrations** and click **Create Integration**. 2. Select **Use existing integration** and choose the integration you created in Step 1. 3. Select the **default ClickHouse destination** (included with CloudQuery Platform). 4. Give the sync a name. 5. Set a schedule: **Daily** is a good starting point. You can also set **No schedule** and trigger runs manually. 6. Click **Save and run** to start the sync immediately. For a more detailed walkthrough of sync options, see [Setting up a Sync](/platform/syncs/setting-up-a-sync). ## Step 3: Monitor the sync After starting the sync, the platform opens the **Sync Runs** page. Here you can see: - The sync status (running, completed, failed) - Duration and row counts - Error details if the sync fails The first sync may take several minutes depending on the number of resources in your cloud account. ## Step 4: View your data Once the sync completes, your cloud resources are available in two places: ### Asset Inventory Navigate to **Asset Inventory** in the sidebar. You should see your cloud resources organized by category (Compute, Storage, Networking, etc.). Use the search bar to find specific resources. For more details, see [Asset Inventory](/platform/features/asset-inventory). ### SQL Console Navigate to **SQL Console** in the sidebar. Try a query against your synced data: ```sql SELECT cloud, account, region, resource_type, count(*) as total FROM cloud_assets GROUP BY cloud, account, region, resource_type ORDER BY total DESC LIMIT 20 ``` This shows a summary of all synced resources by provider, account, region, and type. For more on writing queries, see [SQL Console](/platform/features/sql-console). ## Next steps - [Ask the AI Assistant](/platform/features/ai-assistant) questions about your synced data in plain language - [Set up additional integrations](/platform/integration-guides) for other cloud providers - [Create policies](/platform/features/policies) to detect misconfigurations - [Configure alerts](/platform/features/alerts) to get notified about policy violations - [Explore query examples](/platform/features/sql-console/query-examples) for security, compliance, and cost optimization - [Insights](/platform/features/insights): Review security, compliance, and cost findings generated from your synced data ## Automate with the API Everything you just did through the UI (setting up integrations, triggering syncs, managing policies) can also be done programmatically via the Platform API. This lets you treat CloudQuery Platform as infrastructure: automate it, drive it from CI/CD pipelines, and version-control your configuration. See [Platform API](/platform/reference/api-reference) to get started. ## Pricing and access You've synced cloud data for the first time and queried it in the SQL Console. CloudQuery Platform does this on a schedule: every asset across every account, updated after every sync, always queryable. To sync more cloud providers, scale to more accounts, or discuss pricing for your team, [contact us](https://www.cloudquery.io/contact-us) or [view pricing](https://www.cloudquery.io/pricing). --- Source: https://www.cloudquery.io/docs/platform/reference/api-reference # Platform API The CloudQuery Platform exposes a REST API that gives you full programmatic control over the platform. **Nearly everything you can do in the CloudQuery Platform UI, you can do through the API**: syncs, integrations, policies, alerts, users, reports, and more. This means you can manage CloudQuery Platform the same way you manage other infrastructure: automate it, version-control it, drive it from CI/CD pipelines, and integrate it with your existing tooling. **Full interactive reference**: [Platform API Reference ↗](https://platform-multi-tenant-api-docs.cloudquery.io/) --- ## Platform API vs CloudQuery Cloud API CloudQuery has two separate APIs. Here's how to pick the right one: - **Managing syncs, users, policies, alerts, or integrations in your Platform deployment?** → You want the **Platform API** (this page). - **Managing your cloudquery.io account (billing, teams, plugin registry)?** → You want the **[CloudQuery Cloud API ↗](https://api-docs.cloudquery.io/)**. | | Platform API | CloudQuery Cloud API | |---|---|---| | **What it controls** | The CloudQuery Platform itself: syncs, users, policies, reports, alerts, integrations | CloudQuery.io account resources: billing, organizations, plugin registry | | **Base URL** | `https:///api` | `https://api.cloudquery.io` | | **Auth** | Platform API key | CloudQuery Cloud API key | | **Interactive docs** | [Platform API Reference ↗](https://platform-multi-tenant-api-docs.cloudquery.io/) | [CloudQuery Cloud API Reference ↗](https://api-docs.cloudquery.io/) | If you're managing your CloudQuery Platform deployment (running syncs, managing users, setting policies), you want the **Platform API**. This page covers the Platform API. --- ## When to use the API The Platform API is useful any time you want to manage CloudQuery Platform outside the UI: - **CI/CD automation**: trigger syncs on a schedule or as part of a pipeline, check sync status, and fail builds when policies are violated - **Infrastructure as code**: manage integrations, policies, alerts, and reports as versioned config rather than clicking through the UI - **User provisioning**: automate onboarding and offboarding, sync users from your identity provider, assign roles programmatically - **Audit and compliance**: pull audit logs into your SIEM or data warehouse, query compliance policy results programmatically - **Custom tooling**: build internal dashboards, Slack bots, or scripts that surface CloudQuery data and trigger actions --- ## Authentication All API requests require an API key passed in the `Authorization` header: ```bash Authorization: Bearer ``` Create an API key in **Organization Settings → API Keys**. See [API Keys](/platform/production-deployment/api-keys) for instructions. ### API key roles API keys are assigned a role that controls what they can access: | Role | Access | |---|---| | `ci` | CI/CD pipeline access: trigger syncs, check policy results | | `general:read` | Read-only access to syncs, assets, reports, and policies | | `general:write` | Read and write access to non-admin features | | `admin:read` | Read-only access including user and platform settings | | `admin:write` | Full access including user management, SSO configuration, and platform settings | | `schema-only` | Access to table schemas only | Choose the least-privileged role for your use case. See [API Keys](/platform/production-deployment/api-keys) for the full list and instructions. Some features (such as the AI Query Writer) are only available in the UI and cannot be called via API. --- ## Base URL ``` https:///api ``` Replace `` with your CloudQuery Platform deployment URL (e.g. `cloudquery.mycompany.com`). --- ## Common use cases ### Trigger a sync Syncs are identified by name. POST to `/syncs//runs` to start a new run: ```bash curl -s -X POST \ -H "Authorization: Bearer " \ https:///api/syncs//runs ``` ### List syncs and check status ```bash curl -s \ -H "Authorization: Bearer " \ https:///api/syncs ``` ### Fetch audit log entries ```bash curl -s \ -H "Authorization: Bearer " \ "https:///api/audit-logs?per_page=50&page=1" ``` ### List users ```bash curl -s \ -H "Authorization: Bearer " \ https:///api/users ``` ### Create a policy ```bash curl -s -X POST \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"name": "My Policy", "query": "SELECT ..."}' \ https:///api/policies ``` For the full list of endpoints, request/response schemas, and a live API explorer, see the [Platform API Reference ↗](https://platform-multi-tenant-api-docs.cloudquery.io/). --- ## API categories The Platform API is organized into the following categories: | Category | What it covers | |---|---| | `syncs` | Create, manage, trigger, and monitor sync jobs | | `plugins` | Manage installed integrations | | `policies` | Create and manage compliance policies and policy groups | | `alerts` | Configure alert rules, notification thresholds, and notification destinations | | `audit-logs` | Query the audit log for user and API activity | | `rbac` | Manage roles and permissions | | `users` | Provision and manage user accounts | | `teams` | Manage team membership and settings | | `reports` | Access and manage reports and report templates | | `queries` | Manage saved SQL queries, filters, and tags | | `custom-columns` | Create custom columns on asset tables | | `admin` | Platform-wide settings, SAML SSO, and user administration | | `healthcheck` | Platform health and version info | --- ## Common patterns ### Pagination Most list endpoints support `per_page` and `page` query parameters: ```bash curl -s \ -H "Authorization: Bearer " \ "https:///api/audit-logs?per_page=100&page=1" ``` ### Error responses The API returns standard HTTP status codes. Error bodies follow this structure: ```json { "message": "Description of the error", "status": 404 } ``` --- ## Next steps - [API Keys](/platform/production-deployment/api-keys): create and manage API keys - [Audit Log](/platform/production-deployment/audit-log): track API usage - [Policies](/platform/features/policies): manage compliance policies via API - [User Management](/platform/production-deployment/user-management): provision users and manage roles - [Platform API Reference ↗](https://platform-multi-tenant-api-docs.cloudquery.io/): complete interactive documentation --- Source: https://www.cloudquery.io/docs/platform/reference/troubleshooting # Troubleshooting ## Sync issues ### A sync is stuck in "running" state A sync may appear stuck if it is processing a large number of resources, or if the underlying process has stalled. - Check the **Sync Runs** page for any error details in the run log. - If the sync has been running longer than expected with no progress, try cancelling it and running it again. - For large cloud accounts, the first sync can take significantly longer than subsequent ones. Allow extra time before concluding something is wrong. ### A sync failed Open the sync run from **Data Pipelines → Integrations** and expand the failed run to see the error message. Common causes: | Error | Likely cause | Fix | |---|---|---| | `permission denied` / `403` | Integration credentials lack required permissions | Review the IAM role or service account permissions for your provider | | `credentials expired` / `401` | Credentials have rotated or expired | Re-enter credentials in the integration settings | | `connection timeout` | Network connectivity issue between the platform and the cloud provider | Check for outbound network restrictions; retry the sync | | `quota exceeded` | Cloud provider API rate limit hit | Reduce the number of tables being synced, or schedule syncs less frequently | For provider-specific permission requirements, see the relevant [Integration Guide](/platform/integration-guides). ### Resources are missing from the Asset Inventory after a sync completes - Confirm the sync completed without errors. Partial failures may result in some tables being skipped. - Check that the integration is scoped to the correct accounts and regions. Resources in accounts or regions not covered by the integration will not appear. - Some resource types may not be included in your integration's table selection. Review the table configuration in the integration settings. - The Asset Inventory reflects the most recent completed sync. If you are expecting newly created resources, run a fresh sync. --- ## Policy issues ### A policy is not showing any violations - Verify the policy query returns results when run manually in the [SQL Console](/platform/features/sql-console). If the query returns no rows, there are no violations, which may be correct. - Check that the tables referenced in the query have been synced. If the integration doesn't include those tables, the query will return empty results. - Confirm the sync has run recently. Policies evaluate against the latest synced data; stale data may not reflect current infrastructure state. ### A policy is showing unexpected violations - Run the policy query directly in the SQL Console to inspect the results. - Check whether the policy scope is set to a saved filter that is broader or narrower than intended. - Verify the query logic. Policies return one row per violation, so a query that inadvertently joins tables or doesn't filter correctly may return duplicate or incorrect results. --- ## API issues ### API requests return `401 Unauthorized` - Confirm you are passing the API key in the `Authorization` header as `Bearer `. - Check that the API key has not been revoked. See [API Keys](/platform/production-deployment/api-keys). - Ensure you are using a **Platform API key**, not a CloudQuery Cloud API key. These are different and not interchangeable. See [Platform API](/platform/reference/api-reference). ### API requests return `403 Forbidden` The API key role does not have permission for the requested operation. - `general:read` and `general:write` keys cannot access user management, SSO configuration, or admin endpoints. - `admin:read` keys have read-only access to admin resources and cannot create or modify anything. - `ci` keys are scoped to CI/CD operations and cannot access admin or user management endpoints. Create a new key with the appropriate role from **Organization Settings → API Keys**. See [API Keys](/platform/production-deployment/api-keys) for the full role reference. ### API requests return `404 Not Found` - Double-check the endpoint path and any path parameters (team name, sync ID, etc.). - Confirm you are using the correct base URL for your deployment: `https:///api`. --- ## SSO issues ### Users can't log in after SSO is configured - Verify the SAML metadata URL or XML is correct and up to date in the SSO configuration. - Confirm that the identity provider's assertion consumer service (ACS) URL matches what is configured in CloudQuery Platform. - Check that the `NameID` format in your IdP matches the expected format (typically email address). - For group-based role mapping, confirm the group attribute name matches what your IdP sends in the SAML assertion. See [Enabling Single Sign-On](/platform/production-deployment/enabling-single-sign-on-sso) for full configuration details. ### SSO is configured but users are not getting the right roles - Review the group-to-role mapping in **Organization Settings → SSO**. - Confirm that your identity provider is sending the expected group claims in the SAML assertion. - Users receive the role mapped to their group at login time. Changes to mapping take effect on next login. See [Map Groups to Roles](/platform/production-deployment/enabling-single-sign-on-sso/map-groups-to-roles) for setup details. --- ## Getting more help If you can't resolve an issue using this guide: - **Service Status**: Check [status.cloudquery.io](https://status.cloudquery.io/) for any ongoing incidents, maintenance windows, or service disruptions that may be affecting your syncs. - **Community Forum**: [community.cloudquery.io](https://community.cloudquery.io): browse existing threads or post a question - **GitHub Issues**: [github.com/cloudquery/cloudquery](https://github.com/cloudquery/cloudquery): report bugs or check known issues - **Getting Help**: [Getting Help](/platform/introduction/getting-help): full list of support resources --- Source: https://www.cloudquery.io/docs/platform/syncs/monitoring-sync-status # Monitoring Sync Status To see ongoing and previous sync runs (executions), navigate to **Data Pipelines** → **Integrations**, then select an integration and open its **Syncs** tab.
    ![CloudQuery Platform Data Pipelines page listing all configured syncs with their status](/images/platform/monitoring-1.png)
    From there, select a sync you want to see the runs for. You will see a list of previous sync runs, their status, how long they took, how many rows the sync transferred and how many errors occurred.
    ![List of previous sync runs showing status, duration, rows synced, and error counts](/images/platform/monitoring-2.png)
    Click a sync run to see its details. The detail drawer shows the tables synced, how long each table took to sync, and how many rows and errors it had.
    ![Sync run detail drawer showing per-table sync duration, row counts, and error breakdown](/images/platform/monitoring-3.png)
    To see the details of the errors reported during the sync, switch to Logs.
    ![Sync run logs tab displaying error details and diagnostic messages from the sync execution](/images/platform/monitoring-4.png)
    After a sync completes, check [Insights](/platform/features/insights) for new security, compliance, and cost findings generated from the synced data. ## Next Steps - [Setting up a Sync](/platform/syncs/setting-up-a-sync) - Create and configure new syncs - [Alerts](/platform/features/alerts) - Get notified about sync failures or data conditions - [Performance Tuning](/platform/advanced-topics/performance-tuning) - Optimize slow syncs - [Troubleshooting](/platform/reference/troubleshooting) - Diagnose common sync failures and errors - [Getting Help](/platform/introduction/getting-help) - Contact support for persistent sync issues --- Source: https://www.cloudquery.io/hub/addons/transformation/cloudquery/aws-asset-inventory/latest/docs # CloudQuery × dbt: AWS Asset Inventory Package ## Overview This AWS Asset Inventory package works on top of the CloudQuery framework. This package offers automated line-item listing of all active resources in your AWS environment. This package supports PostgreSQL database only. We recommend using this transformation with our [AWS Asset Inventory Dashboard](https://hub.cloudquery.io/addons/visualization/cloudquery/aws-asset-inventory/latest/docs) ![AWS Asset Inventory Dashboard](https://storage.googleapis.com/cq-cloud-images/cloudquery/f939272f17c16ff2440024c36250992cdc9f95bd_asset_inventory_dash.png) ### Example Queries Which accounts have the most resources? (PostgreSQL) ```sql select account_id, count(*) from aws_resources group by account_id order by count(*) desc ``` Which services are most used in each account? (PostgreSQL) ```sql select account_id, service, count(*) from aws_resources group by account_id, service order by count(*) desc; ``` Which resources are not tagged? (PostgreSQL) ```sql select * from aws_resources where tags is null or tags = '{}'; ``` ### Requirements - [CloudQuery](https://docs.cloudquery.io/docs/quickstart/) - [CloudQuery AWS plugin](https://hub.cloudquery.io/plugins/source/cloudquery/aws) - [A CloudQuery Account](https://www.cloudquery.io/auth/register) - [dbt](https://docs.getdbt.com/docs/core/pip-install) - [PostgreSQL](https://hub.cloudquery.io/plugins/destination/cloudquery/postgresql) ### Models Included - **aws_resources**: AWS Resources View, available for PostgreSQL. - Required tables: This model has no specific table dependencies, other than requiring a single CloudQuery table from the AWS plugin that has an ARN. #### Columns Included - `_cq_id` - `_cq_source_name` - `_cq_sync_time` - `account_id` - `request_account_id` - `type` - `arn` - `region` - `tags` - `partition` - `service` - `_cq_table` ## To run this package you need to complete the following steps ### 1. Download and extract this package. Click the **Download now** button on the top. You may need to create a CloudQuery account first. ### 2. Install DBT and set up the DBT profile [Install `dbt`](https://docs.getdbt.com/docs/core/pip-install): ```bash pip install dbt-postgres ``` Create the profile directory: ```bash mkdir -p ~/.dbt ``` Create a `profiles.yml` file in your profile directory (e.g. `~/.dbt/profiles.yml`): ```yaml aws_asset_inventory: # This should match the name in your dbt_project.yml target: dev outputs: dev: type: postgres host: 127.0.0.1 user: postgres pass: pass port: 5432 dbname: postgres schema: public # default schema where dbt will build the models threads: 1 # number of threads to use when running in parallel ``` Test the Connection: After setting up your `profiles.yml`, you should test the connection to ensure everything is configured correctly. First, switch to the directory where you extracted this package. Then run this command: ```bash dbt debug ``` This command will tell you if dbt can successfully connect to your PostgreSQL instance: ``` ... 09:37:00 retries: 1 09:37:00 Registered adapter: postgres=1.9.0 09:37:00 Connection test: [OK connection ok] 09:37:00 All checks passed! ``` ### 3. Login to CloudQuery To run a sync with CloudQuery, you will need to create an account and log in. ``` cloudquery login ``` ### 4. Sync AWS data This is an example sync config with the minimum set of tables for this transformation. Save this to a file named `aws.yaml`. For detailed AWS authentication and configuration options and additional tables to add to the sync config, see the [AWS plugin documentation](https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs). ```yml kind: source spec: name: aws # The source type, in this case, AWS. path: cloudquery/aws # The plugin path for handling AWS sources. registry: cloudquery # The registry from which the AWS plugin is sourced. version: "v32.38.0" # The version of the AWS plugin. tables: ["aws_ec2_instances"] # Include any tables that meet your requirements, separated by commas destinations: ["postgresql"] # The destination for the data, in this case, PostgreSQL. spec: --- kind: destination spec: name: "postgresql" # The type of destination, in this case, PostgreSQL. path: "cloudquery/postgresql" # The plugin path for handling PostgreSQL as a destination. registry: "cloudquery" # The registry from which the PostgreSQL plugin is sourced. version: "v8.9.0" # The version of the PostgreSQL plugin. spec: connection_string: "${POSTGRESQL_CONNECTION_STRING}" # set the environment variable in a format like # postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable # You can also specify the connection string in DSN format, which allows for special characters in the password: # connection_string: "user=postgres password=pass+0-[word host=localhost port=5432 dbname=postgres" ``` Run the sync: ```shell cloudquery sync aws.yaml ``` ### 5. Create the views Navigate to your dbt project directory, where you extracted this package. Before executing the `dbt run` command, it might be useful to check for any potential issues: ```bash dbt compile ``` If everything compiles without errors, you can then execute: ```bash dbt run ``` This command will run your `dbt` models and create tables/views in your destination database as defined in your models. **Note:** If running locally, ensure you are using `dbt-core` and not `dbt-cloud-cli` as dbt-core does not require extra authentication. ### 6. Explore the data Connect to your PostgreSQL database and start exploring the data ```sql select * from aws_resources limit 10 ``` ### 7. Set up Grafana dashboard (Optional) Follow the instructions in [AWS Asset Inventory Dashboard](https://hub.cloudquery.io/addons/visualization/cloudquery/aws-asset-inventory/latest/docs) to set up a dashboard in Grafana. ### 8. Production deployment This transformation creates a view on top of the database tables synced by CloudQuery CLI. You need to re-run the transformation (using the `dbt run` command) only if you add or remove tables from the sync. --- Source: https://www.cloudquery.io/hub/addons/transformation/cloudquery/aws-cost/latest/docs # AWS Cost Policy This AWS Cost Policy package is designed to help you analyze and optimize your AWS spending. By leveraging CloudQuery, Cost and Usage Report, and DBT, AWS Cost Policy provides insightful views into your AWS usage and costs, identifying under-utilized resources, and allocating costs based on tags. This tool is ideal for cloud engineers, finance teams, and anyone looking to gain better visibility into their AWS costs. ## Prerequisites Before you begin, ensure you have the following: - An active AWS account with [Data Exports from Billing and Cost Management](https://docs.aws.amazon.com/cur/latest/userguide/what-is-data-exports.html?icmpid=docs_costmanagement_hp-dataexports-overview) enabled. - [CloudQuery](https://docs.cloudquery.io/docs/quickstart/) - [CloudQuery AWS plugin](https://hub.cloudquery.io/plugins/source/cloudquery/aws) - [CloudQuery S3 plugin](https://hub.cloudquery.io/plugins/source/cloudquery/s3) - [A CloudQuery Account](https://www.cloudquery.io/auth/register) - [dbt](https://docs.getdbt.com/docs/core/pip-install) - [PostgreSQL](https://hub.cloudquery.io/plugins/destination/cloudquery/postgresql) - A Postgresql instance running (Local or cloud based). [Running Postgres with Docker](https://www.docker.com/blog/how-to-use-the-postgres-docker-official-image/) - Basic familiarity with YAML and SQL. ## To run the policy you need to complete the following steps ### 1. Download and extract this package. Click the **Download now** button on the top. You may need to create a CloudQuery account first. ### 2. Install DBT and set up the DBT profile [Install `dbt`](https://docs.getdbt.com/docs/core/pip-install): ```bash pip install dbt-postgres ``` Create the profile directory: ```bash mkdir -p ~/.dbt ``` Create a `profiles.yml` file in your profile directory (e.g. `~/.dbt/profiles.yml`): ```yaml aws_cost: # This should match the name in your dbt_project.yml target: dev outputs: dev: type: postgres host: 127.0.0.1 user: postgres pass: pass port: 5432 dbname: postgres schema: public # default schema where dbt will build the models threads: 1 # number of threads to use when running in parallel ``` Test the Connection: After setting up your `profiles.yml`, you should test the connection to ensure everything is configured correctly. First, switch to the directory where you extracted this package. Then run this command: ```bash dbt debug ``` This command will tell you if dbt can successfully connect to your PostgreSQL instance: ``` ... 09:37:00 retries: 1 09:37:00 Registered adapter: postgres=1.9.0 09:37:00 Connection test: [OK connection ok] 09:37:00 All checks passed! ``` ### 3. Login to CloudQuery To run a sync with CloudQuery, you will need to create an account and log in. ``` cloudquery login ``` ### 4. Sync the Cost and Usage Report Using the S3 Plugin and the Postgres Plugin, sync the exported cost and usage report file. You can use this example config yaml, make sure to fill the necessary values for `bucket` and `region`. For detailed authentication options see the [S3 plugin documentation](https://hub.cloudquery.io/plugins/source/cloudquery/s3/latest/docs). ```yml kind: source spec: name: s3-cur # The type of source, in this case, a s3-cur source. path: cloudquery/s3 # The plugin path for handling s3 sources. registry: cloudquery # The registry from which the plugin is sourced. version: "v1.0.1" # The version of the s3 plugin. tables: ["*"] # Specifies that all tables in the source should be considered. destinations: ["postgresql"] # The destination for the data, in this case, PostgreSQL. spec: bucket: "" # specify the name of the bucket with the exported reports region: "" # specify the bucket region # path_prefix: "" # Optional. Only sync files with this prefix # concurrency: 50 # Optional. Defines the number of files to sync in parallel. Defaults to 50 if not set. --- kind: destination spec: name: "postgresql" # The type of destination, in this case, PostgreSQL. path: "cloudquery/postgresql" # The plugin path for handling PostgreSQL as a destination. registry: "cloudquery" # The registry from which the PostgreSQL plugin is sourced. version: "v8.9.0"" # The version of the PostgreSQL plugin. spec: connection_string: "${POSTGRESQL_CONNECTION_STRING}" # set the environment variable in a format like # postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable # You can also specify the connection string in DSN format, which allows for special characters in the password: # connection_string: "user=postgres password=pass+0-[word host=localhost port=5432 dbname=postgres" ``` Run the sync (assuming the file above is saved to `s3_to_postgres.yaml`): ```shell cloudquery sync s3_to_postgres.yaml ``` ### 4. Sync AWS data Based on the models you are interested in running you need to sync the relevant tables. Below is an example sync for the relevant tables for all the models (views) in the policy. _Do note_ you will have to configure the cloudwatch spec to suit your data. For detailed AWS authentication and configuration options and additional tables to add to the sync config, see the [AWS plugin documentation](https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs). Save this to a file named `aws.yaml`: ```yml kind: source spec: name: aws # The source type, in this case, AWS. path: cloudquery/aws # The plugin path for handling AWS sources. registry: cloudquery # The registry from which the AWS plugin is sourced. version: "v32.38.0" # The version of the AWS plugin. tables: [ "aws_rds_clusters", "aws_ec2_instance_statuses", "aws_cloudwatch_metrics", "aws_cloudwatch_metric_statistics", "aws_ec2_instances", "aws_rds_instances", "aws_cloudhsmv2_backups", "aws_docdb_cluster_snapshots", "aws_dynamodb_backups", "aws_dynamodb_table_continuous_backups", "aws_ec2_ebs_snapshots", "aws_elasticache_snapshots", "aws_fsx_backups", "aws_fsx_snapshots", "aws_lightsail_database_snapshots", "aws_lightsail_disk_snapshots", "aws_lightsail_instance_snapshots", "aws_neptune_cluster_snapshots", "aws_rds_cluster_snapshots", "aws_rds_db_snapshots", "aws_redshift_snapshots", "aws_computeoptimizer_autoscaling_group_recommendations", "aws_autoscaling_groups", "aws_computeoptimizer_ebs_volume_recommendations", "aws_computeoptimizer_ec2_instance_recommendations", "aws_ec2_instances", "aws_computeoptimizer_ecs_service_recommendations", "aws_ecs_cluster_services", "aws_computeoptimizer_lambda_function_recommendations", "aws_lambda_functions", "aws_acm_certificates", "aws_backup_vaults", "aws_cloudfront_distributions", "aws_directconnect_connections", "aws_dynamodb_tables", "aws_ec2_ebs_volumes", "aws_ec2_eips", "aws_ec2_hosts", "aws_ec2_images", "aws_ec2_internet_gateways", "aws_ec2_network_acls", "aws_ec2_transit_gateways", "aws_ec2_transit_gateway_attachments", "aws_ecr_repositories", "aws_ecr_repository_images", "aws_efs_filesystems", "aws_lightsail_container_service_deployments", "aws_lightsail_container_services", "aws_lightsail_disks", "aws_lightsail_distributions", "aws_lightsail_load_balancers", "aws_lightsail_static_ips", "aws_elbv2_listeners", "aws_elbv2_target_groups", "aws_elbv2_load_balancers", "aws_route53_hosted_zones", "aws_sns_subscriptions", "aws_sns_topics", "aws_support_trusted_advisor_checks", "aws_support_trusted_advisor_check_results", ] destinations: ["postgresql"] # The destination for the data, in this case, PostgreSQL. use_paid_apis: true skip_dependent_tables: true spec: table_options: aws_cloudwatch_metrics: - list_metrics: namespace: AWS/RDS # Specifies the AWS service namespace for RDS metrics. get_metric_statistics: - period: 300 # The granularity, in seconds, of the returned data points. start_time: # The starting point for the data collection. example: 2024-01-01T00:00:01Z end_time: # The ending point for the data collection. example: 2024-01-30T23:59:59Z statistics: ["Average", "Maximum", "Minimum"] # The statistical values to retrieve. - list_metrics: namespace: AWS/EC2 # Specifies the AWS service namespace for EC2 metrics. get_metric_statistics: - period: 300 # The granularity, in seconds, of the returned data points. start_time: # The starting point for the data collection. example: 2024-01-01T00:00:01Z end_time: # The ending point for the data collection. example: 2024-01-30T23:59:59Z statistics: ["Average", "Maximum", "Minimum"] # The statistical values to retrieve. --- kind: destination spec: name: "postgresql" # The type of destination, in this case, PostgreSQL. path: "cloudquery/postgresql" # The plugin path for handling PostgreSQL as a destination. registry: "cloudquery" # The registry from which the PostgreSQL plugin is sourced. version: "v8.9.0" # The version of the PostgreSQL plugin. spec: connection_string: "${POSTGRESQL_CONNECTION_STRING}" # set the environment variable in a format like # postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable # You can also specify the connection string in DSN format, which allows for special characters in the password: # connection_string: "user=postgres password=pass+0-[word host=localhost port=5432 dbname=postgres" ``` Run the sync: ```shell cloudquery sync aws.yaml ``` ### 5. Create the views To run this policy you need to specify the name of the cost and usage report in your database, if you used the file plugin to load the report the table name will be the same as the file name (without the file extension). Navigate to your dbt project directory, where you extracted this package. Before executing the `dbt run` command, it might be useful to check for any potential issues (replace `` with the actual file name): ```bash dbt compile --vars '{"cost_usage_table": ""}' ``` If everything compiles without errors, you can then execute: ```bash dbt run --vars '{"cost_usage_table": ""}' ``` To run specific models (views) only: ```bash dbt run --vars '{"cost_usage_table": ""}' --select aws_cost__by_regions aws_cost__by_resources ``` ### 6. Explore the data Connect to your PostgreSQL database and start exploring the data. Below are a few examples of what you can do with the newly created views. #### Top 10 Most Cost Consuming Under-Utilized Resources To quickly identify which resources are both under-utilized and consuming a significant portion of your budget, use the following query: ```sql SELECT service, arn, cost FROM aws_cost__by_under_utilized_resources ORDER BY cost DESC LIMIT 10; ``` #### Most Cost Consuming Resource Types for Unused Resources To uncover resources that are no longer in use but still incurring costs, facilitating decisions on resource cleanup: ```sql SELECT resource_type, SUM(cost) AS total_cost FROM aws_cost__by_unused_resources GROUP BY resource_type ORDER BY total_cost DESC; ``` #### Cost Allocation by CloudFormation Tags For organizations utilizing AWS CloudFormation, understanding cost distribution across different stacks is crucial: ```sql SELECT cloudformation_stack_name, SUM(sum_line_item_unblended_cost) AS total_cost FROM aws_cost__cloudformation_tag_spend_allocation GROUP BY cloudformation_stack_name ORDER BY total_cost DESC; ``` #### ECS Service Cost Optimization Opportunities Identify ECS services that may benefit from optimization to reduce costs without compromising on performance: ```sql SELECT service_name, region, current_performance_risk, finding, recommended_performance_risk, cost FROM ecs_service_optimization_recommendations WHERE finding = 'Over-provisioned' ORDER BY cost DESC; ``` #### Lambda Functions with Optimization Potential Spot Lambda functions that might be running with more memory than required, indicating an opportunity for cost savings: ```sql SELECT function_name, region, current_memory_size, recommend_memory_size, number_of_invocations, cost FROM lambda_function_optimization_recommendations WHERE current_memory_size > recommend_memory_size ORDER BY cost DESC; ``` #### Insights into Trusted Advisor Cost Optimization Recommendations Gain insights from AWS Trusted Advisor's cost optimization checks to further reduce expenses: ```sql SELECT name, description, category, arn, account_id FROM aws_cost__trusted_advisor_by_arn ORDER BY account_id, arn; ``` ### 7. Production deployment This transformation creates a view on top of the database tables synced by CloudQuery CLI. You need to re-run the transformation (using the `dbt run` command) only if you add or remove tables from the sync. ## Data Dictionary In this section you can see all the models (views) that are included in the policy with an explanation about the data inside and the columns available. cost will be in the same units as it is in the CUR file which are USD ($). line_item_resource_id is usually the resource ARN except in certain cases where it is a volume_id or instance_id of certain services. By default the models related to tags are disabled (Tags are only available in the CUR if they are activated), to enable this model change them to enabled in the models section in `dbt_project.yml` #### `aws_cost__by_under_utilized_resources` Identifies resources that are under-utilized based on specific metrics (e.g., CPUUtilization, storage usage), highlighting opportunities for cost optimization. Supported services include EC2 Instances, RDS Clusters, and DynamoDB. - `arn` - The resource identifier. - `service` - The AWS service (e.g., EC2, RDS). - `instance_type` - The type of instance (e.g., t3.mini). - `metric` - The metric indicating under-utilization. - `value` - The value for the metric (Usually percentage %). - `cost` - The cost of the resource. #### `aws_cost__of_unused_resources` Identifies resources and are completely unused (not metric based), highlighting 'To Be Deleted' resources. Supported services are acm certs, backup vaults, cloudfront distributions, directconnect connections, dynamodb tables, ec2 ebs volumes, ec2 eips, ec2 internet gateways, ec2 hosts, ec2 images, ec2 network acls, ec2 transit gateways, ecr repositories, efs filesystems, lightsail container services, lightsail disks, lightsail distributions, lightsail load balancers, lightsail static ips, load balancers, route53 hosted zones, sns topics. - `account_id` - the account that owns the resource. - `resource_id` - the arn of the resource - `cost` - the total cost of the resource - `resource_type` - the type of resource (i.e lightsail load balancer) #### `aws_cost__cloudformation_tag_spend_allocation` Provides a breakdown of costs by CloudFormation tags, helping in cost allocation and chargeback calculations. - `cloudformation_logical_id` - Identifier for the CloudFormation resource. - `cloudformation_stack_name` - Name of the CloudFormation stack. - `month` - The billing period month. - `sum_line_item_unblended_cost` - Raw cost per CloudFormation tag group. - `sum_spend` - Total spend for all resources. - `untagged_spend` - Spend allocated to untagged resources. - `tagged_spend` - Spend allocated to tagged resources. - `percent_spend` - Percentage of spend per tag group. - `untagged_cost_distribution` - Percentage of spend for untagged resources. - `chargeback` - Chargeback amount for tagged resources. #### `aws_cost__by_cloudformation_tag` Focuses on cost allocation for resources tagged with CloudFormation tags. - `cloudformation_logical_id` - Identifier for the CloudFormation resource. - `cloudformation_stack_name` - Name of the CloudFormation stack. - `month` - The billing period month. - `sum_line_item_unblended_cost` - Raw cost per CloudFormation tag group. #### `aws_cost__beanstalk_tag_spend_allocation` Analyzes costs associated with Elastic Beanstalk environments, aiding in understanding spend distribution across tagged and untagged resources. - `elasticbeanstalk_environment_id` - Identifier for the Elastic Beanstalk environment. - `month` - The billing period month. - `sum_line_item_unblended_cost` - Raw cost per Elastic Beanstalk tag group. - `sum_spend` - Total spend for all resources. - `untagged_spend` - Spend allocated to untagged resources. - `tagged_spend` - Spend allocated to tagged resources. - `percent_spend` - Percentage of spend per tag group. - `untagged_cost_distribution` - Percentage of spend for untagged resources. - `chargeback` - Chargeback amount for tagged resources. #### `aws_cost__by_beanstalk_tag` Details cost allocation for Elastic Beanstalk tagged resources. - `elasticbeanstalk_environment_id` - Identifier for the Elastic Beanstalk environment. - `month` - The billing period month. - `sum_line_item_unblended_cost` - Raw cost per Elastic Beanstalk tag. #### `aws_cost__ecs_tag_spend_allocation` Offers insights into costs by ECS cluster tags, facilitating detailed spend analysis and chargeback for ECS resources. - `ecs_cluster` - Identifier for the ECS cluster. - `month` - The billing period month. - `sum_line_item_unblended_cost` - Raw cost per ECS cluster tag group. - `sum_spend` - Total spend for all resources. - `untagged_spend` - Spend allocated to untagged resources. - `tagged_spend` - Spend allocated to tagged resources. - `percent_spend` - Percentage of spend per tag group. - `untagged_cost_distribution` - Percentage of spend for untagged resources. - `chargeback` - Chargeback amount for tagged resources. #### `aws_cost__by_ecs_tag` Concentrates on cost breakdown for resources tagged within ECS clusters. - `ecs_cluster` - Identifier for the ECS cluster. - `month` - The billing period month. - `sum_line_item_unblended_cost` - Raw cost per ECS tag. #### `aws_cost__lambda_tag_spend_allocation` Breaks down Lambda function costs by tags, aiding in the management and allocation of Lambda-related expenses. - `lambda_function_name` - Identifier for the Lambda function. - `month` - The billing period month. - `sum_line_item_unblended_cost` - Raw cost per Lambda tag groups. - `sum_spend` - Total spend for all resources. - `untagged_spend` - Spend allocated to untagged resources. - `tagged_spend` - Spend allocated to tagged resources. - `percent_spend` - Percentage of spend per tag group. - `untagged_cost_distribution` - Percentage of spend for untagged resources. - `chargeback` - Chargeback amount for tagged resources. #### `aws_cost__by_lambda_tag` Details cost allocation for Lambda functions based on tagging. - `lambda_function_name` - Identifier for the Lambda function. - `month` - The billing period month. - `sum_line_item_unblended_cost` - Raw cost per Lambda tag. #### `aws_cost__by_tag` General view for analyzing costs by AWS tags, facilitating broad cost management across different AWS resources. - `aws_tag_name` - The name of the AWS tag. - `aws_tag_value` - The value of the AWS tag. - `month` - The billing period month. - `sum_line_item_unblended_cost` - Sum of the raw cost per tag. #### `aws_cost__by_untagged_resource` Highlights costs associated with untagged resources, offering insights into potential areas for cost optimization through better resource tagging. - `account_id` - AWS account ID. - `resource_id` - Identifier for the resource. - `service` - The AWS service name. - `month` - The billing period month. - `sum_line_item_unblended_cost` - Sum of the raw cost per untagged resource. #### `aws_cost__by_recovery_resources` Provides insights into the cost associated with recovery resources. - `account_id` - The account ID that owns the resource. - `resource_id` - The identifier of the resource. - `cost` - The cost associated with the resource. - `resource_type` - The type of the resource, e.g., `rds_db_snapshots`. #### `autoscaling_group_optimization_recommendations` Offers recommendations for optimizing Auto Scaling Groups. - `account_id` - The account ID that owns the Auto Scaling Group resource. - `auto_scaling_group_arn` - The Amazon Resource Name (ARN) of the Auto Scaling Group. - `auto_scaling_group_name` - The name of the Auto Scaling Group. - `region` - The region where the Auto Scaling Group is located. - `status` - The status of the Auto Scaling Group. - `current_performance_risk` - The current performance risk of the Auto Scaling Group. - `finding` - The finding classification of the Auto Scaling Group. - `inferred_workload_types` - The inferred workload types running on the Auto Scaling Group. - `current_desired_capacity` - The current desired capacity configuration of the Auto Scaling Group. - `recommended_desired_capacity` - The recommended desired capacity configuration of the Auto Scaling Group. - `current_instance_type` - The current instance type configuration of the Auto Scaling Group. - `recommended_instance_type` - The recommended instance type configuration of the Auto Scaling Group. - `current_max_size` - The current maximum size configuration of the Auto Scaling Group. - `recommended_max_size` - The recommended maximum size configuration of the Auto Scaling Group. - `current_min_size` - The current minimum size configuration of the Auto Scaling Group. - `recommended_min_size` - The recommended minimum size configuration of the Auto Scaling Group. - `current_gpus` - The number of GPUs (if any) in the current configuration of the Auto Scaling Group. - `recommended_gpus` - The number of GPUs (if any) in the recommended configuration of the Auto Scaling Group. - `migration_effort` - The level of effort required for migrating to the recommended configuration. - `recommended_performance_risk` - The performance risk of the recommended configuration of the Auto Scaling Group. - `cost` - The cost associated with the Auto Scaling Group. #### `ebs_volume_optimization_recommendations` Analyzes EBS volumes to provide recommendations for optimizing performance and cost. - `account_id` - The account ID that owns the EBS volume resource. - `volume_arn` - The Amazon Resource Name (ARN) of the EBS volume. - `ecs_attached` - Indicates if the EBS volume is attached to an ECS container instance. - `encrypted` - Indicates if the EBS volume is encrypted. - `fast_restored` - Indicates if the EBS volume was fast restored. - `availability_zone` - The availability zone where the EBS volume is located. - `region` - The region where the EBS volume is located. - `state` - The state of the EBS volume. - `snapshot_id` - The ID of the snapshot associated with the EBS volume. - `sse_type` - The type of server-side encryption (SSE) used for the EBS volume. - `current_performance_risk` - The current performance risk of the EBS volume. - `finding` - The finding classification of the EBS volume. - `current_baseline_iops` - The current baseline IOPS configuration of the EBS volume. - `recommended_baseline_iops` - The recommended baseline IOPS configuration of the EBS volume. - `current_baseline_throughput` - The current baseline throughput configuration of the EBS volume. - `recommended_baseline_throughput` - The recommended baseline throughput configuration of the EBS volume. - `current_burst_iops` - The current burst IOPS configuration of the EBS volume. - `recommended_burst_iops` - The recommended burst IOPS configuration of the EBS volume. - `current_burst_throughput` - The current burst throughput configuration of the EBS volume. - `recommended_burst_throughput` - The recommended burst throughput configuration of the EBS volume. - `current_volume_size` - The current size of the EBS volume. - `recommended_volume_size` - The recommended size of the EBS volume. - `current_volume_type` - The current volume type of the EBS volume. - `recommended_volume_type` - The recommended volume type of the EBS volume. - `recommended_performance_risk` - The performance risk of the recommended configuration of the EBS volume. - `cost` - The cost associated with the EBS volume. #### `ec2_instances_optimization_recommendations` Provides optimization recommendations for EC2 instances. - `account_id` - The account ID that owns the EC2 instance resource. - `instance_arn` - The Amazon Resource Name (ARN) of the EC2 instance. - `region` - The region where the EC2 instance is located. - `state` - The state of the EC2 instance. - `tags` - The tags associated with the EC2 instance. - `instance_type` - The instance type of the EC2 instance. - `architecture` - The architecture of the EC2 instance. - `ami_launch_index` - The AMI launch index of the EC2 instance. - `private_ip_address` - The private IP address of the EC2 instance. - `ipv6_address` - The IPv6 address of the EC2 instance. - `current_performance_risk` - The current performance risk of the EC2 instance. - `current_gpus` - The number of GPUs of the current EC2 instance. - `recommended_gpus` - The number of GPUs recommended for the EC2 instance. - `current_instance_type` - The current instance type of the EC2 instance. - `recommended_instance_type` - The recommended instance type for the EC2 instance. - `migration_effort` - The level of effort required for migrating to the recommended instance type. - `recommended_performance_risk` - The performance risk of the recommended instance type. - `platform_differences` - The platform differences between the current and recommended instance types. - `cost` - The cost associated with the EC2 instance. #### `ecs_service_optimization_recommendations` Targets ECS services for optimization. - `account_id` - The account ID that owns the ECS service. - `service_arn` - The Amazon Resource Name (ARN) of the ECS service. - `service_name` - The name of the ECS service. - `region` - The region where the ECS service is located. - `current_performance_risk` - The current performance risk of the ECS service. - `finding` - The finding classification of the ECS service. - `finding_reason_codes` - The reason codes for the finding classification of the ECS service. - `container_cpu_recommendation` - The CPU recommendation for the container in the ECS service. - `container_memory_size_recommendation` - The memory size recommendation for the container in the ECS service. - `status` - The status of the ECS service. - `tags` - The tags associated with the ECS service. - `cluster_arn` - The ARN of the ECS cluster to which the service belongs. - `desired_count` - The desired count of tasks for the ECS service. - `launch_type` - The launch type of the ECS service. - `load_balancers` - The load balancers associated with the ECS service. - `platform_family` - The platform family of the ECS service. - `platform_version` - The platform version of the ECS service. - `running_count` - The number of tasks running for the ECS service. - `cost` - The cost associated with the ECS service. #### `lambda_function_optimization_recommendations` Provides optimization recommendations for Lambda functions. - `account_id` - The account ID that owns the Lambda function. - `function_arn` - The Amazon Resource Name (ARN) of the Lambda function. - `function_name` - The name of the Lambda function. - `region` - The region where the Lambda function is located. - `tags` - The tags associated with the Lambda function. - `function_version` - The version of the Lambda function. - `code_size` - The size of the Lambda function's code. - `lookback_period_in_days` - The number of days for which utilization metrics were analyzed for the Lambda function. - `number_of_invocations` - The number of invocations of the Lambda function. - `current_performance_risk` - The current performance risk of the Lambda function. - `finding` - The finding classification of the Lambda function. - `finding_reason_codes` - The reason codes for the finding classification of the Lambda function. - `current_memory_size` - The current memory size configuration of the Lambda function. - `recommend_memory_size` - The recommended memory size for the Lambda function. #### `aws_cost__trusted_advisor_by_arn` This view aggregates data from AWS Trusted Advisor checks specifically related to cost optimization. It joins information about the checks with the resources they flag, providing a focused view on areas where cost efficiency can be improved. - `account_id` - The AWS account ID that owns the resource. This field helps identify which account a particular piece of advice applies to, making it easier for organizations with multiple accounts to allocate advice to the correct account. - `arn` - The Amazon Resource Name (ARN) of the resource. ARNs are unique identifiers for AWS resources and here they specify exactly which resource the Trusted Advisor check is referring to. - `category` - The category of the Trusted Advisor check. In this view, it will always be 'cost_optimizing', indicating that the advice is aimed at improving cost efficiency. - `id` - The ID of the Trusted Advisor check. This is a unique identifier for each type of check that Trusted Advisor performs, allowing users to reference the specific advice or test being applied. - `name` - The name of the Trusted Advisor check. This provides a human-readable description of what the check is about, such as "Low Utilization Amazon EC2 Instances". - `description` - The description of the Trusted Advisor check. This field offers more detailed information about what the check entails and possibly how to address the advice given. ### `aws_cost__anomaly_per_service` Identifies resources within an AWS service that have costs considered statistically anomalous compared to other resources in the same service. - `line_item_product_code` - The AWS service. - `line_item_resource_id` - The resource ARN. - `cost` - The total cost of the resource. - `mean_cost` - The average cost for the service (`product_code`). - `std_cost` - The standard deviation of the cost for the service (`product_code`). ### `aws_cost__by_account` Aggregates costs by account. - `line_item_usage_account_id` - The account that incurred the cost. - `cost` - The total cost for the account. ### `aws_cost__by_region` Aggregates costs by region. - `product_location` - The region where the resource is located. - `cost` - The total cost for the region. ### `aws_cost__by_resource` Aggregates costs by resource. - `line_item_resource_id` - The resource ARN. - `line_item_product_code` - The AWS service. - `cost` - The total cost of the resource. ### `aws_cost__by_product` Aggregates costs by AWS Product. - `line_item_product_code` - The AWS product (i.e EC2, RDS). - `cost` - The total cost for the region. ### `aws_cost__gp2_ebs_volumes` Details the cost of GP2 EBS volumes. - `line_item_resource_id` - The resource ID. - `cost` - The total cost of the resource. - `volume_type` - The volume type. - `attachments` - The number of attachments. - `arn` - The resource ARN. - `tags` - The tags associated with the volume. - `state` - The state of the volume. - `snapshot_id` - The snapshot ID. - `size` - The volume size. - `create_time` - The volume creation time. ### `aws_cost__over_time` Aggregates cost by time period. - `line_item_usage_start_date` - The start date of the billing period. - `line_item_usage_end_date` - The end date of the billing period. - `cost` - The total cost in the time period. ## Required Tables By Model (View) ### `aws_cost__by_under_utilized_resources` - `cost table` - `aws_cloudwatch_metrics` - `aws_cloudwatch_metric_statistics` - `aws_ec2_instances` - `aws_rds_instances` ### `aws_cost__by_recovery_resources` - `cost table` - `aws_cloudhsmv2_backups` - `aws_docdb_cluster_snapshots` - `aws_dynamodb_backups` - `aws_dynamodb_table_continuous_backups` - `aws_ec2_ebs_snapshots` - `aws_elasticache_snapshots` - `aws_fsx_backups` - `aws_fsx_snapshots` - `aws_lightsail_database_snapshots` - `aws_lightsail_disk_snapshots` - `aws_lightsail_instance_snapshots` - `aws_neptune_cluster_snapshots` - `aws_rds_cluster_snapshots` - `aws_rds_db_snapshots` - `aws_redshift_snapshots` ### `compute_optimizer` - **Autoscaling Group** - `aws_computeoptimizer_autoscaling_group_recommendations` - `aws_autoscaling_groups` - **EC2 EBS Volumes** - `aws_computeoptimizer_ebs_volume_recommendations` - `aws_ec2_ebs_volumes` - **EC2 Instances** - `aws_computeoptimizer_ec2_instance_recommendations` - `aws_ec2_instances` - **ECS Cluster Service** - `aws_computeoptimizer_ecs_service_recommendations` - `aws_ecs_cluster_services` - **Lambda Function** - `aws_computeoptimizer_lambda_function_recommendations` - `aws_lambda_functions` ### `aws_cost__by_unused_resources` - `cost table` - `aws_cost__by_resource` - `aws_acm_certificates` - `aws_backup_vaults` - `aws_cloudfront_distributions` - `aws_directconnect_connections` - `aws_dynamodb_tables` - `aws_ec2_ebs_volumes` - `aws_ec2_eips` - `aws_ec2_hosts` - `aws_ec2_images` - `aws_ec2_internet_gateways` - `aws_ec2_network_acls` - `aws_ec2_transit_gateways` - `aws_ec2_transit_gateway_attachments` - `aws_ecr_repositories` - `aws_ecr_repository_images` - `aws_efs_filesystems` - `aws_lightsail_container_service_deployments` - `aws_lightsail_container_services` - `aws_lightsail_disks` - `aws_lightsail_distributions` - `aws_lightsail_load_balancers` - `aws_lightsail_static_ips` - `aws_elbv2_listeners` - `aws_elbv2_target_groups` - `aws_elbv2_load_balancers` - `aws_route53_hosted_zones` - `aws_sns_subscriptions` - `aws_sns_topics` ### `aws_cost__cloudformation_tag_spend_allocation` - `cost table` ### `aws_cost__by_cloudformation_tag` - `cost table` ### `aws_cost__beanstalk_tag_spend_allocation` - `cost table` ### `aws_cost__by_beanstalk_tag` - `cost table` ### `aws_cost__ecs_tag_spend_allocation` - `cost table` ### `aws_cost__by_ecs_tag` - `cost table` ### `aws_cost__lambda_tag_spend_allocation` - `cost table` ### `aws_cost__by_lambda_tag` - `cost table` ### `aws_cost__by_tag` - `cost table` ### `aws_cost__by_untagged_resource` - `cost table` ### `autoscaling_group_optimization_recommendations` - `cost table` - `aws_computeoptimizer_autoscaling_group_recommendations` - `aws_autoscaling_groups` ### `ebs_volume_optimization_recommendations` - `cost table` - `aws_computeoptimizer_ebs_volume_recommendations` - `aws_ec2_ebs_volumes` ### `ec2_instances_optimization_recommendations` - `cost table` - `aws_computeoptimizer_ec2_instance_recommendations` - `aws_ec2_instances` ### `ecs_service_optimization_recommendations` - `cost table` - `aws_computeoptimizer_ecs_service_recommendations` - `aws_ecs_cluster_services` ### `lambda_function_optimization_recommendations` - `cost table` - `aws_computeoptimizer_lambda_function_recommendations` - `aws_lambda_functions` ### `aws_cost__trusted_advisor_by_arn` - `aws_support_trusted_advisor_checks` - `aws_support_trusted_advisor_check_results`- ` ### `aws_cost__over_time` - `cost table` ### `aws_cost__by_resource` - `cost table` ### `aws_cost__by_region` - `cost table` ### `aws_cost__by_account` - `cost table` ### `aws_cost__by_product` - `cost table` ### `aws_cost__anomaly_per_service` - `cost table` ### `aws_cost__gcp2_ebs_volumes` - `cost table` - `aws_ec2_ebs_volumes` --- Source: https://www.cloudquery.io/hub/addons/transformation/cloudquery/aws-data-resilience/latest/docs # CloudQuery + dbt AWS Data Resilience (AWS Backup) ## Overview This AWS Data Resilience package works on top of the CloudQuery framework. This package offers automated insight into your AWS Backup posture in your AWS environment. This package supports PostgreSQL database only. We recommend using this transformation with our [AWS Data Resilience Dashboard](https://hub.cloudquery.io/addons/visualization/cloudquery/aws-data-resilience/latest/docs) ![AWS Data Resilience Dashboard](https://storage.googleapis.com/cq-cloud-images/cloudquery/92edc525ec9ab41558e4fe9c8135978213e51f3a_aws_resiliency_dash.png) ### Requirements - [CloudQuery](https://docs.cloudquery.io/docs/quickstart/) - [CloudQuery AWS plugin](https://hub.cloudquery.io/plugins/source/cloudquery/aws) - [A CloudQuery Account](https://www.cloudquery.io/auth/register) - [dbt](https://docs.getdbt.com/docs/core/pip-install) - [PostgreSQL](https://hub.cloudquery.io/plugins/destination/cloudquery/postgresql) ### Models Included - **aws_data_resilience\_\_overview**: AWS Backup overview for `aws_dynamodb_tables`, `aws_ec2_instances` and `aws_s3_buckets`, available for PostgreSQL. #### Columns Included - `account_id` - `resource_arn` - `tags` - `last_backup_time` - `resource_type` ## To run this package you need to complete the following steps ### 1. Download and extract this package. Click the **Download now** button on the top. You may need to create a CloudQuery account first. ### 2. Install DBT and set up the DBT profile [Install `dbt`](https://docs.getdbt.com/docs/core/pip-install): ```bash pip install dbt-postgres ``` Create the profile directory: ```bash mkdir -p ~/.dbt ``` Create a `profiles.yml` file in your profile directory (e.g. `~/.dbt/profiles.yml`): ```yaml aws_data_resilience: # This should match the name in your dbt_project.yml target: dev outputs: dev: type: postgres host: 127.0.0.1 user: postgres pass: pass port: 5432 dbname: postgres schema: public # default schema where dbt will build the models threads: 1 # number of threads to use when running in parallel ``` Test the Connection: After setting up your `profiles.yml`, you should test the connection to ensure everything is configured correctly. First, switch to the directory where you extracted this package. Then run this command: ```bash dbt debug ``` This command will tell you if dbt can successfully connect to your PostgreSQL instance: ``` ... 09:37:00 retries: 1 09:37:00 Registered adapter: postgres=1.9.0 09:37:00 Connection test: [OK connection ok] 09:37:00 All checks passed! ``` ### 3. Login to CloudQuery To run a sync with CloudQuery, you will need to create an account and log in. ``` cloudquery login ``` ### 4. Sync AWS data This is an example sync config with the minimum set of tables for this transformation. Save this to a file named `aws.yaml`. For detailed AWS authentication and configuration options and additional tables to add to the sync config, see the [AWS plugin documentation](https://hub.cloudquery.io/plugins/source/cloudquery/aws/latest/docs). ```yml kind: source spec: name: aws # The source type, in this case, AWS. path: cloudquery/aws # The plugin path for handling AWS sources. registry: cloudquery # The registry from which the AWS plugin is sourced. version: "v32.38.0" # The version of the AWS plugin. tables: ["aws_dynamodb_tables", "aws_ec2_instances", "aws_s3_buckets"] # Include any tables that meet your requirements, separated by commas destinations: ["postgresql"] # The destination for the data, in this case, PostgreSQL. spec: --- kind: destination spec: name: "postgresql" # The type of destination, in this case, PostgreSQL. path: "cloudquery/postgresql" # The plugin path for handling PostgreSQL as a destination. registry: "cloudquery" # The registry from which the PostgreSQL plugin is sourced. version: "v8.9.0" # The version of the PostgreSQL plugin. spec: connection_string: "${POSTGRESQL_CONNECTION_STRING}" # set the environment variable in a format like # postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable # You can also specify the connection string in DSN format, which allows for special characters in the password: # connection_string: "user=postgres password=pass+0-[word host=localhost port=5432 dbname=postgres" ``` Run the sync: ```shell cloudquery sync aws.yaml ``` ### 5. Create the views Navigate to your dbt project directory, where you extracted this package. Before executing the `dbt run` command, it might be useful to check for any potential issues: ```bash dbt compile ``` If everything compiles without errors, you can then execute: ```bash dbt run ``` This command will run your `dbt` models and create tables/views in your destination database as defined in your models. **Note:** If running locally, ensure you are using `dbt-core` and not `dbt-cloud-cli` as dbt-core does not require extra authentication. ### 6. Explore the data Connect to your PostgreSQL database and start exploring the data ```sql select * from aws_data_resilience__overview limit 10 ``` ### 7. Set up Grafana dashboard (Optional) Follow the instructions in [AWS Data Resilience Dashboard](https://hub.cloudquery.io/addons/visualization/cloudquery/aws-data-resilience/latest/docs) to set up a dashboard in Grafana. Note the visualization supports PostgreSQL database only. ### 8. Production deployment This transformation creates a view on top of the database tables synced by CloudQuery CLI. You need to re-run the transformation (using the `dbt run` command) only if you add or remove tables from the sync. --- Source: https://www.cloudquery.io/hub/addons/transformation/cloudquery/azure-asset-inventory/latest/docs # CloudQuery × dbt: Azure Asset Inventory Package ## Overview This Azure Asset Inventory package works on top of the CloudQuery framework. This package offers automated line-item listing of all active resources in your Azure environment. This package supports usage with PostgreSQL, Snowflake, and BigQuery databases. ### Example Queries How many resources are there per subscription? (PostgreSQL) ```sql select subscription_id, count(*) from azure_resources group by subscription_id order by count(*) desc ``` Resources by id and location (PostgreSQL) ```sql select id, location, count(*) from azure_resources group by id, location order by count(*) desc ``` ### Requirements - [A CloudQuery Account](https://www.cloudquery.io/auth/register) - [CloudQuery](https://cloud.cloudquery.io/getting-started/) - [CloudQuery Azure plugin](https://hub.cloudquery.io/plugins/source/cloudquery/azure) - [dbt](https://docs.getdbt.com/docs/core/pip-install) One of the below destination plugins based on your database: - [PostgreSQL](https://hub.cloudquery.io/plugins/destination/cloudquery/postgresql) - [Snowflake](https://hub.cloudquery.io/plugins/destination/cloudquery/snowflake) - [BigQuery](https://hub.cloudquery.io/plugins/destination/cloudquery/bigquery) #### Models Included - **azure_resources**: Azure Resources View, available for PostgreSQL. - Required tables: This model has no specific table dependencies, other than requiring a single CloudQuery table from the Azure plugin that has a subscription id. #### Columns Included - `_cq_id` - `_cq_source_name` - `_cq_sync_time` - `subscription_id` - `id` - `location` - `name` - `kind` - `_cq_table` ## To run this package you need to complete the following steps ### 1. Download and extract this package. Click the **Download now** button on the top. You may need to create a CloudQuery account first. ### 2. Install DBT and set up the DBT profile [Install `dbt`](https://docs.getdbt.com/docs/core/pip-install): ```bash pip install dbt-postgres ``` Create the profile directory: ```bash mkdir -p ~/.dbt ``` Create a `profiles.yml` file in your profile directory (e.g. `~/.dbt/profiles.yml`): ```yaml azure_asset_inventory: # This should match the name in your dbt_project.yml target: dev outputs: dev: type: postgres host: 127.0.0.1 user: postgres pass: pass port: 5432 dbname: postgres schema: public # default schema where dbt will build the models threads: 1 # number of threads to use when running in parallel ``` Test the Connection: After setting up your `profiles.yml`, you should test the connection to ensure everything is configured correctly. First, switch to the directory where you extracted this package. Then run this command: ```bash dbt debug ``` This command will tell you if dbt can successfully connect to your PostgreSQL instance: ``` ... 09:37:00 retries: 1 09:37:00 Registered adapter: postgres=1.9.0 09:37:00 Connection test: [OK connection ok] 09:37:00 All checks passed! ``` ### 3. Login to CloudQuery To run a sync with CloudQuery, you will need to create an account and log in. ``` cloudquery login ``` ### 4. Syncing Azure data This is an example sync config for the relevant tables for all the models (views) in this transformation. Save this to a file named `azure.yaml`. For detailed Azure authentication and configuration options and additional tables to add to the sync config, see the [Azure plugin documentation](https://hub.cloudquery.io/plugins/source/cloudquery/azure/latest/docs). ```yml kind: source spec: name: "azure" # The source type, in this case, Azure. path: "cloudquery/azure" # The plugin path for handling Azure sources. registry: "cloudquery" # The registry from which the Azure plugin is sourced. version: "v17.11.0" # The version of the Azure plugin. destinations: ["postgresql"] # The destination for the data, in this case, PostgreSQL. tables: ["azure_compute_virtual_machines"] # Include any tables that meet your requirements, separated by commas spec: --- kind: destination spec: name: "postgresql" # The type of destination, in this case, PostgreSQL. path: "cloudquery/postgresql" # The plugin path for handling PostgreSQL as a destination. registry: "cloudquery" # The registry from which the PostgreSQL plugin is sourced. version: "v8.9.0" # The version of the PostgreSQL plugin. spec: connection_string: "${POSTGRESQL_CONNECTION_STRING}" # set the environment variable in a format like # postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable # You can also specify the connection string in DSN format, which allows for special characters in the password: # connection_string: "user=postgres password=pass+0-[word host=localhost port=5432 dbname=postgres" ``` Run the sync: ```shell cloudquery sync azure.yaml ``` ### 5. Create the views Navigate to your dbt project directory, where you extracted this package. Before executing the `dbt run` command, it might be useful to check for any potential issues: ```bash dbt compile ``` If everything compiles without errors, you can then execute: ```bash dbt run ``` This command will run your `dbt` models and create tables/views in your destination database as defined in your models. **Note:** If running locally, ensure you are using `dbt-core` and not `dbt-cloud-cli` as dbt-core does not require extra authentication. ### 6. Explore the data Connect to your database and start exploring the data ```sql select * from azure_resources limit 10 ``` ### 7. Production deployment This transformation creates a view on top of the database tables synced by CloudQuery CLI. You need to re-run the transformation (using the `dbt run` command) only if you add or remove tables from the sync. --- Source: https://www.cloudquery.io/hub/addons/transformation/cloudquery/gcp-asset-inventory/latest/docs # CloudQuery × dbt: GCP Asset Inventory Package ## Overview This GCP Asset Inventory package works on top of the CloudQuery framework. This package offers automated line-item listing of all active resources in your GCP environment. This package supports usage with PostgreSQL, BigQuery, and Snowflake databases. ### Example Queries How many resources are there per project? (PostgreSQL) ```sql select project_id, count(*) from gcp_resources group by project_id order by count(*) desc ``` How many resources by project and region? (PostgreSQL) ```sql select project_id, region, count(*) from gcp_resources group by project_id, region order by count(*) desc ``` ### Requirements - [A CloudQuery Account](https://www.cloudquery.io/auth/register) - [CloudQuery](https://cloud.cloudquery.io/getting-started/) - [CloudQuery GCP plugin](https://hub.cloudquery.io/plugins/source/cloudquery/gcp) - [dbt](https://docs.getdbt.com/docs/core/pip-install) One of the below destination plugins based on your database: - [PostgreSQL](https://hub.cloudquery.io/plugins/destination/cloudquery/postgresql) - [Snowflake](https://hub.cloudquery.io/plugins/destination/cloudquery/snowflake) - [BigQuery](https://hub.cloudquery.io/plugins/destination/cloudquery/bigquery) ### Models Included - **gcp_resources**: GCP Resources View, available for PostgreSQL. - Required tables: This model has no specific table dependencies, other than requiring a single CloudQuery table from the GCP plugin that has a project id. #### Columns Included - `_cq_id` - `_cq_source_name` - `_cq_sync_time` - `project_id` - `id` - `region` - `name` - `description` - `_cq_table` ## To run this package you need to complete the following steps ### 1. Download and extract this package. Click the **Download now** button on the top. You may need to create a CloudQuery account first. ### 2. Install DBT and set up the DBT profile [Install `dbt`](https://docs.getdbt.com/docs/core/pip-install): ```bash pip install dbt-postgres ``` Create the profile directory: ```bash mkdir -p ~/.dbt ``` Create a `profiles.yml` file in your profile directory (e.g. `~/.dbt/profiles.yml`): ```yaml gcp_asset_inventory: # This should match the name in your dbt_project.yml target: dev outputs: dev: type: postgres host: 127.0.0.1 user: postgres pass: pass port: 5432 dbname: postgres schema: public # default schema where dbt will build the models threads: 1 # number of threads to use when running in parallel ``` Test the Connection: After setting up your `profiles.yml`, you should test the connection to ensure everything is configured correctly. First, switch to the directory where you extracted this package. Then run this command: ```bash dbt debug ``` This command will tell you if dbt can successfully connect to your PostgreSQL instance: ``` ... 09:37:00 retries: 1 09:37:00 Registered adapter: postgres=1.9.0 09:37:00 Connection test: [OK connection ok] 09:37:00 All checks passed! ``` ### 3. Login to CloudQuery To run a sync with CloudQuery, you will need to create an account and log in. ``` cloudquery login ``` ### 4. Sync GCP data This is an example sync config for the relevant tables for all the models (views) in this transformation. Save this to a file named `gcp.yaml`. For detailed GCP authentication and configuration options and additional tables to add to the sync config, see the [GCP plugin documentation](https://hub.cloudquery.io/plugins/source/cloudquery/gcp/latest/docs). ```yml kind: source spec: name: "gcp" # The source type, in this case, GCP. path: "cloudquery/gcp" # The plugin path for handling GCP sources. registry: "cloudquery" # The registry from which the GCP plugin is sourced. version: "v19.1.0" # The version of the GCP plugin. tables: ["gcp_storage_buckets"] # Include any tables that meet your requirements, separated by commas destinations: ["postgresql"] # The destination for the data, in this case, PostgreSQL. spec: # GCP Spec project_ids: ["my-project"] # The name of the GCP project you are working in --- kind: destination spec: name: "postgresql" # The type of destination, in this case, PostgreSQL. path: "cloudquery/postgresql" # The plugin path for handling PostgreSQL as a destination. registry: "cloudquery" # The registry from which the PostgreSQL plugin is sourced. version: "v8.9.0" # The version of the PostgreSQL plugin. spec: connection_string: "${POSTGRESQL_CONNECTION_STRING}" # set the environment variable in a format like # postgresql://postgres:pass@localhost:5432/postgres?sslmode=disable # You can also specify the connection string in DSN format, which allows for special characters in the password: # connection_string: "user=postgres password=pass+0-[word host=localhost port=5432 dbname=postgres" ``` Run the sync: ```shell cloudquery sync gcp.yaml ``` ### 5. Create the views Navigate to your dbt project directory, where you extracted this package. Before executing the `dbt run` command, it might be useful to check for any potential issues: ```bash dbt compile ``` If everything compiles without errors, you can then execute: ```bash dbt run ``` This command will run your `dbt` models and create tables/views in your destination database as defined in your models. **Note:** If running locally, ensure you are using `dbt-core` and not `dbt-cloud-cli` as dbt-core does not require extra authentication. ### 6. Explore the data Connect to your database and start exploring the data ```sql select * from gcp_resources limit 10 ``` ### 7. Production deployment This transformation creates a view on top of the database tables synced by CloudQuery CLI. You need to re-run the transformation (using the `dbt run` command) only if you add or remove tables from the sync. --- Source: https://www.cloudquery.io/hub/addons/visualization/cloudquery/aws-asset-inventory/latest/docs # CloudQuery AWS Asset Inventory Dashboard for Grafana ## Overview This contains an AWS Asset Inventory Dashboard for Grafana on top of CloudQuery [AWS plugin](https://hub.cloudquery.io/plugins/source/cloudquery/aws) and [AWS Asset Inventory pack](https://hub.cloudquery.io/addons/transformation/cloudquery/aws-asset-inventory/). ## Requirements - [CloudQuery](https://docs.cloudquery.io/docs/quickstart/) - [AWS Plugin](https://hub.cloudquery.io/plugins/source/cloudquery/aws) - [PostgreSQL Plugin](https://hub.cloudquery.io/plugins/destination/cloudquery/postgresql) - [AWS Asset Inventory Pack](https://hub.cloudquery.io/addons/transformation/cloudquery/aws-asset-inventory/) ### Setting up the dashboard 1. Follow the instructions to set up the [AWS Asset Inventory Transformation](https://hub.cloudquery.io/addons/transformation/cloudquery/aws-asset-inventory/). 2. In Grafana, make sure your PostgreSQL database is added as a data source. 3. Download this package (using the **Download now** button on top) and extract this package. 4. [Import](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/) the `asset_inventory.json` dashboard definition from this package. - Note: If you have installed Postgres via Docker, ensure that Grafana is also installed via Docker. Once installed, you can use the IP address of your postgres container found by running `docker inspect container ` along with the port as your host connection. ### Example dashboard Once you have connected Grafana to your Postgres database and have imported the dashboard template, you should see a dashboard similar to this one: ![AWS Asset Inventory](https://storage.googleapis.com/cq-cloud-images/cloudquery/f939272f17c16ff2440024c36250992cdc9f95bd_asset_inventory_dash.png) --- Source: https://www.cloudquery.io/hub/addons/visualization/cloudquery/aws-data-resilience/latest/docs # CloudQuery AWS Data Resilience and Backup Dashboard for Grafana ## Overview This package contains an AWS Resilience and Backup Dashboard for Grafana on top of CloudQuery [AWS plugin](https://hub.cloudquery.io/plugins/source/cloudquery/aws) and [AWS Data Resilience transformation](https://hub.cloudquery.io/addons/transformation/cloudquery/aws-data-resilience) with PostgreSQL database. ## Requirements - [CloudQuery](https://cli-docs.cloudquery.io/docs/quickstart/) - [AWS Plugin](https://hub.cloudquery.io/plugins/source/cloudquery/aws) - [PostgreSQL Plugin](https://hub.cloudquery.io/plugins/destination/cloudquery/postgresql) - [AWS Data Resilience Pack](https://hub.cloudquery.io/addons/transformation/cloudquery/aws-data-resilience/) ### Setting up the dashboard 1. Follow the instructions to set up the [AWS Data Resilience Transformation](https://hub.cloudquery.io/addons/transformation/cloudquery/aws-data-resilience/) 2. In Grafana, make sure your PostgreSQL database is added as a data source. 3. Download this package (using the **Download now** button on top) and extract this package. 4. [Import](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/) the `backup-health.json` dashboard definition from this package. - Note: If you have installed Postgres via Docker, ensure that Grafana is also installed via Docker. Once installed, you can use the IP address of your postgres container found by running `docker inspect container ` along with the port as your host connection. 5. Update Grafana variables as needed for Data Source and Frameworks. ### Example dashboard Once you have connected Grafana to your Postgres database and have imported the AWS Asset Inventory template, you will see a dashboard similar to this one: ![AWS Data Resilience](https://storage.googleapis.com/cq-cloud-images/cloudquery/92edc525ec9ab41558e4fe9c8135978213e51f3a_aws-resiliency-dash.png) --- Source: https://www.cloudquery.io/hub/plugins/destination/cloudquery/bigquery/latest/docs # bigquery Destination Integration --- name: BigQuery stage: GA title: BigQuery Destination Plugin description: CloudQuery BigQuery destination plugin documentation --- # BigQuery Destination Plugin :badge The BigQuery plugin syncs data from any CloudQuery source plugin(s) to a BigQuery database running on Google Cloud Platform. The plugin currently only supports a streaming mode through the legacy streaming API. This is suitable for small- to medium-sized datasets, and will stream the results directly to the BigQuery database. A batch mode of operation is being developed to support larger datasets, but this is not currently supported. :::callout{type="info"} Streaming is not available for the [Google Cloud free tier](https://cloud.google.com/bigquery/pricing#free-tier). ::: ## Before you begin 1. Make sure that billing is enabled for your Cloud project. Learn how to [check if billing is enabled on a project](https://cloud.google.com/billing/docs/how-to/verify-billing-enabled). 2. Create a BigQuery dataset that will contain the tables synced by CloudQuery. CloudQuery will automatically create the tables as part of a migration run on the first `sync`. 3. Ensure that you have write access to the dataset. See [Required Permissions](https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery) for details. ## Example config :configuration The BigQuery destination utilizes batching, and supports [`batch_size`](https://www.cloudquery.io/docs/cli/integrations/destinations#batch_size) and [`batch_size_bytes`](https://www.cloudquery.io/docs/cli/integrations/destinations#batch_size_bytes). Note that the BigQuery plugin only supports the `append` write mode. ## Authentication :authentication ## BigQuery Spec This is the top-level spec used by the BigQuery destination plugin. - `project_id` (`string`) (required) The id of the project where the destination BigQuery database resides. - `dataset_id` (`string`) (required) The name of the BigQuery dataset within the project, e.g. `my_dataset`. This dataset needs to be created before running a sync or migration. - `dataset_location` (`string`) (optional) The data location of the BigQuery dataset. If set, will be used as the default location for job operations. Pro-tip: this can solve "dataset not found" issues for newly created datasets. - `time_partitioning` (`string`) (options: `none`, `hour`, `day`) (default: `none`) The time partitioning to use when creating tables. The partition time column used will always be `_cq_sync_time` so that all rows for a sync run will be partitioned on the hour/day the sync started. - `time_partitioning_expiration` (`duration`) (optional) The time after which the partition will be automatically deleted. The duration is specified in seconds, minutes or hours, e.g. `3600s`, `60m`, `24h`, `720h`. This option is only valid if `time_partitioning` is set a value other than `none`. - `service_account_key_json` (`string`) (optional) (default: empty). GCP service account key content. This allows for using different service accounts for the GCP source and BigQuery destination. If using service account keys, it is best to use [environment or file variable substitution](https://www.cloudquery.io/docs/cli/managing-cloudquery/environment-variables). - `endpoint` (`string`) (optional) The BigQuery API endpoint to use. This is useful for testing against a local emulator. - `batch_size` (`integer`) (optional) (default: `10000`) Number of records to write before starting a new object. - `batch_size_bytes` (`integer`) (optional) (default: `5242880` (5 MiB)) Number of bytes (as Arrow buffer size) to write before starting a new object. - `batch_timeout` (`duration`) (optional) (default: `10s` (10 seconds)) Maximum interval between batch writes. - `text_embeddings` (`object`) (optional) Configuration for creating text embeddings for certain tables using a Vertex AI model. A remote model must be attached to the dataset before using this feature, and its name supplied in the `remote_model_name` field. - `remote_model_name` (`string`) (required) The name of the remote model to use for text embeddings. - `tables` (`array`) (required) The tables to create text embeddings for. Each one must have its own configuration. - `source_table_name` (`string`) (required) The name of the source table to create text embeddings for. - `target_table_name` (`string`) (required) The name of the target table in which to store the embeddings. - `embed_columns` (`array`) (required) The columns to use as content for the embeddings generation function. They will be concatenated in the order they are provided. - `metadata_columns` (`array`) (optional) Which columns to copy as-is from the source table to the target table. `_cq_id` is always included. - `text_splitter` (`object`) (optional) The text splitter configuration to use for text embeddings. Currently only `recursive_text` is supported. - `recursive_text` (`object`) (required) - `chunk_size` (`integer`) (required) The size of the chunks in characters (not tokens). Defaults to `1000`. - `chunk_overlap` (`integer`) (required) The overlap between chunks in characters (not tokens). Defaults to `100`. ## Underlying library We use the official [cloud.google.com/go/bigquery](https://pkg.go.dev/cloud.google.com/go/bigquery) package for database connection. # BigQuery Types The BigQuery destination (`v3.0.0` and later) supports most [Apache Arrow](https://arrow.apache.org/docs/index.html) types. The following table shows the supported types and how they are mapped to [BigQuery data types](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types). | Arrow Column Type | Supported? | BigQuery Type | |------------------------|------------|----------------------------------------------------------| | Binary | ✅ Yes | `BYTES` | | Boolean | ✅ Yes | `BOOL` | | Date32 | ✅ Yes | `DATE` | | Date64 | ✅ Yes | `DATE` | | Decimal | ✅ Yes | `BIGNUMERIC` | | Dense Union | ❌ No | | | Dictionary | ❌ No | | | Duration | ✅ Yes | `INT64` | | Fixed Size List | ✅ Yes | (Repeated column) † | | Float16 | ✅ Yes | `FLOAT64` | | Float32 | ✅ Yes | `FLOAT64` | | Float64 | ✅ Yes | `FLOAT64` | | Inet | ✅ Yes | `STRING` | | Int8 | ✅ Yes | `INT64` | | Int16 | ✅ Yes | `INT64` | | Int32 | ✅ Yes | `INT64` | | Int64 | ✅ Yes | `INT64` | | Interval[DayTime] | ✅ Yes | `RECORD` | | Interval[MonthDayNano] | ✅ Yes | `RECORD` | | Interval[Month] | ✅ Yes | `RECORD` | | JSON | ✅ Yes | `JSON` | | Large Binary | ✅ Yes | `BYTES` | | Large List | ✅ Yes | (Repeated column) † | | Large String | ✅ Yes | `STRING` | | List | ✅ Yes | (Repeated column) † | | MAC | ✅ Yes | `STRING` | | Map | ❌ No | | | String | ✅ Yes | `STRING` | | Struct | ✅ Yes | `RECORD` | | Timestamp | ✅ Yes | `TIMESTAMP` | | UUID | ✅ Yes | `STRING` | | Uint8 | ✅ Yes | `INT64` | | Uint16 | ✅ Yes | `INT64` | | Uint32 | ✅ Yes | `INT64` | | Uint64 | ✅ Yes | `NUMERIC` | | Union | ❌ No | | ## Notes † Repeated columns in BigQuery do not support null values. Right now, if an array contains null values, these null values will be dropped when writing to BigQuery. Also, because we use `REPEATED` columns to represent lists, lists of lists are not supported right now. The BigQuery plugin authenticates using your [Application Default Credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default). Available options are all the same options described [here](https://cloud.google.com/docs/authentication/provide-credentials-adc) in detail: Local Environment: - `gcloud auth application-default login` (recommended when running locally) Google Cloud cloud-based development environment: - When you run on Cloud Shell or Cloud Code credentials are already available. Google Cloud containerized environment: - When running on GKE use [workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity). [Google Cloud services that support attaching a service account](https://cloud.google.com/docs/authentication/provide-credentials-adc#attached-sa): - Services such as Compute Engine, App Engine and functions supporting attaching a user-managed service account which will CloudQuery will be able to utilize. On-premises or another cloud provider - The suggested way is to use [Workload identity federation](https://cloud.google.com/iam/docs/workload-identity-federation) - If not available you can always use service account keys and export the location of the key via `GOOGLE_APPLICATION_CREDENTIALS`. (**Not recommended as long-lived keys are a security risk**) ```yaml copy kind: destination spec: name: bigquery path: cloudquery/bigquery registry: cloudquery version: "v4.7.11" write_mode: "append" send_sync_summary: true # Learn more about the configuration options at https://cql.ink/bigquery_destination spec: project_id: ${PROJECT_ID} dataset_id: ${DATASET_ID} # Optional parameters # service_account_key_json: '${file:./path-to-your-file.json}' # GCP service account key, can be placed in a file and referenced with this syntax. # dataset_location: "" # time_partitioning: none # options: "none", "hour", "day", "month", "year" # time_partitioning_expiration: 0h # duration, e.g. "24h" or "720h" (30 days) # service_account_key_json: "" # endpoint: "" # batch_size: 10000 # batch_size_bytes: 5242880 # 5 MiB # batch_timeout: 10s # client_project_id: "*detect-project-id*" ``` This example above expects the following environment variables to be set: * `PROJECT_ID` - The Google Cloud Project ID * `DATASET_ID` - The Google Cloud BigQuery Dataset ID `client_project_id` variable can be used to run BigQuery queries in a project different from where the destination table is located. If you set client_project_id to `*detect-project-id*`, it will automatically detect the project ID from the environment variable or application default credentials. --- Source: https://www.cloudquery.io/hub/plugins/destination/cloudquery/clickhouse/latest/docs # clickhouse Destination Integration --- name: ClickHouse stage: GA title: ClickHouse Destination Plugin description: CloudQuery ClickHouse destination plugin documentation --- # ClickHouse destination plugin :badge This destination plugin lets you sync data from a CloudQuery source to [ClickHouse](https://clickhouse.com/) database. It supports `append` write mode only. Write mode selection is required through [`write_mode`](https://www.cloudquery.io/docs/cli/integrations/destinations#write_mode). Supported database versions: >= `24.8.1` ## Configuration ### Example :configuration :::callout{type="info"} Make sure you use environment variable expansion in production instead of committing the credentials to the configuration file directly. ::: ### ClickHouse spec This is the (nested) spec used by the ClickHouse destination plugin. - `connection_string` (`string`) (required) Connection string to connect to the database. See [SDK documentation](https://github.com/ClickHouse/clickhouse-go#dsn) for more details. Example connection string: - `"clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60"` - `cluster` (`string`) (optional) (default: not used) Cluster name to be used for [distributed DDL](https://clickhouse.com/docs/en/sql-reference/distributed-ddl). If the value is empty, DDL operations will affect only the server the plugin is connected to. - `ca_cert` (`string`) (optional) (default: not used) PEM-encoded certificate authorities. When set, a certificate pool will be created by appending the certificates to the system pool. See [file variable substitution](https://www.cloudquery.io/docs/cli/managing-cloudquery/environment-variables#file-variable-substitution-example) for how to read this value from a file. - `engine` (optional, [table engine settings](#clickhouse-table-engine). Default: `MergeTree` engine) Engine to be used for tables. Only [`*MergeTree` family](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family) is supported at the moment. - `batch_size` (`integer`) (optional) (default: `10000`) Maximum number of items that may be grouped together to be written in a single write. - `batch_size_bytes` (`integer`) (optional) (default: `5242880` (= 5 MiB)) Maximum size of items that may be grouped together to be written in a single write. - `batch_timeout` (`duration`) (optional) (default: `20s`) Maximum interval between batch writes. - `partition` (optional, [partitioning](#partitioning)) (default: no partitioning) Partitioning strategy to be used for tables (i.e. `PARTITION BY` clause in `CREATE TABLE` statements). - `order` (optional, [ordering](#ordering)) (default: use existing primary key) Ordering strategy to be used for tables (i.e. `ORDER BY` clause in `CREATE TABLE` statements). #### ClickHouse table engine This option allows to specify a custom table engine to be used. - `name` (`string`) (required) Name of the table engine. Only [`*MergeTree` family](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family) is supported at the moment. - `parameters` (array of parameters) (optional) (default: empty) Engine parameters. Currently, no restrictions are imposed on the parameter types. ```yaml copy kind: destination spec: name: "clickhouse" path: "cloudquery/clickhouse" registry: "cloudquery" version: "v8.2.4" write_mode: "append" send_sync_summary: true spec: connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}" engine: name: ReplicatedMergeTree parameters: - "/clickhouse/tables/{shard}/{database}/{table}" - "{replica}" ``` #### Partitioning This option allows you to specify a partitioning strategy to be used for tables. It is an array of objects. Each object has the following fields: - `tables` (array of strings) (optional) (default: `["*"]`) List of glob patterns to match table names against. Follows the same rules as the top-level spec `tables` option. If a table matches both a pattern in `tables` and `skip_tables`, the table will be skipped. Partition strategy table patterns should be disjointed sets: if a table matches two partition strategies, an error will be raised at runtime. - `skip_tables` (array of strings) (optional) (default: empty) List of glob patterns to skip matching table names against. Follows the same rules as the top-level spec `skip_tables` option. If a table matches both a pattern in `tables` and `skip_tables`, the table will be skipped. Partition strategy table patterns should be disjointed sets: if a table matches two partition strategies, an error will be raised at runtime. - `skip_incremental_tables` (boolean) (optional) (default: `false`) If set to `true`, incremental tables will not be partitioned by this strategy. - `partition_by` (string) (required) Partitioning strategy to use, e.g. `toYYYYMM(_cq_sync_time)`, the string is passed as is after "PARTITION BY" clause with no validation or quoting. An unset `partition_by` is not valid. Example: ```yaml copy partition: - tables: ["*"] skip_tables: ["special_partition_table", "non_partitioned_table"] partition_by: "toYYYYMM(_cq_sync_time)" - tables: ["special_partition_table"] partition_by: "toYYYYMMDD(_cq_sync_time)" ``` #### Ordering This option allows you to specify custom `ORDER BY` clauses for tables or groups of tables. It is an array of objects. Each object has the following fields: - `tables` (array of strings) (optional) (default: `["*"]`) List of glob patterns to match table names against. Follows the same rules as the top-level spec `tables` option. If a table matches both a pattern in `tables` and `skip_tables`, the table will be skipped. Ordering strategy table patterns should be disjointed sets: if a table matches two ordering strategies, an error will be raised at runtime. - `skip_tables` (array of strings) (optional) (default: empty) List of glob patterns to skip matching table names against. Follows the same rules as the top-level spec `skip_tables` option. If a table matches both a pattern in `tables` and `skip_tables`, the table will be skipped. Ordering strategy table patterns should be disjointed sets: if a table matches two ordering strategies, an error will be raised at runtime. - `order_by` (array of strings) (required) Sort key to use, the strings are passed as is after "ORDER BY" clause with no validation or quoting. Example: ```yaml copy order: - tables: ["aws_ec2_instances"] order_by: - "`account_id`" - "`region`" - "toYYYYMM(`_cq_sync_time`) DESC" - "`_cq_id`" ``` #### Table TTLs This option allows you to specify a TTL strategy to be used for tables. It is an array of objects. Each object has the following fields: - `tables` (array of strings) (optional) (default: `["*"]`) List of glob patterns to match table names against. Follows the same rules as the top-level spec `tables` option. If a table matches both a pattern in `tables` and `skip_tables`, the table will be skipped. TTL strategy table patterns should be disjointed sets: if a table matches two TTL strategies, an error will be raised at runtime. - `skip_tables` (array of strings) (optional) (default: empty) List of glob patterns to skip matching table names against. Follows the same rules as the top-level spec `skip_tables` option. If a table matches both a pattern in `tables` and `skip_tables`, the table will be skipped. TTL strategy table patterns should be disjointed sets: if a table matches two TTL strategies, an error will be raised at runtime. - `ttl` (string) (required) TTL interval to use, relative to _cq_sync_time. E.g. `INTERVAL 60 DAY`, the string is passed as-is after a "TTL _cq_sync_time + " clause with no validation or quoting. Example: ```yaml copy ttl: - tables: ["*"] skip_tables: ["no_ttl_table"] ttl: "INTERVAL 60 DAY" ``` ### Connecting to ClickHouse Cloud To connect to [ClickHouse Cloud](https://clickhouse.com/cloud), you need to set the `secure=true` parameter, username is `default`, and the port is `9440`. Use a connection string similar to: ```yaml copy connection_string: "clickhouse://default:${CH_PASSWORD}@...clickhouse.cloud:9440/${CH_DATABASE}?secure=true" ``` See [Quick Start: Using the ClickHouse Client](https://clickhouse.com/docs/en/cloud-quick-start#5-using-the-clickhouse-client) for more details. #### Verbose logging for debug The ClickHouse destination can be run in debug mode. To achieve this pass the `debug=true` option to `connection_string`. See [SDK documentation](https://github.com/ClickHouse/clickhouse-go#dsn) for more details. Note: This will use [SDK](https://github.com/ClickHouse/clickhouse-go) built-in logging and might output data and sensitive information to logs. Make sure not to use it in production environment. ```yaml copy kind: destination spec: name: "clickhouse" path: "cloudquery/clickhouse" registry: "cloudquery" version: "v8.2.4" write_mode: "append" send_sync_summary: true spec: connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}?debug=true" ``` ## Apache Arrow type conversion The ClickHouse destination plugin supports most of [Apache Arrow](https://arrow.apache.org/docs/index.html) types. It uses the same approach as documented in [ClickHouse reference](https://clickhouse.com/docs/en/sql-reference/formats#data-format-arrow). The following table shows the supported types and how they are mapped to [ClickHouse data types](https://clickhouse.com/docs/en/sql-reference/data-types). :::callout{type="info"} Unsupported types are always mapped to [String](https://clickhouse.com/docs/en/sql-reference/data-types/string). ::: | Arrow Column Type | ClickHouse Type | |-----------------------------|------------------------------------------------------------------------------------| | Binary | [String](https://clickhouse.com/docs/en/sql-reference/data-types/string) | | Binary View | [String](https://clickhouse.com/docs/en/sql-reference/data-types/string) | | Boolean | [Bool](https://clickhouse.com/docs/en/sql-reference/data-types/boolean) | | Date32 | [Date32](https://clickhouse.com/docs/en/sql-reference/data-types/date32) | | Date64 | [DateTime](https://clickhouse.com/docs/en/sql-reference/data-types/datetime) | | Decimal128 (Decimal) | [Decimal](https://clickhouse.com/docs/en/sql-reference/data-types/decimal) | | Decimal256 | [Decimal](https://clickhouse.com/docs/en/sql-reference/data-types/decimal) | | Fixed Size Binary | [FixedString](https://clickhouse.com/docs/en/sql-reference/data-types/fixedstring) | | Fixed Size List | [Array](https://clickhouse.com/docs/en/sql-reference/data-types/array) | | Float16 | [Float32](https://clickhouse.com/docs/en/sql-reference/data-types/float) | | Float32 | [Float32](https://clickhouse.com/docs/en/sql-reference/data-types/float) | | Float64 | [Float64](https://clickhouse.com/docs/en/sql-reference/data-types/float) | | Int8 | [Int8](https://clickhouse.com/docs/en/sql-reference/data-types/int-uint) | | Int16 | [Int16](https://clickhouse.com/docs/en/sql-reference/data-types/int-uint) | | Int32 | [Int32](https://clickhouse.com/docs/en/sql-reference/data-types/int-uint) | | Int64 | [Int64](https://clickhouse.com/docs/en/sql-reference/data-types/int-uint) | | Large Binary | [String](https://clickhouse.com/docs/en/sql-reference/data-types/string) | | Large List | [Array](https://clickhouse.com/docs/en/sql-reference/data-types/array) | | Large String | [String](https://clickhouse.com/docs/en/sql-reference/data-types/string) | | List | [Array](https://clickhouse.com/docs/en/sql-reference/data-types/array) | | Map | [Map](https://clickhouse.com/docs/en/sql-reference/data-types/map) | | String | [String](https://clickhouse.com/docs/en/sql-reference/data-types/string) | | String View | [String](https://clickhouse.com/docs/en/sql-reference/data-types/string) | | Struct | [Tuple](https://clickhouse.com/docs/en/sql-reference/data-types/tuple) | | Time32 | [DateTime64](https://clickhouse.com/docs/en/sql-reference/data-types/datetime64) | | Time64 | [DateTime64](https://clickhouse.com/docs/en/sql-reference/data-types/datetime64) | | Timestamp | [DateTime64](https://clickhouse.com/docs/en/sql-reference/data-types/datetime64) | | UUID (CloudQuery extension) | [UUID](https://clickhouse.com/docs/en/sql-reference/data-types/uuid) | | Uint8 | [UInt8](https://clickhouse.com/docs/en/sql-reference/data-types/int-uint) | | Uint16 | [UInt16](https://clickhouse.com/docs/en/sql-reference/data-types/int-uint) | | Uint32 | [UInt32](https://clickhouse.com/docs/en/sql-reference/data-types/int-uint) | | Uint64 | [UInt64](https://clickhouse.com/docs/en/sql-reference/data-types/int-uint) | :::callout{type="info"} [Nested](https://clickhouse.com/docs/en/sql-reference/data-types/nested-data-structures/nested) ClickHouse types have their values converted according to the aforementioned rules. ::: ```yaml copy kind: destination spec: name: "clickhouse" path: "cloudquery/clickhouse" registry: "cloudquery" version: "v8.2.4" write_mode: "append" send_sync_summary: true # Learn more about the configuration options at https://cql.ink/clickhouse_destination spec: connection_string: "clickhouse://${CH_USER}:${CH_PASSWORD}@localhost:9000/${CH_DATABASE}" # Optional parameters # cluster: "" # ca_cert: "" # engine: # name: MergeTree # parameters: [] # # batch_size: 10000 # batch_size_bytes: 5242880 # 5 MiB # batch_timeout: 20s ``` This example configures a ClickHouse instance, located at `localhost:9000`. It expects `CH_USER`, `CH_PASSWORD` and `CH_DATABASE` environment variables to be set. The (top level) spec section is described in the [Destination Spec Reference](https://www.cloudquery.io/docs/cli/integrations/destinations#complete-destination-spec-reference). --- Source: https://www.cloudquery.io/hub/plugins/destination/cloudquery/databricks/latest/docs # databricks Destination Integration --- name: Databricks stage: Preview title: Databricks Destination Plugin description: CloudQuery Databricks destination plugin documentation --- # Databricks destination plugin :badge This destination plugin lets you sync data from a CloudQuery source to [Databricks](https://databricks.com/). Supported Databricks versions: >= `12` ## Configuration ### Example :configuration :::callout{type="info"} Make sure you use environment variable expansion in production instead of committing the credentials to the configuration file directly. ::: ### Databricks spec This is the (nested) spec used by the Databricks destination plugin. - `hostname` (`string`) (required) SQL compute hostname. May optionally include `protocol` value as well (like `https://server.databricks.com`). - `http_path` (`string`) (required) SQL compute HTTP path. - `staging_path` (`string`) (required) Unity volume path where temporary (staging) files should be uploaded to. - `auth` ([Auth spec](#databricks-authentication-spec)) (required) Authentication options. - `catalog` (`string`) (required) Catalog to be used. - `protocol` (`string`) (optional) (default: `https`) Protocol for connecting to Databricks. Can be also specified in the `hostname`. - `port` (`integer`) (optional) (default: `443`) Port for connecting to Databricks. - `schema` (`string`) (optional) (default: `cloudquery`) Schema to be used. If it doesn't exist, it will be created. - `batch` ([Batching spec](#batching-spec)) (optional) Batching options. - `migration_concurrency` (`integer`) (optional) (default: `10`) How many table operations will be performed in parallel during migration. - `timeout` (`duration`) (optional) (default: `1m` (= 1 minute)) Timeout for the queries. - `retries` ([Retries spec](#retries-spec)) (optional) Retry options for staging file operations. #### Databricks authentication spec This section allows specifying authentication method to connect to Databricks. Currently only personal access tokens are supported. - `access_token` (`string`) (required) Personal access token. #### Batching spec This section controls how data is batched for writing. - `size` (`integer`) (optional) (default: `10000`) Maximum number of items that may be grouped together to be written in a single write. - `bytes` (`integer`) (optional) (default: `5242880` (= 5 MiB)) Maximum size of items that may be grouped together to be written in a single write. - `timeout` (`duration`) (optional) (default: `1m` (= 1 minute)) Maximum interval between batch writes. #### Retries spec This section controls how staging file operations (put, remove) are retried on failure. - `max_attempts` (`integer`) (optional) (default: `3`) Maximum number of attempts for each staging file operation. - `delay` (`duration`) (optional) (default: `1s`) Delay between attempts. ## Apache Arrow type conversion The Databricks destination plugin supports most of [Apache Arrow](https://arrow.apache.org/docs/index.html) types. The following table shows the supported types and how they are mapped to [Databricks data types](https://docs.databricks.com/en/sql/language-manual/sql-ref-datatypes.html). :::callout{type="info"} Unsupported types are always mapped to [STRING](https://docs.databricks.com/en/sql/language-manual/data-types/string-type.html). ::: | Arrow Column Type | Databricks Type | |-----------------------------|------------------------------------------------------------------------------------------------| | Binary | [BINARY](https://docs.databricks.com/en/sql/language-manual/data-types/binary-type.html) | | Binary View | [BINARY](https://docs.databricks.com/en/sql/language-manual/data-types/binary-type.html) | | Boolean | [BOOLEAN](https://docs.databricks.com/en/sql/language-manual/data-types/boolean-type.html) | | Date32 | [DATE](https://docs.databricks.com/en/sql/language-manual/data-types/date-type.html) | | Date64 | [DATE](https://docs.databricks.com/en/sql/language-manual/data-types/date-type.html) | | Decimal128 (Decimal) | [DECIMAL](https://docs.databricks.com/en/sql/language-manual/data-types/decimal-type.html) | | Decimal256 | [DECIMAL](https://docs.databricks.com/en/sql/language-manual/data-types/decimal-type.html) | | Fixed Size Binary | [BINARY](https://docs.databricks.com/en/sql/language-manual/data-types/binary-type.html) | | Fixed Size List | [ARRAY](https://docs.databricks.com/en/sql/language-manual/data-types/array-type.html) | | Float16 | [FLOAT](https://docs.databricks.com/en/sql/language-manual/data-types/float-type.html) | | Float32 | [FLOAT](https://docs.databricks.com/en/sql/language-manual/data-types/float-type.html) | | Float64 | [DOUBLE](https://docs.databricks.com/en/sql/language-manual/data-types/double-type.html) | | Int8 | [TINYINT](https://docs.databricks.com/en/sql/language-manual/data-types/tinyint-type.html) | | Int16 | [SMALLINT](https://docs.databricks.com/en/sql/language-manual/data-types/smallint-type.html) | | Int32 | [INTEGER](https://docs.databricks.com/en/sql/language-manual/data-types/int-type.html) | | Int64 | [BIGINT](https://docs.databricks.com/en/sql/language-manual/data-types/bigint-type.html) | | Large Binary | [BINARY](https://docs.databricks.com/en/sql/language-manual/data-types/binary-type.html) | | Large List | [ARRAY](https://docs.databricks.com/en/sql/language-manual/data-types/array-type.html) | | Large String | [STRING](https://docs.databricks.com/en/sql/language-manual/data-types/string-type.html) | | List | [ARRAY](https://docs.databricks.com/en/sql/language-manual/data-types/array-type.html) | | Null | [VOID](https://docs.databricks.com/en/sql/language-manual/data-types/null-type.html) | | Map | [MAP](https://docs.databricks.com/en/sql/language-manual/data-types/map-type.html) | | String | [STRING](https://docs.databricks.com/en/sql/language-manual/data-types/string-type.html) | | String View | [STRING](https://docs.databricks.com/en/sql/language-manual/data-types/string-type.html) | | Struct | [STRUCT](https://docs.databricks.com/en/sql/language-manual/data-types/struct-type.html) | | Time32 | [TIMESTAMP](https://docs.databricks.com/en/sql/language-manual/data-types/timestamp-type.html) | | Time64 | [TIMESTAMP](https://docs.databricks.com/en/sql/language-manual/data-types/timestamp-type.html) | | Timestamp | [TIMESTAMP](https://docs.databricks.com/en/sql/language-manual/data-types/timestamp-type.html) | | UUID (CloudQuery extension) | [STRING](https://docs.databricks.com/en/sql/language-manual/data-types/string-type.html) | | Uint8 | [SMALLINT](https://docs.databricks.com/en/sql/language-manual/data-types/smallint-type.html) | | Uint16 | [INTEGER](https://docs.databricks.com/en/sql/language-manual/data-types/int-type.html) | | Uint32 | [BIGINT](https://docs.databricks.com/en/sql/language-manual/data-types/bigint-type.html) | | Uint64 | [BIGINT](https://docs.databricks.com/en/sql/language-manual/data-types/bigint-type.html) | :::callout{type="info"} Nested Apache Arrow types (lists, structs, map) have their element values converted according to the aforementioned rules. ::: ```yaml copy kind: destination spec: name: "databricks" path: "cloudquery/databricks" registry: "cloudquery" version: "v1.6.12" write_mode: "append" spec: hostname: ${DATABRICKS_HOSTNAME} # optionally it can include protocol like https://abc.cloud.databricks.com http_path: ${DATABRICKS_HTTP_PATH} # HTTP path for SQL compute staging_path: ${DATABRICKS_STAGING_PATH} # Databricks FileStore or Unity volume path to store temporary files for staging auth: access_token: ${DATABRICKS_ACCESS_TOKEN} # Optional parameters # protocol: https # port: 443 # catalog: "" # schema: "default" # migration_concurrency: 10 # timeout: 1m # batch: # size: 10000 # bytes: 5242880 # 5 MiB # timeout: 20s # retries: # retry policy for staging file operations (put, remove) # max_attempts: 3 # delay: 1s ``` The (top level) spec section is described in the [Destination Spec Reference](/docs/reference/destination-spec). --- Source: https://www.cloudquery.io/hub/plugins/destination/cloudquery/elasticsearch/latest/docs # elasticsearch Destination Integration --- name: Elasticsearch title: Elasticsearch Destination Plugin description: CloudQuery Elasticsearch destination plugin documentation --- # Elasticsearch Destination Plugin :badge The Elasticsearch plugin syncs data from any CloudQuery source plugin(s) to an Elasticsearch cluster. ## Example config :configuration The Elasticsearch destination utilizes batching, and supports [`batch_size`](https://www.cloudquery.io/docs/cli/integrations/destinations#batch_size) and [`batch_size_bytes`](https://www.cloudquery.io/docs/cli/integrations/destinations#batch_size_bytes). It supports `append`, `overwrite` and `overwrite-delete-stale` write modes. The default write mode is `overwrite-delete-stale`. ## Elasticsearch Spec This is the spec used by the Elasticsearch destination plugin. - `addresses` (`[]string`) (optional) (default: `["http://localhost:9200"]`) A list of Elasticsearch nodes to use. Mutually exclusive with `cloud_id`. - `username` (`string`) (optional) Username for HTTP Basic Authentication. - `password` (`string`) (optional) Password for HTTP Basic Authentication. - `cloud_id` (`string`) (optional) (example: `MyDeployment:abcdefgh`) Endpoint for the Elasticsearch Service (https://elastic.co/cloud). Mutually exclusive with `addresses`. - `api_key` (`string`) (optional) Base64-encoded token for authorization; if set, overrides username/password and service token. - `service_token` (`string`) (optional) Service token for authorization; if set, overrides username/password. - `certificate_fingerprint` (`string`) (optional) SHA256 hex fingerprint given by Elasticsearch on first launch. - `ca_cert` (`string`) (optional) PEM-encoded certificate authorities. When set, an empty certificate pool will be created, and the certificates will be appended to it. See [file variable substitution](https://www.cloudquery.io/docs/cli/managing-cloudquery/environment-variables#file-variable-substitution-example) for how to read this value from a file. - `concurrency` (`string`) (optional) (default: number of CPUs) Number of concurrent worker goroutines to use for indexing. - `batch_size` (`integer`) (optional) (default: `1000`) Maximum number of items that may be grouped together to be written in a single write. - `batch_size_bytes` (`integer`) (optional) (default: `5242880` (5 MiB)) Maximum size of items that may be grouped together to be written in a single write. ## Index Template Creation The Elasticsearch destination will create an index template for every table during the migration step. It is recommended that you use the generated index templates, as it will automatically create indexes with the correct mappings for the table. However, to skip index template creation (or use your own), you may use the `--no-migrate` option when running `cloudquery sync`. ## Index Naming Index names will be formatted according to the selected write mode: - `append`: indexes will be named using the format `-`. In other words, a new index will be created every day the table is synced. Entries will never be overwritten. - `overwrite`: indexes will be named using the format ``. Objects with duplicate primary keys will be overwritten. - `overwrite-delete-stale`: indexes will be named using the format ``. Objects with duplicate primary keys will be overwritten, and any objects that are not present in the current sync will be deleted. Index templates will also be created such that they match the index names generated by the selected write mode. ## Querying From Kibana To query data from Kibana, you will need to create [data views](https://www.elastic.co/guide/en/kibana/9.0/data-views.html) (previously also known as "index patterns"). To query a specific table, the data view's index pattern should be in the format `-*`. For example, if you have a table named `aws_ec2_instances`, you should create a data view with index pattern named `aws_ec2_instances-*`. One useful feature of Elasticsearch and Kibana, however, is the ability to query across all data. To do this for the `aws` source plugin, for example, you may use an index pattern named `aws_*`. This will then allow queries across all tables synced by the `aws` source plugin. ## Underlying library We use the official [go-elasticsearch](https://github.com/elastic/go-elasticsearch) package. It is tested against Elasticsearch 9.0 and above. Please [open an issue](https://github.com/cloudquery/cloudquery/issues/new?title=Elasticsearch+destination+plugin) if you encounter any problems with this (or another) version. # Elasticsearch Types The Elasticsearch destination (`v2.0.0` and later) supports most [Apache Arrow](https://arrow.apache.org/docs/index.html) types. The following table shows the supported types and how they are mapped to [Elasticsearch field data types](https://www.elastic.co/guide/en/elasticsearch/reference/9.0/mapping-types.html). | Arrow Column Type | Supported? | Elasticsearch Type | |------------------------|------------|----------------------------------------------------------| | Binary | ✅ Yes | `binary` | | Boolean | ✅ Yes | `boolean` | | Date32 | ✅ Yes | `date` with format `yyyy-MM-dd` | | Date64 | ✅ Yes | `date` with format `yyyy-MM-dd` | | Decimal | ✅ Yes | `text` | | Dense Union | ✅ Yes | `text` | | Dictionary | ✅ Yes | `text` | | Duration[ms] | ✅ Yes | `text` | | Duration[ns] | ✅ Yes | `text` | | Duration[s] | ✅ Yes | `text` | | Duration[us] | ✅ Yes | `text` | | Fixed Size List | ✅ Yes | Uses type from list elements | | Float16 | ✅ Yes | `half_float` | | Float32 | ✅ Yes | `float` | | Float64 | ✅ Yes | `double` | | Inet | ✅ Yes | `text` | | Int8 | ✅ Yes | `byte` | | Int16 | ✅ Yes | `short` | | Int32 | ✅ Yes | `integer` | | Int64 | ✅ Yes | `long` | | Interval[DayTime] | ✅ Yes | `object` | | Interval[MonthDayNano] | ✅ Yes | `object` | | Interval[Month] | ✅ Yes | `object` | | JSON | ✅ Yes | `text` | | Large Binary | ✅ Yes | `byte` | | Large List | ✅ Yes | Uses type from list elements | | Large String | ✅ Yes | `text` | | List | ✅ Yes | Uses type from list elements | | MAC | ✅ Yes | `text` | | Map | ✅ Yes | `object` with `key` and `value` fields | | String | ✅ Yes | `text` | | Struct | ✅ Yes | `object` | | Time32[s] | ✅ Yes | `date` with format `HH:mm:ss` | | Time32[ms] | ✅ Yes | `date` with format `HH:mm:ss.SSS` | | Time64[us] | ✅ Yes | `text` | | Time64[ns] | ✅ Yes | `text` | | Timestamp[s] | ✅ Yes | `date` with format `2006-01-02T15:04:05Z` | | Timestamp[ms] | ✅ Yes | `date` with format `2006-01-02T15:04:05.999Z` | | Timestamp[us] | ✅ Yes | `date` with format `2006-01-02T15:04:05.999999Z"` | | Timestamp[ns] | ✅ Yes | `date_nanos` with format `2006-01-02T15:04:05.99999999Z` | | UUID | ✅ Yes | `text` | | Uint8 | ✅ Yes | `unsigned_long` | | Uint16 | ✅ Yes | `unsigned_long` | | Uint32 | ✅ Yes | `unsigned_long` | | Uint64 | ✅ Yes | `unsigned_long` | | Union | ✅ Yes | `text` | The following config will sync data to an Elasticsearch cluster running on `localhost:9200`: ```yaml copy kind: destination spec: name: elasticsearch path: cloudquery/elasticsearch registry: cloudquery version: "v4.0.5" write_mode: "overwrite-delete-stale" send_sync_summary: true spec: # Elastic Cloud configuration parameters cloud_id: "${ELASTICSEARCH_CLOUD_ID}" api_key: "${ELASTICSEARCH_API_KEY}" # Self-hosted Elasticsearch configuration parameters # addresses: ["http://localhost:9200"] # username: "" # password: "" # service_token: "" # certificate_fingerprint: "" # ca_cert: "" # Optional parameters # concurrency: 5 # default: number of CPUs # batch_size: 1000 # batch_size_bytes: 5242880 # 5 MiB ``` --- Source: https://www.cloudquery.io/hub/plugins/destination/cloudquery/file/latest/docs # file Destination Integration --- name: File stage: GA title: File Destination Plugin description: CloudQuery File destination plugin for exporting to CSV, JSON and Parquet --- # File Destination Plugin :badge This destination plugin lets you sync data from a CloudQuery source to local files in various formats. It currently supports CSV, line-delimited JSON and Parquet. This plugin is useful in local environments, but also in production environments where scalability, performance and cost are requirements. For example, this plugin can be used as part of a system that syncs sources across multiple virtual machines, uploads Parquet files to a remote storage (such as S3 or GCS), and finally loads them to data lakes such as BigQuery or Athena in batch mode. If this is your end goal, you may also want to look at more specific destination cloud storage destination plugins such as [S3](https://www.cloudquery.io/hub/plugins/destination/cloudquery/s3/latest/docs), [GCS](https://www.cloudquery.io/hub/plugins/destination/cloudquery/gcs/latest/docs) or [Azure Blob Storage](https://www.cloudquery.io/hub/plugins/destination/cloudquery/azblob/latest/docs). ## Authentication :authentication ## Example :configuration ## File Spec This is the (nested) spec used by the file destination Plugin. - `path` (`string`) (**required**) Path template string that determines where files will be written, for example `path/to/files/{{TABLE}}/{{UUID}}.parquet`. The path supports the following placeholder variables: - `{{TABLE}}` will be replaced with the table name - `{{FORMAT}}` will be replaced with the file format, such as `csv`, `json` or `parquet`. If compression is enabled, the format will be `csv.gz`, `json.gz` etc. - `{{UUID}}` will be replaced with a random UUID to uniquely identify each file - `{{YEAR}}` will be replaced with the current year in `YYYY` format - `{{MONTH}}` will be replaced with the current month in `MM` format - `{{DAY}}` will be replaced with the current day in `DD` format - `{{HOUR}}` will be replaced with the current hour in `HH` format - `{{MINUTE}}` will be replaced with the current minute in `mm` format **Note** that timestamps are in `UTC` and will be the current time at the time the file is written, not when the sync started. - `format` (`string`) (**required**) Format of the output file. Supported values are `csv`, `json` and `parquet`. - `format_spec` ([format_spec](#format_spec)) (optional) Optional parameters to change the format of the file. - `no_rotate` (`boolean`) (optional) (default: `false`) If set to `true`, the plugin will write to one file per table. Otherwise, for every batch a new file will be created with a different `.` suffix. - `compression` (`string`) (optional) (default: `""`) Compression algorithm to use. Supported values are `""` and `gzip`. Not supported for `parquet` format. - `batch_size` (`integer`) (optional) (default: `10000`) Number of records to write before starting a new file. - `batch_size_bytes` (`integer`) (optional) (default: `52428800` (50 MiB)) Number of bytes (as Arrow buffer size) to write before starting a new file. - `batch_timeout` (`duration`) (optional) (default: `30s` (30 seconds)) Maximum interval between batch writes. ### format_spec #### CSV - `delimiter` (`string`) (optional) (default: `,`) Delimiter to use in the CSV file. - `skip_header` (`boolean`) (optional) (default: `false`) If set to `true`, the CSV file will not contain a header row as the first row. #### JSON Reserved for future use. #### Parquet - `version` (`string`) (optional) (default: `v2Latest`) Parquet format version to use. Supported values are `v1.0`, `v2.4`, `v2.6` and `v2Latest`. `v2Latest` is an alias for the latest version available in the Parquet library which is currently `v2.6`. Useful when the reader consuming the Parquet files does not support the latest version. - `root_repetition` (`string`) (optional) (default: `repeated`) [Repetition option to use for the root node](https://github.com/apache/arrow/issues/20243). Supported values are `undefined`, `required`, `optional` and `repeated`. Some Parquet readers require a specific root repetition option to be able to read the file. For example, importing Parquet files into [Snowflake](https://www.snowflake.com/en/) requires the root repetition to be `undefined`. - `max_row_group_length` (`integer`) (optional) (default: `134217728` (= 128 _ 1024 _ 1024)) The maximum number of rows in a single row group. Use a lower number to reduce memory usage when reading the Parquet files, and a higher number to increase the efficiency of reading the Parquet files. This plugin does not require any authentication. This example configures the file destination, to create CSV files in `./cq_csv_output`. You can also choose `json` or `parquet` as the output format. ```yaml copy kind: destination spec: name: "file" path: "cloudquery/file" registry: "cloudquery" version: "v5.5.10" write_mode: "append" send_sync_summary: true # Learn more about the configuration options at https://cql.ink/file_destination spec: path: "path/to/files/{{TABLE}}/{{UUID}}.{{FORMAT}}" format: "parquet" # options: parquet, json, csv # Optional parameters # format_spec: # CSV specific parameters: # delimiter: "," # skip_header: false # Parquet specific parameters: # version: "v2Latest" # root_repetition: "repeated" # max_row_group_length: 134217728 # 128 * 1024 * 1024 # compression: "" # options: gzip # no_rotate: false # batch_size: 10000 # batch_size_bytes: 52428800 # 50 MiB # batch_timeout: 30s ``` Note that the file plugin only supports `append` `write_mode`. The (top level) spec section is described in the [Destination Spec Reference](https://www.cloudquery.io/docs/cli/integrations/destinations#complete-destination-spec-reference). --- Source: https://www.cloudquery.io/hub/plugins/destination/cloudquery/gcs/latest/docs # gcs Destination Integration --- name: GCS stage: GA title: GCS Destination Plugin description: CloudQuery GCS destination plugin documentation --- # GCS (Google Cloud Storage) Destination Plugin :badge This destination plugin lets you sync data from a CloudQuery source to remote GCS (Google Cloud Storage) storage in various formats such as CSV, JSON and Parquet. This is useful in various use-cases, especially in data lakes where you can query the data direct from Athena or load it to various data warehouses such as BigQuery, RedShift, Snowflake and others. ## Example :configuration The GCS destination utilizes batching, and supports `batch_size`, `batch_size_bytes` and `batch_timeout` options (see below). ## GCS Spec This is the (nested) spec used by the CSV destination Plugin. - `bucket` (`string`) (required) Bucket where to sync the files. - `path` (`string`) (required) Path to where the files will be uploaded in the above bucket, for example `path/to/files/{{TABLE}}/{{UUID}}.parquet`. If no path variables are present, the path will be appended with TABLE, FORMAT and Compression extension by default. The path supports the following placeholder variables: - `{{TABLE}}` will be replaced with the table name - `{{SYNC_ID}}` will be replaced with the unique identifier of the sync. This value is a UUID and is randomly generated for each sync. - `{{FORMAT}}` will be replaced with the file format, such as `csv`, `json` or `parquet`. If compression is enabled, the format will be `csv.gz`, `json.gz` etc. - `{{UUID}}` will be replaced with a random UUID to uniquely identify each file - `{{YEAR}}` will be replaced with the current year in `YYYY` format - `{{MONTH}}` will be replaced with the current month in `MM` format - `{{DAY}}` will be replaced with the current day in `DD` format - `{{HOUR}}` will be replaced with the current hour in `HH` format - `{{MINUTE}}` will be replaced with the current minute in `mm` format - `format` (`string`) (required) Format of the output file. Supported values are `csv`, `json` and `parquet`. - `format_spec` ([format_spec](#format_spec)) (optional) Optional parameters to change the format of the file. - `compression` (`string`) (optional) (default: empty) Compression algorithm to use. Supported values are empty or `gzip`. Not supported for `parquet` format. - `no_rotate` (`boolean`) (optional) (default: `false`) If set to `true`, the plugin will write to one file per table. Otherwise, for every batch a new file will be created with a different `.` suffix. - `batch_size` (`integer`) (optional) (default: `10000`) Number of records to write before starting a new object. - `batch_size_bytes` (`integer`) (optional) (default: `52428800` (50 MiB)) Number of bytes (as Arrow buffer size) to write before starting a new object. - `batch_timeout` (`duration`) (optional) (default: `30s` (30 seconds)) Maximum interval between batch writes. ### format_spec #### CSV - `delimiter` (`string`) (optional) (default: `,`) Delimiter to use in the CSV file. - `skip_header` (`boolean`) (optional) (default: `false`) If set to `true`, the CSV file will not contain a header row as the first row. #### JSON Reserved for future use. #### Parquet - `version` (`string`) (optional) (default: `v2Latest`) Parquet format version to use. Supported values are `v1.0`, `v2.4`, `v2.6` and `v2Latest`. `v2Latest` is an alias for the latest version available in the Parquet library which is currently `v2.6`. Useful when the reader consuming the Parquet files does not support the latest version. - `root_repetition` (`string`) (optional) (default: `repeated`) [Repetition option to use for the root node](https://github.com/apache/arrow/issues/20243). Supported values are `undefined`, `required`, `optional` and `repeated`. Some Parquet readers require a specific root repetition option to be able to read the file. For example, importing Parquet files into [Snowflake](https://www.snowflake.com/en/) requires the root repetition to be `undefined`. - `max_row_group_length` (`integer`) (optional) (default: `134217728` (= 128 * 1024 * 1024)) The maximum number of rows in a single row group. Use a lower number to reduce memory usage when reading the Parquet files, and a higher number to increase the efficiency of reading the Parquet files. ## Authentication :authentication The GCS plugin authenticates using your [Application Default Credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default). Available options are all the same options described [here](https://cloud.google.com/docs/authentication/provide-credentials-adc) in detail: Local Environment: - `gcloud auth application-default login` (recommended when running locally) Google Cloud cloud-based development environment: - When you run on Cloud Shell or Cloud Code credentials are already available. Google Cloud containerized environment: - When running on GKE use [workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity). [Google Cloud services that support attaching a service account](https://cloud.google.com/docs/authentication/provide-credentials-adc#attached-sa): - Services such as Compute Engine, App Engine and functions supporting attaching a user-managed service account which will CloudQuery will be able to utilize. On-premises or another cloud provider - The suggested way is to use [Workload identity federation](https://cloud.google.com/iam/docs/workload-identity-federation) - If not available you can always use service account keys and export the location of the key via `GOOGLE_APPLICATION_CREDENTIALS`. (**Not recommended as long-lived keys are a security risk**) This example configures a GCS destination, to create CSV files in `gcs://bucket_name/path/to/files`. ```yaml copy kind: destination spec: name: "gcs" path: "cloudquery/gcs" registry: "cloudquery" version: "v5.5.11" write_mode: "append" send_sync_summary: true spec: bucket: "bucket_name" path: "path/to/files/{{TABLE}}/{{UUID}}.{{FORMAT}}" format: "parquet" # options: parquet, json, csv format_spec: # CSV specific parameters: # delimiter: "," # skip_header: false # Parquet specific parameters: # version: "v2Latest" # root_repetition: "repeated" # max_row_group_length: 134217728 # 128 * 1024 * 1024 # Optional parameters # compression: "" # options: gzip # no_rotate: false # batch_size: 10000 # batch_size_bytes: 52428800 # 50 MiB # batch_timeout: 30s ``` Note that the GCS plugin only supports `append` `write_mode`. The (top level) spec section is described in the [Destination Spec Reference](https://www.cloudquery.io/docs/cli/integrations/destinations#complete-destination-spec-reference). --- Source: https://www.cloudquery.io/hub/plugins/destination/cloudquery/mongodb/latest/docs # mongodb Destination Integration --- name: MongoDB stage: GA title: MongoDB Destination Plugin description: CloudQuery MongoDB destination plugin documentation --- # MongoDB Destination Plugin :badge This destination plugin lets you sync data from any CloudQuery source to a MongoDB database. Supported database versions: - MongoDB >= 4.4 (The same minimum version supported by the official [Go driver v2](https://github.com/mongodb/mongo-go-driver)) ## Configuration ### Example :configuration :::callout{type="info"} Make sure to use [environment variable substitution](https://www.cloudquery.io/docs/cli/managing-cloudquery/environment-variables) in production instead of committing the credentials to the configuration file directly. ::: The MongoDB destination utilizes batching, and supports [`batch_size`](https://www.cloudquery.io/docs/cli/integrations/destinations#batch_size) and [`batch_size_bytes`](https://www.cloudquery.io/docs/cli/integrations/destinations#batch_size_bytes). ### MongoDB Spec This is the (nested) spec used by the MongoDB destination Plugin. - `connection_string` (`string`) (required) MongoDB URI as described in the official MongoDB [documentation](https://www.mongodb.com/docs/manual/reference/connection-string/). Example connection strings: - `"mongodb://username:password@hostname:port/database"` basic connection - `"mongodb+srv://username:password@cluster.example.com/database"` connecting to a MongoDB Atlas cluster - `"mongodb://localhost:27017/myDatabase?authSource=admin"` specify authentication source - `database` (`string`) (required) Database to sync the data to. - `batch_size` (`integer`) (optional) (default: `1000`) Maximum amount of items that may be grouped together to be written in a single write. - `batch_size_bytes` (`integer`) (optional) (default: `4194304` (= 4 MiB)) Maximum size of items that may be grouped together to be written in a single write. - `aws_credentials` ([aws_credentials](#aws_credentials)) (optional) Optional parameters to enable usage of AWS IAM credentials - `write_retry` ([write_retry](#write_retry)) (optional) Opt-in exponential-backoff retry around each write batch to absorb transient network errors (e.g. `write tcp ...: broken pipe`) that the driver's single built-in retry can't recover from. Disabled by default. ### write_retry Retries are **disabled by default**. Set `max_attempts` >= 2 to enable application-level retries. - `max_attempts` (`integer`) (optional) (default: `1`) Maximum number of write attempts per batch, including the initial attempt. The default of `1` means no application-level retries — the MongoDB driver's own single retry still applies. - `max_backoff` (`duration`) (optional) (default: `"10s"`) Maximum backoff between retry attempts. Initial backoff and jitter use the underlying retry library's defaults. - `use_transactions` (`bool`) (optional) (default: `false`) When `true`, each retried write batch runs inside a MongoDB [transaction](https://www.mongodb.com/docs/manual/core/transactions/) so a retry that follows a partially-applied write rolls back the partial state before re-attempting. This is what makes retries safe on `write_mode: append` tables: without a transaction, an ambiguous failure (server processed the write but the response was lost) can produce duplicate documents, because `_id` is server-generated and the retry has no way to dedupe against the previous attempt. Requires a replica set, sharded cluster, or load-balanced deployment — transactions are not supported on standalone MongoDB, and enabling this against a standalone server will surface a driver error at write time. Has no effect when `max_attempts` is `1`. :::callout{type="warning"} Only enable retries (`max_attempts` >= 2) when the source uses `write_mode: overwrite`, **or** set `use_transactions: true` on a replica-set/sharded deployment. With `write_mode: append` and no transaction, a retry triggered by an ambiguous failure (e.g. the server processed the write but the response was lost) can produce duplicate documents, since `_id` is server-generated and the retry has no way to dedupe against the previous attempt. Overwrite mode is unaffected — writes are performed as upserts keyed on the primary key and are naturally idempotent. ::: ### aws_credentials - `default` (`bool`) (optional) If set to `true` then AWS SDK will use the default credentials based on the AWS Credential chain - `local_profile` (`string`) [Local profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) to use to authenticate this account with. Please note this should be set to the name of the profile. For example, with the following credentials file: ```toml copy [default] aws_access_key_id=xxxx aws_secret_access_key=xxxx [user1] aws_access_key_id=xxxx aws_secret_access_key=xxxx ``` `local_profile` should be set to either `default` or `user1`. - `role_arn` (`string`) If specified will use this to assume role. - `role_session_name` (`string`) If specified will use this session name when assume role to `role_arn`. - `external_id` (`string`) If specified will use this when assuming role to `role_arn`. This example configures a MongoDB destination, located at `localhost:27017`. The (top level) spec section is described in the [Destination Spec Reference](https://www.cloudquery.io/docs/cli/integrations/destinations#complete-destination-spec-reference). ```yaml copy kind: destination spec: name: "mongodb" path: "cloudquery/mongodb" registry: "cloudquery" version: "v3.1.1" send_sync_summary: true spec: # required, a connection string in the format mongodb://localhost:27017 connection_string: "${MONGODB_CONNECTION_STRING}" # required, the name of the database to sync to database: "${MONGODB_DATABASE_NAME}" # Optional parameters: # batch_size: 10000 # 10K # batch_size_bytes: 4194304 # 4 MiB # write_retry: # <- Opt-in retries on transient network errors. Safe with write_mode: overwrite, or with use_transactions on a replica set / sharded cluster. # max_attempts: 5 # Total attempts per batch, including the first. Default 1 (retries disabled). # max_backoff: "10s" # use_transactions: false # Set true to wrap each retried batch in a MongoDB transaction (requires replica set / sharded cluster). # aws_credentials: # <- Use this to specify non-default role assumption parameters # default: true # Use the default credentials chain # local_profile: "mongodb-profile" # Use a local profile instead of the default one # role_arn: "arn:aws:iam::123456789012:role/role_name" # Specify the role to assume # external_id: "external_id" # Used when assuming a role # role_session_name: "session_name" # Used when assuming a role ``` --- Source: https://www.cloudquery.io/hub/plugins/destination/cloudquery/mysql/latest/docs # mysql Destination Integration --- name: MySQL title: MySQL Destination Plugin description: CloudQuery MySQL destination plugin documentation --- # MySQL destination plugin :badge This destination plugin lets you sync data from a CloudQuery source to a MySQL database. Supported database versions are >= 8.0. > If you need support for older versions, please [contact us](https://www.cloudquery.io/pricing) MariaDB is not fully supported. If MariaDB compatibility is needed please upvote the open [GitHub issue](https://github.com/cloudquery/cloudquery/issues/12058) # MySQL destination plugin configuration reference :badge ## Example Configuration :configuration :::callout{type="info"} Make sure you use environment variable expansion in production instead of committing the credentials to the configuration file directly. ::: The MySQL destination utilizes batching, and supports [`batch_size`](https://www.cloudquery.io/docs/cli/integrations/destinations#batch_size) and [`batch_size_bytes`](https://www.cloudquery.io/docs/cli/integrations/destinations#batch_size_bytes). ## MySQL spec This is the (nested) spec used by the MySQL destination plugin. - `connection_string` (`string`) (required) Connection string to connect to the database. See the [Go driver documentation](https://github.com/go-sql-driver/mysql#dsn-data-source-name) for details. - `"user:password@tcp(127.0.0.1:3306)/dbname"` connect with TCP - `"user:password@127.0.0.1:3306/dbname?charset=utf8mb4\u0026parseTime=True\u0026loc=Local"` connect and set character set, time parsing, and location - `"user:password@localhost:3306/dbname?timeout=30s\u0026readTimeout=1s\u0026writeTimeout=1s"` connect and set various timeouts - `"user:password@/dbname?loc=UTC\u0026allowNativePasswords=true\u0026tls=preferred"` connect and set location and native password allowance, and prefer TLS - `batch_size` (`integer`) (optional) (default: `1000`) Maximum number of items that may be grouped together to be written in a single write. - `batch_size_bytes` (`integer`) (optional) (default: `4194304` (= 4 MiB)) Maximum size of items that may be grouped together to be written in a single write. # MySQL destination plugin example :badge In this article we will show you a simple example of configuring MySQL destination plugin. ## Prerequisites - [Docker](https://docs.docker.com/get-docker/) installed and running ### Start MySQL locally ```sh copy docker run -p 3306:3306 --name mysql -e MYSQL_ROOT_PASSWORD=test -e MYSQL_DATABASE=cloudquery -d mysql:latest ``` :::callout{type="info"} For brevity we only set the `MYSQL_ROOT_PASSWORD` and `MYSQL_DATABASE` environment variables. You should create dedicated and secure credentials for production use. For the docker image documentation see [here](https://hub.docker.com/_/mysql). Also, if you're running on Apple silicon based Macs you might need to add the `--platform linux/amd64` flag to the above command. ::: ## Configure MySQL destination plugin Once you've completed the steps from previous sections you should be able to connect to the local `cloudquery` MySQL database via the following connection string: ```text copy root:password@/cloudquery ``` The (top level) spec section is described in the [Destination Spec Reference](https://www.cloudquery.io/docs/cli/integrations/destinations#complete-destination-spec-reference). The full configuration for the MySQL destination plugin should look like this: ```yaml copy kind: destination spec: name: "mysql" path: "cloudquery/mysql" registry: "cloudquery" version: "v5.6.10" send_sync_summary: true spec: connection_string: "root:password@/cloudquery" ``` :::callout{type="info"} Make sure you use [environment variable expansion](https://www.cloudquery.io/docs/cli/managing-cloudquery/environment-variables) in production instead of committing the credentials to the configuration file directly. ::: The MySQL destination (MySQL 8.0 and later) supports most [Apache Arrow](https://arrow.apache.org/docs/index.html) types. The following table shows the supported types and how they are mapped to [MySQL data types](https://dev.mysql.com/doc/refman/8.0/en/data-types.html). :::callout{type="info"} Unsupported types are always mapped to `text`. ::: | Arrow Column Type | Supported? | MySQL Type | |-------------------|------------|-------------------| | Binary | ✅ Yes | `blob` | | Boolean | ✅ Yes | `tinyint(1)` | | Date32 | ✅ Yes | `text` | | Date64 | ✅ Yes | `text` | | Decimal | ✅ Yes | `text` | | Dense Union | ✅ Yes | `text` | | Dictionary | ✅ Yes | `text` | | Duration | ✅ Yes | `text` | | Fixed Size List | ✅ Yes | `text` | | Float16 | ✅ Yes | `text` | | Float32 | ✅ Yes | `float` | | Float64 | ✅ Yes | `double` | | Inet | ✅ Yes | `text` | | Int8 | ✅ Yes | `tinyint` | | Int16 | ✅ Yes | `smallint` | | Int32 | ✅ Yes | `int` | | Int64 | ✅ Yes | `bigint` | | Interval[DayTime] | ✅ Yes | `text` | | Interval[MonthDayNano] | ✅ Yes | `text` | | Interval[Month] | ✅ Yes | `text` | | JSON | ✅ Yes | `json` | | Large Binary | ✅ Yes | `blob` | | Large List | ✅ Yes | `text` | | Large String | ✅ Yes | `text` | | List | ✅ Yes | `json` | | MAC | ✅ Yes | `text` | | Map | ✅ Yes | `text` | | String | ✅ Yes | `text` | | Struct | ✅ Yes | `json` | | Timestamp | ✅ Yes | `datetime(6)` | | UUID | ✅ Yes | `binary(16)` | | Uint8 | ✅ Yes | `tinyint unsigned`| | Uint16 | ✅ Yes | `smallint unsigned`| | Uint32 | ✅ Yes | `int unsigned` | | Uint64 | ✅ Yes | `bigint unsigned` | | Union | ✅ Yes | `text` | ## Notes - Boolean values are stored as `tinyint(1)` to align with MySQL's information schema representation - Timestamps include microsecond precision with `datetime(6)` - Complex data types like Struct and List are stored as JSON when supported natively - All other unsupported types fall back to `text` for maximum compatibility ```yaml copy kind: destination spec: name: "mysql" path: "cloudquery/mysql" registry: "cloudquery" version: "v5.6.10" send_sync_summary: true # Learn more about the configuration options at https://cql.ink/mysql_destination spec: connection_string: "user:password@/dbname" # Optional parameters: # batch_size: 1000 # 1K entries # batch_size_bytes: 4194304 # 4 MiB ``` --- Source: https://www.cloudquery.io/hub/plugins/destination/cloudquery/postgresql/latest/docs # postgresql Destination Integration --- name: PostgreSQL stage: GA title: PostgreSQL Destination Plugin description: CloudQuery PostgreSQL destination plugin documentation --- # PostgreSQL Destination Plugin :badge This destination plugin lets you sync data from a CloudQuery source to a PostgreSQL compatible database. Supported database versions: - PostgreSQL >= v10 - CockroachDB >= v20.2 ## Configuration ### Example :configuration The (top level) spec section is described in the [Destination Spec Reference](https://www.cloudquery.io/docs/cli/integrations/destinations#complete-destination-spec-reference). :::callout{type="info"} Make sure you use environment variable expansion in production instead of committing the credentials to the configuration file directly. ::: The PostgreSQL destination utilizes batching, and supports [`batch_size`](https://www.cloudquery.io/docs/cli/integrations/destinations#batch_size) and [`batch_size_bytes`](https://www.cloudquery.io/docs/cli/integrations/destinations#batch_size_bytes). ### PostgreSQL Spec This is the (nested) spec used by the PostgreSQL destination Plugin. - `connection_string` (`string`) (required) Connection string to connect to the database. This can be a DSN or a URI, as per [`pgxpool`](https://pkg.go.dev/github.com/jackc/pgx/v4/pgxpool#ParseConfig) - `"user=user password=pass+0-[word host=localhost port=5432 dbname=mydb sslmode=disable"` _DSN format_ - `"postgres://user:pass@localhost:5432/mydb?sslmode=prefer"` _connect with tcp and prefer TLS_ - `"postgres://user:pass@localhost:5432/mydb?sslmode=disable&application_name=pgxtest&search_path=myschema&connect_timeout=5"` _be explicit with all options_ - `"postgres://localhost:5432/mydb?sslmode=disable"` _connect with os username cloudquery is being run as_ - `"postgres:///mydb?host=/tmp"` _connect over unix socket_ - `"dbname=mydb"` _unix domain socket, just specifying the db name - useful if you want to use peer authentication_ - `pgx_log_level` (`string`) (optional) (default: `error`) Available: `error`, `warn`, `info`, `debug`, `trace`. Defines what [`pgx`](https://github.com/jackc/pgx) call events should be logged. - `batch_size` (`integer`) (optional) (default: `10000`) Maximum number of items that may be grouped together to be written in a single write. - `batch_size_bytes` (`integer`) (optional) (default: `100000000` (= 100 MB)) Maximum size of items that may be grouped together to be written in a single write. - `batch_timeout` (`duration`) (optional) (default: `60s` (= 60 seconds)) Maximum interval between batch writes. - `create_performance_indexes` (`boolean`) (optional) (default: `false`) Creates indexes on tables that help with performance when using `write_mode: overwrite-delete-stale`. - `lakebase` (`object`) (optional) Configuration to connect to [Databricks Lakebase](https://docs.databricks.com/aws/en/oltp), a PostgreSQL-compatible managed database. When set, the plugin uses the Databricks SDK to generate a short-lived OAuth database credential before each new connection and uses it as the connection password. The `connection_string` still supplies the host, port, database name and user (the service principal client ID), and must use `sslmode=require` (or `verify-ca`/`verify-full`); TLS is required and enforced. See the [Databricks Lakebase example](#databricks-lakebase) below. - `endpoint` (`string`) (required) The Lakebase database endpoint resource name, in the format `projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id}`. - `host` (`string`) (optional) Databricks workspace host, for example `https://your-workspace.cloud.databricks.com`. If empty, the Databricks SDK resolves it from the `DATABRICKS_HOST` environment variable (or other default Databricks configuration sources). - `client_id` (`string`) (optional) Databricks service principal OAuth client ID. If empty, the Databricks SDK resolves it from the `DATABRICKS_CLIENT_ID` environment variable (or other default Databricks configuration sources). - `client_secret` (`string`) (optional) Databricks service principal OAuth client secret. If empty, the Databricks SDK resolves it from the `DATABRICKS_CLIENT_SECRET` environment variable (or other default Databricks configuration sources). - `pgvector_config` (`object`) (optional) Optional configuration to enable PgVector embedding support. Note: source plugin must sync the `_cq_id` column on target tables if this is enabled. - `tables` (`array`) (required) Tables to create embeddings for. For each entry, embeddings are created from a source table and stored in a configured target table. - `source_table_name` (`string`) (required) Name of the source table from which text columns are read to generate embeddings. - `target_table_name` (`string`) (required) Name of the embeddings table to create/populate. This table will contain the `embedding` vector column, a `chunk` text column, and the configured metadata columns. The `_cq_id` column is always included and indexed. - `embed_columns` (`array`) (required) Columns on the source table to concatenate and create embeddings for. - `metadata_columns` (`array`) (optional) These columns will be added as-is from the source table for context. The `_cq_id` column will be added automatically and an index will be created on it. - `text_splitter` (`object`) (optional) Optional text splitting configuration for the embeddings. If unset, defaults are used. - `recursive_text` (`object`) (required) - `chunk_size` (`integer`) (required) Chunk size for the text splitting. - `chunk_overlap` (`integer`) (required) Chunk overlap for the text splitting. - `openai_embedding` (`object`) (required) OpenAI Embedding API configuration. Currently only OpenAI is supported. - `dimensions` (`integer`) (required) The number of dimensions to use for the embeddings. For `text-embedding-3-small`, this is 1536. For `text-embedding-3-large`, this is 3072. - `api_key` (`string`) (required) The OpenAI API key to use for the embedding API. - `model_name` (`string`) (required) The model name to use for the embedding API. Currently, `text-embedding-3-small` and `text-embedding-3-large` are supported. - `retry_on_deadlock` (`integer`) (optional) (default: `0`) Number of times to retry a transaction if a deadlock is detected by Postgres (Postgres error code `40P01`). ### Databricks Lakebase [Databricks Lakebase](https://docs.databricks.com/aws/en/oltp) is a PostgreSQL-compatible managed database. To sync into it, set the `lakebase` block. The plugin authenticates with a Databricks service principal (OAuth M2M) and mints a fresh short-lived database credential for every new connection, so you do not put a password in the `connection_string`. The connection string supplies the Lakebase host, port, database, user (the service principal client ID) and must use `sslmode=require`. ```yaml copy kind: destination spec: name: postgresql path: cloudquery/postgresql registry: cloudquery version: "v8.15.1" write_mode: "overwrite-delete-stale" send_sync_summary: true spec: # No password in the connection string - it is generated per-connection from the Databricks credentials below. connection_string: "host=${PGHOST} port=5432 dbname=databricks_postgres user=${DATABRICKS_CLIENT_ID} sslmode=require" lakebase: # projects/{project_id}/branches/{branch_id}/endpoints/{endpoint_id} endpoint: "${LAKEBASE_ENDPOINT_NAME}" host: "${DATABRICKS_HOST}" client_id: "${DATABRICKS_CLIENT_ID}" client_secret: "${DATABRICKS_CLIENT_SECRET}" ``` `host`, `client_id` and `client_secret` are optional. If omitted, the Databricks SDK resolves them from the standard `DATABRICKS_HOST`, `DATABRICKS_CLIENT_ID` and `DATABRICKS_CLIENT_SECRET` environment variables (or other [default Databricks configuration sources](https://docs.databricks.com/aws/en/dev-tools/auth)). ### Verbose logging for debug The PostgreSQL destination can be run in debug mode. Note: This will use [`pgx`](https://github.com/jackc/pgx) built-in logging and might output data/sensitive information to logs so make sure to not use it in production but only for debugging. ```yaml copy kind: destination spec: name: postgresql path: cloudquery/postgresql registry: cloudquery version: "v8.15.1" send_sync_summary: true spec: connection_string: ${PG_CONNECTION_STRING} pgx_log_level: debug # Available: error, warn, info, debug, trace. Default: "error" ``` In order to store information in a PostgreSQL database, cloudquery needs to be authenticated. Insert the PostgreSQL connection string in the `connection_string` field in the sync config file. Acceptable formats are: - URI: `postgres://postgres:pass@localhost:5432/postgres?sslmode=disable`. Any special URI characters need to be percent-encoded. - DSN: `"user=postgres password=pass+0-[word host=localhost port=5432 dbname=postgres sslmode=disable"` The PostgreSQL destination supports most [Apache Arrow](https://arrow.apache.org/docs/index.html) types. The following table shows the supported types and how they are mapped to [PostgreSQL data types](https://www.postgresql.org/docs/current/datatype.html). :::callout{type="info"} Unsupported types are converted to text using their string representation. ::: | Arrow Column Type | Supported? | PostgreSQL Type | |------------------------|------------|-----------------| | Binary | ✅ Yes | `bytea` | | Boolean | ✅ Yes | `boolean` | | Date32 | ✅ Yes | `date` | | Date64 | ✅ Yes | `date` | | Decimal | ✅ Yes | `text` | | Dense Union | ✅ Yes | `text` | | Dictionary | ✅ Yes | `text` | | Duration | ✅ Yes | `text` | | Fixed Size List | ✅ Yes | `text` | | Float16 | ✅ Yes | `text` | | Float32 | ✅ Yes | `real` | | Float64 | ✅ Yes | `double precision` | | Inet | ✅ Yes | `inet` | | Int8 | ✅ Yes | `smallint` | | Int16 | ✅ Yes | `smallint` | | Int32 | ✅ Yes | `integer` | | Int64 | ✅ Yes | `bigint` | | Interval[DayTime] | ✅ Yes | `text` | | Interval[MonthDayNano] | ✅ Yes | `text` | | Interval[Month] | ✅ Yes | `text` | | JSON | ✅ Yes | `jsonb` | | Large Binary | ✅ Yes | `bytea` | | Large List | ✅ Yes | Array of element type | | Large String | ✅ Yes | `text` | | List | ✅ Yes | Array of element type | | MAC | ✅ Yes | `text` | | Map | ✅ Yes | `text` | | String | ✅ Yes | `text` | | Struct | ✅ Yes | `text` | | Time32 | ✅ Yes | `time without time zone` | | Time64 | ✅ Yes | `time without time zone` | | Timestamp | ✅ Yes | `timestamp without time zone` | | UUID | ✅ Yes | `uuid` | | Uint8 | ✅ Yes | `smallint` | | Uint16 | ✅ Yes | `integer` | | Uint32 | ✅ Yes | `bigint` | | Uint64 | ✅ Yes | `numeric(20,0)` | | Union | ✅ Yes | `text` | ## Notes - Null characters (`\x00`) are automatically stripped from string values for PostgreSQL compatibility - JSON data is stored as `jsonb` with null characters stripped from string values - List types are converted to PostgreSQL arrays with recursive transformation - Time values are stored with microsecond precision - Timestamps are stored as `timestamp without time zone` in UTC :::callout{type="info"} For CrateDB compatibility, Uint64 values are stored as strings instead of numeric ::: This example configures a Postgresql destination using an environment variable called `POSTGRESQL_CONNECTION_STRING`: ```yaml copy kind: destination spec: name: "postgresql" path: "cloudquery/postgresql" registry: "cloudquery" version: "v8.15.1" write_mode: "overwrite-delete-stale" send_sync_summary: true # Learn more about the configuration options at https://cql.ink/postgresql_destination spec: # set the environment variable in DSN format like "user=postgres password=pass+0-[word host=localhost port=5432 dbname=postgres sslmode=disable" # you can also specify it in URI format like "postgres://postgres:pass@localhost:5432/postgres?sslmode=disable". any special URI characters need to be percent-encoded connection_string: "${POSTGRESQL_CONNECTION_STRING}" # Optional parameters: # pgx_log_level: error # batch_size: 10000 # 10K entries # batch_size_bytes: 100000000 # 100 MB # batch_timeout: 60s # create_performance_indexes: false #create indexes that help with performance when using `write_mode: overwrite-delete-stale` ``` --- Source: https://www.cloudquery.io/hub/plugins/destination/cloudquery/snowflake/latest/docs # snowflake Destination Integration --- name: Snowflake title: Snowflake Destination Plugin description: CloudQuery Snowflake destination plugin documentation --- # Snowflake Destination Plugin :badge The snowflake plugin helps you sync data to your Snowflake data warehouse. There are two ways to sync data to Snowflake: 1. Direct (easy but not recommended for production or large data sets): This is the default mode of operation where CQ plugin will stream the results directly to the Snowflake database. There is no additional setup needed apart from authentication to Snowflake. 2. Loading via CSV/JSON from a remote storage: This is the standard way of loading data into Snowflake, it is recommended for production and large data sets. This mode requires a remote storage (e.g. S3, GCS, Azure Blob Storage) and a Snowflake stage to be created. The CQ plugin will stream the results to the remote storage. You can then load those files via a cronjob or via SnowPipe. This method is still in the works and will be updated soon with a guide. ## Example Config :configuration The Snowflake destination utilizes batching, and supports [`batch_size`](https://www.cloudquery.io/docs/cli/integrations/destinations#batch_size) and [`batch_size_bytes`](https://www.cloudquery.io/docs/cli/integrations/destinations#batch_size_bytes). ## Authentication Authentication of the connection to Snowflake can be specified using: * A username and password in the DSN: ```yaml kind: destination spec: name: snowflake send_sync_summary: true ... spec: connection_string: "user:pass@account/db/schema?warehouse=wh" ``` * A private key inline: ```yaml kind: destination spec: name: snowflake send_sync_summary: true ... spec: connection_string: "user@account/database/schema?warehouse=wh" private_key: | -----BEGIN PRIVATE KEY----- MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC2ajPRIbPtbxZ1 3DONLA02eZJuCzsgIkBWov/Me5TL6cKN0gnY+mbA8OnNCH+9HSzgiU9P8XhTUrIN 85diD+rj6uK+E0sSyxGk6HG17TyR5oBq8nz2hbZlbaNi/HO9qYoHQgAgMq908YBz ... DUmOIrBYEMf2nDTlTu/QVcKb -----END PRIVATE KEY----- ``` * A private key included from a file: ```yaml kind: destination spec: name: snowflake send_sync_summary: true ... spec: connection_string: "user@account/database/schema?warehouse=wh" private_key: "${file:./private.key}" ``` where ./private.key is PEM-encoded private key file with contents of the form: ```txt -----BEGIN PRIVATE KEY----- MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC2ajPRIbPtbxZ1 3DONLA02eZJuCzsgIkBWov/Me5TL6cKN0gnY+mbA8OnNCH+9HSzgiU9P8XhTUrIN 85diD+rj6uK+E0sSyxGk6HG17TyR5oBq8nz2hbZlbaNi/HO9qYoHQgAgMq908YBz ... DUmOIrBYEMf2nDTlTu/QVcKb -----END PRIVATE KEY----- ``` * OAuth authentication when running in Snowpark container service ```yaml kind: destination spec: name: snowflake send_sync_summary: true ... spec: connection_string: "user:pass@account/db/schema?warehouse=wh&authenticator=oauth&token=token" ``` ### Private Key Authentication Setup The Snowflake guide for [Key Pair Authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth) demonstrates how to create an RSA private key with the ability to authenticate as a Snowflake user. Note that encrypted private keys are not supported by the Snowflake Go SQL driver, and hence not supported by the CloudQuery Snowflake plugin. You can decrypt a private key in file enc.key and store it in a file dec.key using the following command: ```bash openssl pkcs8 -topk8 -nocrypt -in enc.key -out dec.key ``` ## Snowflake Spec This is the top level spec used by the Snowflake destination plugin. - `connection_string` (`string`) (required) Snowflake `connection_string`. Example: ```yaml copy # user[:password]@account/database/schema?warehouse=user_warehouse[¶m1=value1¶mN=valueN] # or # user[:password]@account/database?warehouse=user_warehouse[¶m1=value1¶mN=valueN] # or # user[:password]@host:port/database/schema?account=user_account&warehouse=user_warehouse[¶m1=value1¶mN=valueN] # or # host:port/database/schema?account=user_account&warehouse=user_warehouse[¶m1=value1¶mN=valueN] ``` From Snowflake documentation: `account` - Name assigned to your Snowflake account. If you are not on us-west-2 or AWS deployment, append the region and platform to the end, e.g., `. or ..`. - `private_key` (`string`) (optional) A PEM-encoded private key for connecting to snowflake. Equivalent to adding `authenticator=snowflake_jwt&privateKey=...` to the `connection_string` but parses, validates, and correctly encodes the key for use with snowflake. - `migrate_concurrency` (`integer`) (optional) (default: `1`) By default, tables are migrated one at a time. This option allows you to migrate multiple tables concurrently. This can be useful if you have a lot of tables to migrate and want to speed up the process. Setting this to a negative number means no limit. - `batch_size` (`integer`) (optional) (default: `1000`) Number of records to batch together before sending to the database. - `batch_size_bytes` (`integer`) (optional) (default: `4194304` (= 4 MiB)) Number of bytes (as Arrow buffer size) to batch together before sending to the database. - `leave_stage_files` (boolean) (optional) (default: false) If set to true, intermediary files used to load data to the Snowflake stage are left in the temp directory. This can be useful for debugging purposes. ## Underlying library We use the official [github.com/snowflakedb/gosnowflake](https://github.com/snowflakedb/gosnowflake) package for database connection. The Snowflake destination supports most [Apache Arrow](https://arrow.apache.org/docs/index.html) types. The following table shows the supported types and how they are mapped to [Snowflake data types](https://docs.snowflake.com/en/sql-reference/data-types). :::callout{type="info"} Unsupported types are always mapped to `text`. ::: | Arrow Column Type | Supported? | Snowflake Type | |------------------------|------------|----------------| | Binary | ✅ Yes | `binary` | | Boolean | ✅ Yes | `boolean` | | Date32 | ✅ Yes | `text` | | Date64 | ✅ Yes | `text` | | Decimal | ✅ Yes | `text` | | Dense Union | ✅ Yes | `text` | | Dictionary | ✅ Yes | `text` | | Duration | ✅ Yes | `text` | | Fixed Size List | ✅ Yes | `array` | | Float16 | ✅ Yes | `text` | | Float32 | ✅ Yes | `float` | | Float64 | ✅ Yes | `float` | | Inet | ✅ Yes | `text` | | Int8 | ✅ Yes | `number` | | Int16 | ✅ Yes | `number` | | Int32 | ✅ Yes | `number` | | Int64 | ✅ Yes | `number` | | Interval[DayTime] | ✅ Yes | `text` | | Interval[MonthDayNano] | ✅ Yes | `text` | | Interval[Month] | ✅ Yes | `text` | | JSON | ✅ Yes | `variant` | | Large Binary | ✅ Yes | `binary` | | Large List | ✅ Yes | `array` | | Large String | ✅ Yes | `text` | | List | ✅ Yes | `array` | | MAC | ✅ Yes | `text` | | Map | ✅ Yes | `text` | | String | ✅ Yes | `text` | | Struct | ✅ Yes | `variant` | | Time32 | ✅ Yes | `text` | | Time64 | ✅ Yes | `text` | | Timestamp | ✅ Yes | `timestamp_tz` | | UUID | ✅ Yes | `text` | | Uint8 | ✅ Yes | `number` | | Uint16 | ✅ Yes | `number` | | Uint32 | ✅ Yes | `number` | | Uint64 | ✅ Yes | `number` | | Union | ✅ Yes | `text` | ## Notes - All integer types (signed and unsigned) are mapped to Snowflake's `number` type, which can handle arbitrary precision - Both Float32 and Float64 are stored as Snowflake's `float` type - List types are stored as Snowflake `array` type, including both regular and fixed-size lists - JSON and Struct types use Snowflake's `variant` type for semi-structured data storage - Timestamps are stored as `timestamp_tz` (timestamp with timezone) - Complex types not natively supported by Snowflake fall back to `text` representation This example sets the connection string to a value read from the `SNOWFLAKE_CONNECTION_STRING` environment variable: ```yaml copy kind: destination spec: name: snowflake path: cloudquery/snowflake registry: cloudquery version: "v5.2.12" write_mode: "append" send_sync_summary: true # Learn more about the configuration options at https://cql.ink/snowflake_destination spec: connection_string: "${SNOWFLAKE_CONNECTION_STRING}" # Optional parameters # migrate_concurrency: 1 # batch_size: 1000 # 1K entries # batch_size_bytes: 4194304 # 4 MiB ``` --- Source: https://www.cloudquery.io/hub/plugins/source/cloudquery/aws/latest/docs # aws Source Integration The AWS Source plugin extracts information from many of the supported services by Amazon Web Services (AWS) and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more). --- hub-title: Authentication hub-order: 1 --- ## Quickstart The plugin needs to be authenticated with your AWS account(s) in order to sync information from your cloud setup. If you are running CloudQuery CLI locally, and have AWS CLI installed, [sign in with AWS CLI](https://docs.aws.amazon.com/signin/latest/userguide/command-line-sign-in.html). Test that your AWS CLI is working: ```shell aws ec2 describe-regions ``` ### Non-interactive Authentication The plugin requires only _read_ permissions (we will never make any changes to your cloud setup), so following the principle of the least privilege, it's recommended to grant it read-only permissions. There are multiple ways to authenticate with AWS, and the plugin respects the AWS credential provider chain. This means that AWS plugin will follow the following priorities when attempting to authenticate: 1. The `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` environment variables - see the [AWS guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). 2. The `credentials` and `config` files in `~/.aws` (the `credentials` file takes priority) 3. A session created using the `aws sso` to authenticate the plugin - see [Configuring IAM Identity Center authentication with the AWS CLI ](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html) 4. IAM roles for AWS compute resources (including EC2 instances, Fargate and ECS containers) For details about configuring the individual authentication options, see [Auth Details](#auth-details). --- hub-title: Auth Details hub-order: 2 --- ### Required Permissions In order for the plugin to discover all enabled regions, the credentials must have the `ec2:DescribeRegions` permission. In the credentials, you must specify the exact regions that it should sync data from. The specific permissions for each table is documented in the table documentation tab. You can generate custom policies based on the required permissions for each table by following the steps in this [blog](https://www.cloudquery.io/blog/how-to-limit-cq-access-to-aws-accounts). ### Environment Variables AWS plugin can use the credentials from the `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` environment variables (`AWS_SESSION_TOKEN` can be optional for some accounts). For information on obtaining credentials, see the [AWS guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). To export the environment variables (On Linux/Mac - similar for Windows): ```bash copy export AWS_ACCESS_KEY_ID={Your AWS Access Key ID} export AWS_SECRET_ACCESS_KEY={Your AWS secret access key} export AWS_SESSION_TOKEN={Your AWS session token} ``` ### Shared Configuration files The plugin can use credentials from your `credentials` and `config` files in the `.aws` directory in your home folder. The contents of these files are practically interchangeable, but AWS plugin will prioritize credentials in the `credentials` file. For information about obtaining credentials, see the [AWS guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html). Here are example contents for a `credentials` file: ```toml copy filename="~/.aws/credentials" [default] aws_access_key_id = YOUR_ACCESS_KEY_ID aws_secret_access_key = YOUR_SECRET_ACCESS_KEY ``` You can also specify credentials for a different profile, and instruct the plugin to use the credentials from this profile instead of the default one. For example: ```toml copy filename="~/.aws/credentials" [myprofile] aws_access_key_id = YOUR_ACCESS_KEY_ID aws_secret_access_key = YOUR_SECRET_ACCESS_KEY ``` Then, you can either export the `AWS_PROFILE` environment variable (On Linux/Mac, similar for Windows): ```bash copy export AWS_PROFILE=myprofile ``` or, configure your desired profile in the `local_profile` field: ```yaml copy filename="aws.yml" accounts: id: local_profile: myprofile ``` ### IAM Roles for AWS Compute Resources The plugin can use IAM roles for AWS compute resources (including EC2 instances, Fargate and ECS containers). If you configured your AWS compute resources with IAM, the plugin will use these roles automatically. For more information on configuring IAM, see the AWS docs [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) and [here](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html). ### User Credentials with MFA In order to leverage IAM User credentials with MFA, `aws sts get-session-token` command may be used with the IAM User's long-term security credentials (Access Key and Secret Access Key). For more information, see [here](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sts/get-session-token.html). ```bash copy aws sts get-session-token --serial-number --token-code --duration-seconds 3600 ``` Then export the temporary credentials to your environment variables. ```bash copy export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export AWS_SESSION_TOKEN= ``` --- hub-title: Configuration --- ### Examples #### Basic example ```yaml copy kind: source spec: # Source spec section name: aws path: cloudquery/aws registry: cloudquery version: "v33.33.1" tables: ["aws_s3_buckets"] destinations: ["DESTINATION_NAME"] # Learn more about the configuration options at https://cql.ink/aws_source spec: # Optional parameters # regions: [] # skip_regions: [] # accounts: [] # org: nil # concurrency: 50000 # initialization_concurrency: 4 # aws_debug: false # max_retries: 10 # max_backoff: 30 # custom_endpoint_url: "" # custom_endpoint_hostname_immutable: nil # required when custom_endpoint_url is set # custom_endpoint_partition_id: "" # required when custom_endpoint_url is set # custom_endpoint_signing_region: "" # required when custom_endpoint_url is set # use_paid_apis: false # table_options: nil # scheduler: shuffle # options are: dfs, round-robin, shuffle, or shuffle-queue # use_nested_table_rate_limiting: false # enable_api_level_tracing: false ``` #### AWS Organization Example The AWS plugin supports discovery of AWS Accounts via AWS Organizations. This means that as Accounts get added or removed from your organization the plugin will be able to handle new or removed accounts without any configuration changes. ```yaml copy kind: source spec: name: aws path: cloudquery/aws registry: cloudquery version: "v33.33.1" tables: ['aws_s3_buckets'] destinations: ["DESTINATION_NAME"] spec: aws_debug: false org: admin_account: local_profile: "" member_role_name: OrganizationAccountAccessRole regions: - '*' # Optional parameters # regions: [] # skip_regions: [] # accounts: [] # org: nil # concurrency: 50000 # initialization_concurrency: 4 # aws_debug: false # max_retries: 10 # max_backoff: 30 # custom_endpoint_url: "" # custom_endpoint_hostname_immutable: nil # required when custom_endpoint_url is set # custom_endpoint_partition_id: "" # required when custom_endpoint_url is set # custom_endpoint_signing_region: "" # required when custom_endpoint_url is set # use_paid_apis: false # table_options: nil # scheduler: shuffle # options are: dfs, round-robin or shuffle ``` ### Configuration spec This is the (nested) spec used by the AWS source plugin. - `regions` (`[]string`) (default: `[]`. Will use all enabled regions) Regions to use. - `skip_regions` (`[]string`) (default: `[]`) Regions to skip. These regions will be excluded after region discovery is complete. This is useful when using `*` for regions but wanting to exclude specific ones. - `accounts` ([`[]Account`](#account)) (default: current account) List of all accounts to fetch information from. - `org` ([`Org`](#org)) (default: not used) In AWS organization mode, the plugin will source all accounts underneath automatically. - `concurrency` (`integer`) (default: `50000`) The best effort maximum number of Go routines to use. Lower this number to reduce memory usage. - `initialization_concurrency` (`integer`) (default: `4`) During initialization the AWS source plugin fetches information about each account and region. This setting controls how many accounts can be initialized concurrently. Only configurations with many accounts (either hardcoded or discovered via Organizations) should require modifying this setting, to either lower it to avoid rate limit errors, or to increase it to speed up the initialization process. - `scheduler` (`string`) (default: `shuffle`): The scheduler to use when determining the priority of resources to sync. Supported values are `dfs` (depth-first search), `round-robin`, `shuffle` and `shuffle-queue`. For more information about this, see [performance tuning](https://www.cloudquery.io/docs/advanced-topics/performance-tuning). - `aws_debug` (`boolean`) (default: `false`) If `true`, will log AWS debug logs, including retries and other request/response metadata. - `max_retries` (`integer`) (default: `10`) Defines the maximum number of times an API request will be retried. - `max_backoff` (`integer` in seconds) (default: `30` meaning `30s`) Defines the maximum duration (in seconds) between retry attempts. - `use_nested_table_rate_limiting` (`boolean`) (default: `false`) If `true`, the plugin will limit the number of nested tables that are synced concurrently. - `enable_api_level_tracing` (`boolean`) (default: `false`) If `true`, the plugin will extend table level traces to include API requests to AWS Services - `custom_endpoint_url` (`string`) (default: not used) The base URL endpoint the SDK API clients will use to make API calls to. The SDK will suffix URI path and query elements to this endpoint - `custom_endpoint_hostname_immutable` (`boolean`) (default: not used) Specifies if the endpoint's hostname can be modified by the SDK's API client. When using something like LocalStack make sure to set it equal to `true`. - `custom_endpoint_partition_id` (`string`) (default: not used) The AWS partition the endpoint belongs to. - `custom_endpoint_signing_region` (`string`) (default: not used) The region that should be used for signing the request to the endpoint. - `use_paid_apis` (`boolean`) (default: `false`) When set to `true` plugin will sync data from APIs that incur a fee. Tables that require this setting to be set to `true` include (but not limited to): - `aws_costexplorer*` - `aws_cloudwatch_metric*` - `skip_specific_apis` (`map`) (default: not used) This feature enables users to skip specific APIs where ever it is used. In cases where the skipped API call is enriching data from a `List` call, the plugin will persist data from the `List` call and skip the enriching API call. The format of the `skip_specific_apis` object is as follows: ```yaml skip_specific_apis: : : true ``` An example of the `skip_specific_apis` object is as follows: ```yaml spec: regions: ["us-east-1","us-east-2"] skip_specific_apis: lambda: GetRuntimeManagementConfig: true GetFunction: true ``` The following Services and API Actions are supported: ```yaml lambda: GetRuntimeManagementConfig GetFunction GetFunctionCodeSigningConfig GetFunctionConcurrency kms: DescribeKey GetKeyRotationStatus ListResourceTags ssm: ListTagsForResource glacier: ListTagsForVault wafv2: ListResourcesForWebACL ``` - `table_options` (`map`) (default: not used) Table options is a premium feature. Even if some tables are free, syncing data for them (& their relations) using table options counts towards paid usage. Please refer to the [Table Options documentation](#table-options) for more information. - `event_based_sync` ([`[]Event-based sync`](#event_based_sync)) (default: empty) Event-based sync is a premium feature. Even if some tables are free, syncing data for them (and their relations) using event-based sync counts towards paid usage. ### Account This is used to specify one or more accounts to extract information from. - `account_name` (`string`) (optional) (default: empty) Account name. Will be used as an alias in the source plugin and in the logs. - `local_profile` (`string`) (default: will use current credentials) [Local profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) to use to authenticate this account with. Please note this should be set to the name of the profile. For example, with the following credentials file: ```toml copy [default] aws_access_key_id=xxxx aws_secret_access_key=xxxx [user1] aws_access_key_id=xxxx aws_secret_access_key=xxxx ``` `local_profile` should be set to either `default` or `user1`. - `role_arn` (`string`) If specified will use this to assume role. - `role_session_name` (`string`) If specified will use this session name when assume role to `role_arn`. - `external_id` (`string`) If specified will use this when assuming role to `role_arn`. - `default_region` (`string`) (default: `us-east-1`) If specified, this region will be used as the default region for the account. - `regions` (`[]string`) Regions to use for this account. Defaults to global `regions` setting. ### org - `admin_account` ([`Account`](#account)) Configuration for how to grab credentials from an Admin account. - `member_trusted_principal` ([`Account`](#account)) Configuration for how to specify the principle to use in order to assume a role in the member accounts. - `member_role_name` (`string`) (required) Role name that the plugin should use to assume a role in the member account from the admin account. Note: This is not a full ARN, it is just the name. - `member_role_session_name` (`string`) Overrides the default session name. - `member_external_id` (`string`) Specify an external ID for use in the trust policy. - `member_regions` (`[]string`) Limit fetching resources within this specific account to only these regions. This will override any regions specified in the provider block. You can specify all regions by using the `*` character as the only argument in the array. - `organization_units` (`[]string`) List of Organizational Units that AWS plugin should use to source accounts from. If you specify an OU, the plugin will also traverse nested OUs. - `skip_organization_units` (`[]string`) List of Organizational Units to skip. This is useful in conjunction with `organization_units` if there are child OUs that should be ignored. - `skip_member_accounts` (`[]string`) List of OU member accounts to skip. This is useful if there are accounts under the selected OUs that should be ignored. - `account_name_filter` (`string`) A regex filter to apply to the account names. This is useful if you want to only include accounts that match a specific pattern. - `error_if_no_accounts` (`boolean`) (default: `false`) If set to `true`, CloudQuery will return an error if no active accounts are found in the organization. By default, CloudQuery will continue without error if no accounts match the org configuration. This is useful for validation purposes to ensure your organization configuration is correct and hasn't accidentally filtered out all accounts. ### Event-based sync Event-based sync is a premium feature. Even if some tables are free, syncing data for them (& their relations) using event-based sync counts towards paid usage. - `kinesis_stream_arn` (`string`) (required if `sqs_queue_url` is not provided) ARN for the Kinesis stream that will hold all the CloudTrail records. - `sqs_queue_url` (`string`) (required if `kinesis_stream_arn` is not provided) URL for the SQS queue that will hold the S3 Bucket Notifications. - `account` ([`Account`](#account)) Configuration for the credentials that will be used to grab records from the specified Kinesis Stream. If this is not specified the default credentials will be used. - `start_time` (`string` for RFC 3339 timestamp) (default: the time at which the sync began) Defines the place in the stream where record processing should begin. The value should follow the RFC 3339 format, for example: `2023-09-04T19:24:14Z`. - `full_sync` (`boolean`) (default: `true`) By default, AWS plugin will do a full sync on the specified tables before starting to consume the events in the stream. This parameter enables users to skip the full pull based sync and go straight to the event based sync. ### Skip Tables AWS has tables that may contain many resources, nested information, and AWS-provided data. These tables may cause certain syncs to be slow due to the amount of AWS-provided data and may not be needed. We recommend only specifying syncing from necessary tables. If `*` is necessary for tables, below is a reference configuration of skip tables, where certain tables are skipped. ```yaml kind: source spec: # Source spec section name: aws path: cloudquery/aws registry: cloudquery version: "v33.33.1" tables: ["*"] skip_tables: - aws_cloudtrail_events - aws_cloudwatchlogs_log_streams - aws_docdb_cluster_parameter_groups - aws_docdb_engine_versions - aws_ec2_instance_types - aws_ec2_vpc_endpoint_services - aws_elasticache_engine_versions - aws_elasticache_parameter_groups - aws_elasticache_reserved_cache_nodes_offerings - aws_elasticache_service_updates - aws_elasticbeanstalk_platform_versions - aws_elasticsearch_versions - aws_emr_release_labels - aws_emr_supported_instance_types - aws_iam_group_last_accessed_details - aws_iam_policy_last_accessed_details - aws_iam_role_last_accessed_details - aws_iam_user_last_accessed_details - aws_neptune_cluster_parameter_groups - aws_neptune_db_parameter_groups - aws_rds_cluster_parameter_groups - aws_rds_db_parameter_groups - aws_rds_engine_versions - aws_servicequotas_quota_utilizations - aws_servicequotas_services - aws_stepfunctions_map_run_executions - aws_stepfunctions_map_runs destinations: ["postgresql"] spec: # AWS Spec section described below ``` --- hub-title: Event-based sync --- :::callout{type="info"} Event-based sync is a premium feature. Even if some tables are free, syncing data for them (and their relations) using event-based sync counts towards paid usage. ::: Event based sync is a type of a long running sync that enables syncing only resources that need to be synced based on the incoming events. By configuring the AWS plugin to listen to the supported events, the plugin will trigger selective syncs to update only the resources that had a configuration change. ### How it works The plugin supports two event sources: 1. **CloudTrail events** (most services): AWS CloudTrail provides an audit log of API calls. These events are delivered to Kinesis via CloudWatch Logs. 2. **EventBridge events** (AWS Health): Some services deliver events directly via Amazon EventBridge rather than CloudTrail. These are routed to the same Kinesis stream using an EventBridge rule. There are two ways to consume CloudTrail events: 1. The fastest and lowest latency is subscribing to a stream of CloudTrail events in a Kinesis Data stream. 2. Alternatively, if you are already using CloudTrail and persisting the logs in an S3 bucket, you can configure CloudQuery to grab the data from the S3 bucket by using Event Notifications to subscribe to events that indicate a new batch of logs has been written. :::callout{type="warning"} The S3-based approach only supports CloudTrail events. EventBridge-based events (such as AWS Health) require the Kinesis Data Stream approach. ::: ### Supported Services and Events Each table in the supported list is a top level table. When an event is received for a table, all child tables are re-synced too by default. To skip some child tables you can use `skip_tables`. #### EC2 | Service | Event | Plugin table | | ------------------- | ------------------------------- | ---------------------------- | | `ec2.amazonaws.com` | `AssociateRouteTable` | `aws_ec2_route_tables` | | `ec2.amazonaws.com` | `AttachInternetGateway` | `aws_ec2_internet_gateways` | | `ec2.amazonaws.com` | `AuthorizeSecurityGroupEgress` | `aws_ec2_security_groups` | | `ec2.amazonaws.com` | `AuthorizeSecurityGroupIngress` | `aws_ec2_security_groups` | | `ec2.amazonaws.com` | `CreateImage` | `aws_ec2_images` | | `ec2.amazonaws.com` | `CreateInternetGateway` | `aws_ec2_internet_gateways` | | `ec2.amazonaws.com` | `CreateNetworkInterface` | `aws_ec2_network_interfaces` | | `ec2.amazonaws.com` | `CreateSecurityGroup` | `aws_ec2_security_groups` | | `ec2.amazonaws.com` | `CreateSubnet` | `aws_ec2_subnets` | | `ec2.amazonaws.com` | `CreateTags` | `aws_ec2_instances` | | `ec2.amazonaws.com` | `CreateVpc` | `aws_ec2_vpcs` | | `ec2.amazonaws.com` | `DeleteTags` | `aws_ec2_instances` | | `ec2.amazonaws.com` | `DeleteInternetGateway` | `aws_ec2_internet_gateways` | | `ec2.amazonaws.com` | `DeleteNetworkInterface` | `aws_ec2_network_interfaces` | | `ec2.amazonaws.com` | `DeleteRouteTable` | `aws_ec2_route_tables` | | `ec2.amazonaws.com` | `DeleteInternetGateway` | `aws_ec2_internet_gateways` | | `ec2.amazonaws.com` | `DeleteSubnet` | `aws_ec2_subnets` | | `ec2.amazonaws.com` | `DeleteVpc` | `aws_ec2_vpcs` | | `ec2.amazonaws.com` | `DeregisterImage` | `aws_ec2_images` | | `ec2.amazonaws.com` | `DetachInternetGateway` | `aws_ec2_internet_gateways` | | `ec2.amazonaws.com` | `ModifySubnetAttribute` | `aws_ec2_subnets` | | `ec2.amazonaws.com` | `RevokeSecurityGroupIngress` | `aws_ec2_security_groups` | | `ec2.amazonaws.com` | `RevokeSecurityGroupEgress` | `aws_ec2_security_groups` | | `ec2.amazonaws.com` | `RunInstances` | `aws_ec2_instances` | | `ec2.amazonaws.com` | `TerminateInstances` | `aws_ec2_instances` | | `ec2.amazonaws.com` | `StartInstances` | `aws_ec2_instances` | | `ec2.amazonaws.com` | `StopInstances` | `aws_ec2_instances` | #### IAM | Service | Event | Plugin table | | ------------------- |-------------------------|--------------------| | `iam.amazonaws.com` | `CreateGroup` | `aws_iam_groups` | | `iam.amazonaws.com` | `CreateRole` | `aws_iam_roles` | | `iam.amazonaws.com` | `CreateUser` | `aws_iam_users` | | `iam.amazonaws.com` | `DeleteGroup` | `aws_iam_groups` | | `iam.amazonaws.com` | `DeleteRole` | `aws_iam_roles` | | `iam.amazonaws.com` | `DeleteUser` | `aws_iam_users` | | `iam.amazonaws.com` | `CreatePolicy` | `aws_iam_policies` | | `iam.amazonaws.com` | `CreatePolicyVersion` | `aws_iam_policies` | | `iam.amazonaws.com` | `DeletePolicy` | `aws_iam_policies` | | `iam.amazonaws.com` | `DeletePolicyVersion` | `aws_iam_policies` | | `iam.amazonaws.com` | `TagRole` | `aws_iam_roles` | | `iam.amazonaws.com` | `TagUser` | `aws_iam_users` | | `iam.amazonaws.com` | `UntagRole` | `aws_iam_roles` | | `iam.amazonaws.com` | `UntagUser` | `aws_iam_users` | | `iam.amazonaws.com` | `UpdateGroup` | `aws_iam_groups` | | `iam.amazonaws.com` | `UpdateRole` | `aws_iam_roles` | | `iam.amazonaws.com` | `UpdateRoleDescription` | `aws_iam_roles` | | `iam.amazonaws.com` | `UpdateUser` | `aws_iam_users` | #### RDS | Service | Event | Plugin table | | ------------------- | ------------------------------- | ---------------------------- | | `rds.amazonaws.com` | `CreateDBCluster` | `aws_rds_clusters` | | `rds.amazonaws.com` | `CreateDBInstance` | `aws_rds_instances` | | `rds.amazonaws.com` | `ModifyDBCluster` | `aws_rds_clusters` | | `rds.amazonaws.com` | `ModifyDBInstance` | `aws_rds_instances` | | `rds.amazonaws.com` | `DeleteDBCluster` | `aws_rds_clusters` | | `rds.amazonaws.com` | `DeleteDBInstance` | `aws_rds_instances` | #### Health :::callout{type="info"} AWS Health events are delivered via Amazon EventBridge, not CloudTrail. The CloudFormation template includes an EventBridge rule that routes Health events to the same Kinesis stream used for CloudTrail events. ::: | Source | Event | Plugin table | Notes | | ------------ | ------------------ | ---------------------------------- | ------------------------------------------- | | `aws.health` | `AWS Health Event` | `aws_health_events` | Always synced when a Health event is received | | `aws.health` | `AWS Health Event` | `aws_health_organization_events` | Only synced when the event includes an `affectedAccount` (org aggregation) | **Organization events**: When [AWS Health organizational view](https://docs.aws.amazon.com/health/latest/ug/aggregate-events.html) is enabled, Health events on member accounts are forwarded to the management account's EventBridge with an `affectedAccount` field. This triggers a sync for `aws_health_organization_events` in addition to `aws_health_events`. To use this: 1. Enable Health organizational view on the management account: ```bash aws health enable-health-service-access-for-organization --region us-east-1 ``` 2. Deploy the streaming infrastructure on the management account. 3. Include `aws_health_organization_events` in your `tables` configuration. If organizational view is not enabled, only `aws_health_events` will be synced. #### Route53 | Service | Event | Plugin table | | ------------------------------ | ---------------------------------------- | ------------------------------------------------- | | `route53.amazonaws.com` | `ChangeTagsForResource` | `aws_route53_hosted_zones` | | `route53domains.amazonaws.com` | `RegisterDomain` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `TransferDomain` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `PushDomain` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `RenewDomain` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `EnableDomainTransferLock` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `DisableDomainTransferLock` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `EnableDomainAutoRenew` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `DisableDomainAutoRenew` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `UpdateTagsForDomain` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `DeleteTagsForDomain` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `UpdateDomainContact` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `UpdateDomainContactPrivacy` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `AssociateDelegationSignerToDomain` | `aws_route53_domains` | | `route53domains.amazonaws.com` | `DisassociateDelegationSignerFromDomain` | `aws_route53_domains` | | `route53.amazonaws.com` | `CreateHostedZone` | `aws_route53_hosted_zones` | | `route53domains.amazonaws.com` | `DeleteDomain` | `aws_route53_domains` | | `route53.amazonaws.com` | `DeleteHostedZone` | `aws_route53_hosted_zones` | | `route53.amazonaws.com` | `DeleteQueryLoggingConfig` | `aws_route53_hosted_zone_query_logging_configs` | | `route53.amazonaws.com` | `DeleteTrafficPolicyInstance` | `aws_route53_hosted_zone_traffic_policy_instances`| | `route53.amazonaws.com` | `EnableHostedZoneDNSSEC` | `aws_route53_hosted_zones` | | `route53.amazonaws.com` | `DisableHostedZoneDNSSEC` | `aws_route53_hosted_zones` | | `route53.amazonaws.com` | `AssociateVPCWithHostedZone` | `aws_route53_hosted_zones` | | `route53.amazonaws.com` | `DisassociateVPCFromHostedZone` | `aws_route53_hosted_zones` | | `route53.amazonaws.com` | `CreateVPCAssociationAuthorization` | `aws_route53_hosted_zones` | | `route53.amazonaws.com` | `DeleteVPCAssociationAuthorization` | `aws_route53_hosted_zones` | | `route53.amazonaws.com` | `UpdateHostedZoneComment` | `aws_route53_hosted_zones` | | `route53.amazonaws.com` | `CreateQueryLoggingConfig` | `aws_route53_hosted_zones` | | `route53.amazonaws.com` | `CreateTrafficPolicyInstance` | `aws_route53_hosted_zones` | | `route53.amazonaws.com` | `ChangeResourceRecordSets` | `aws_route53_hosted_zones` | ### Configuration Using Kinesis Data Stream 1. Configure an AWS CloudTrail Trail to send management events to a Kinesis Data Stream via CloudWatch Logs. The most straight-forward way to do this is to use the CloudFormation template provided by CloudQuery. The CloudFormation template will deploy the following architecture: ![Event based syncing cloud infrastructure](https://cli-docs.cloudquery.io/images/docs/aws/event-based-sync-architecture.png) The template contents can be found in CloudFormation Template contents section below. **Note:** If you plan to also sync AWS Health events (step 2), deploy this stack in **us-west-2**. The Health EventBridge rule must be in us-west-2 to capture events from all regions, and EventBridge can only route to a Kinesis stream in the same region. If this doesn't work with your architecture please reach out to us at support@cloudquery.io and we would be happy to help you implement a pattern that works for your organization. ```bash aws cloudformation deploy --template-file ./streaming-deployment.yml --stack-name --capabilities CAPABILITY_IAM --disable-rollback --region ``` 2. **(Optional) Deploy the Health events template** to receive AWS Health events via EventBridge. This must be deployed in **us-west-2** to capture Health events from all regions. The Kinesis stream from step 1 must also be in us-west-2 since EventBridge can only target resources in the same region. ```bash KINESIS_ARN=$(aws cloudformation describe-stacks --stack-name --query "Stacks[0].Outputs[?OutputKey=='KinesisStreamArn'].OutputValue" --output text --region ) aws cloudformation deploy --template-file ./streaming-deployment-health.yml --stack-name -health --capabilities CAPABILITY_IAM --disable-rollback --region us-west-2 --parameter-overrides KinesisStreamArn=$KINESIS_ARN ``` For organization-aggregated Health events, deploy this in the management account (or delegated administrator) with [organizational view](https://docs.aws.amazon.com/health/latest/ug/aggregate-events.html) enabled. 3. Copy the ARN of the Kinesis stream. If you used the CloudFormation template you can run the following command: ```bash aws cloudformation describe-stacks --stack-name --query "Stacks[].Outputs" --region ``` 4. Define a `config.yml` file like the one below ```yaml kind: source spec: name: aws path: cloudquery/aws registry: cloudquery version: "v33.33.1" tables: - aws_ec2_instances - aws_ec2_internet_gateways - aws_ec2_security_groups - aws_ec2_subnets - aws_ec2_vpcs - aws_ecs_cluster_tasks - aws_iam_groups - aws_iam_roles - aws_iam_users - aws_iam_policies - aws_rds_instances - aws_health_events # - aws_health_organization_events # Enable if using org aggregation destinations: ["DESTINATION_NAME"] spec: event_based_sync: # account: # local_profile: "" kinesis_stream_arn: ``` 5. Sync the data! ```bash cloudquery sync config.yml ``` This will start a long-lived process that will only stop when there is an error, or you stop it. #### Limitations - Kinesis Stream can only have a single shard (this is a limitation that we expect to remove in the future). - Stale records will only be deleted if the plugin stops consuming the Kinesis Stream, which only can occur if there is an error. ### Configuration Using S3 Bucket Notifications 1. Create a new SQS queue: ```bash aws sqs create-queue --queue-name bucket-notifications ``` 2. Create a file defining the permissions for the SQS queue and save it as `sqs-policy.json`: ```json { "Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": { "Service": "s3.amazonaws.com" }, "Action": [ "SQS:SendMessage" ], "Resource": "arn:aws:sqs:::", "Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:" }, "StringEquals": { "aws:SourceAccount": "" } } } ] } ``` and then attach it by running the following command: ```bash aws sqs set-queue-attributes --queue-url --policy file://sqs-policy.json ``` 3. Create a file defining the integration between the S3 bucket and the SQS queue and save it as `s3-notification.json`: ```json { "QueueConfigurations": [ { "QueueArn": "arn:aws:sqs:::", "Events": [ "s3:ObjectCreated:*" ] } ] } ``` and then create it by running the following command: ```bash aws s3api put-bucket-notification-configuration --bucket --notification-configuration file://s3-notification.json ``` 4. Define a `config.yml` file like the one below ```yaml kind: source spec: name: aws path: cloudquery/aws registry: cloudquery version: "v33.33.1" tables: - aws_ec2_instances - aws_ec2_internet_gateways - aws_ec2_security_groups - aws_ec2_subnets - aws_ec2_vpcs - aws_ecs_cluster_tasks - aws_iam_groups - aws_iam_roles - aws_iam_users - aws_iam_policies - aws_rds_instances - aws_health_events # - aws_health_organization_events # Enable if using org aggregation destinations: ["DESTINATION_NAME"] spec: event_based_sync: # account: # local_profile: "" sqs_queue_url: ``` 5. Sync the data! ```bash cloudquery sync config.yml ``` This will start a long-lived process that will only stop when there is an error, or you stop it. #### Limitations - This method is not the fastest way to consume CloudTrail events as the data gets buffered before being sent to S3 and then bucket notifications can also have a delay. - Stale records will only be deleted if the plugin stops consuming the events from SQS, which only can occur if there is an error. --- hub-title: Event-based sync CloudFormation templates --- Two CloudFormation templates are provided for event-based sync infrastructure. The first sets up CloudTrail event streaming via Kinesis and can be deployed in any region. The second adds AWS Health event routing via EventBridge and must be deployed in **us-west-2**. If you plan to use both, deploy the CloudTrail stack in **us-west-2** as well, since EventBridge can only route to a Kinesis stream in the same region. Both templates are intended as references that users should amend to fit their needs. ### CloudTrail streaming template (`streaming-deployment.yml`) Creates a Kinesis Data Stream, CloudWatch Logs group, and CloudTrail trail. CloudTrail management events are routed to the Kinesis stream for CloudQuery to consume. Can be deployed in any region. ```yaml AWSTemplateFormatVersion: 2010-09-09 Description: Configures Cloudtrail Events to be piped to a Kinesis Data stream via CloudWatch Logs. Parameters: KinesisMessageDuration: Type: Number Description: Number of hours Kinesis will persist a record before it is purged. Default: 24 ExistingS3BucketName: Type: String Description: Name of the S3 Bucket that CloudTrail will use to store logs. Default: "" Conditions: CreateS3Bucket: !Equals [!Ref ExistingS3BucketName, ""] Resources: # Stream that CQ will poll for changes CQSyncingKinesisStream: Type: AWS::Kinesis::Stream Properties: ShardCount: 1 RetentionPeriodHours: !Ref KinesisMessageDuration # IAM Role for allowing CloudWatch Log to write to Kinesis Stream CloudWatchLogsToKinesisRole: Type: AWS::IAM::Role Properties: Policies: - PolicyName: CloudWatchLogsToKinesisPolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - kinesis:PutRecord Resource: !GetAtt CQSyncingKinesisStream.Arn AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: logs.amazonaws.com Action: - sts:AssumeRole CloudTrailS3Bucket: Type: AWS::S3::Bucket Condition: CreateS3Bucket Properties: LifecycleConfiguration: Rules: - ExpirationInDays: 30 Status: Enabled CloudTrailS3BucketPolicy: Type: AWS::S3::BucketPolicy Properties: Bucket: !If [CreateS3Bucket,!Ref CloudTrailS3Bucket, !Ref ExistingS3BucketName] PolicyDocument: Version: '2012-10-17' Statement: - Sid: AWSCloudTrailAclCheck Effect: Allow Principal: Service: cloudtrail.amazonaws.com Action: s3:GetBucketAcl Resource: !Sub - arn:${AWS::Partition}:s3:::${Bucket} - { Bucket: !If [CreateS3Bucket,!Ref CloudTrailS3Bucket, !Ref ExistingS3BucketName] } Condition: StringEquals: 'aws:SourceAccount': !Sub ${AWS::AccountId} - Sid: AWSCloudTrailWrite Effect: Allow Principal: Service: cloudtrail.amazonaws.com Action: s3:PutObject Resource: !Sub - arn:${AWS::Partition}:s3:::${Bucket}/* - { Bucket: !If [CreateS3Bucket,!Ref CloudTrailS3Bucket, !Ref ExistingS3BucketName] } Condition: StringEquals: 's3:x-amz-acl': bucket-owner-full-control 'aws:SourceAccount': !Sub ${AWS::AccountId} CloudWatchLogsGroup: Type: AWS::Logs::LogGroup UpdateReplacePolicy: Delete DeletionPolicy: Delete Properties: LogGroupName: "CloudTrailLogGroup" RetentionInDays: 1 # Role for allowing CloudTrail to write to CloudWatch Logs CloudWatchRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Sid: AssumeRole Effect: Allow Principal: Service: 'cloudtrail.amazonaws.com' Action: 'sts:AssumeRole' Policies: - PolicyName: 'cloudtrail-policy' PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: 'logs:CreateLogStream' Resource: !GetAtt CloudWatchLogsGroup.Arn - Effect: Allow Action: 'logs:PutLogEvents' Resource: !GetAtt CloudWatchLogsGroup.Arn CloudTrailTrail: Type: AWS::CloudTrail::Trail DependsOn: - CloudTrailS3BucketPolicy Properties: CloudWatchLogsLogGroupArn: !GetAtt CloudWatchLogsGroup.Arn CloudWatchLogsRoleArn: !GetAtt CloudWatchRole.Arn EventSelectors: - IncludeManagementEvents: True ReadWriteType: WriteOnly IncludeGlobalServiceEvents: True IsLogging: True IsMultiRegionTrail: True S3BucketName: !If [CreateS3Bucket,!Ref CloudTrailS3Bucket, !Ref ExistingS3BucketName] SubscriptionFilter: Type: AWS::Logs::SubscriptionFilter Properties: LogGroupName: !Ref CloudWatchLogsGroup DestinationArn: !GetAtt CQSyncingKinesisStream.Arn RoleArn: !GetAtt CloudWatchLogsToKinesisRole.Arn FilterPattern: "" Outputs: KinesisStreamArn: Description: The ARN of the Kinesis Data Stream that CloudQuery will use to listen for changes. Value: !GetAtt CQSyncingKinesisStream.Arn ``` ### Health events template (`streaming-deployment-health.yml`) Routes AWS Health events to an existing Kinesis Data Stream via EventBridge. **Must be deployed in us-west-2** to receive Health events from all regions. AWS Health delivers events to both the originating region and a backup region, and us-west-2 is the backup for all other regions. For organization-aggregated Health events, deploy in the management account (or delegated administrator account) with [organizational view](https://docs.aws.amazon.com/health/latest/ug/aggregate-events.html) enabled. ```yaml AWSTemplateFormatVersion: 2010-09-09 Description: >- Routes AWS Health events to an existing Kinesis Data Stream via EventBridge. Deploy this stack in us-west-2 to receive Health events from all regions. For organization-aggregated Health events, deploy in the management account (or delegated administrator account) with organizational view enabled. Rules: DeployRegion: Assertions: - Assert: !Equals [!Ref "AWS::Region", "us-west-2"] AssertDescription: >- This stack must be deployed in us-west-2. AWS Health delivers events to both the originating region and a backup region (us-west-2 for all regions except us-west-2 itself). Deploying here captures Health events from all regions with a single EventBridge rule. Parameters: KinesisStreamArn: Type: String Description: >- ARN of the Kinesis Data Stream to route Health events to. This should be the stream created by the streaming-deployment.yml CloudFormation stack. Resources: EventBridgeToKinesisRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: events.amazonaws.com Action: sts:AssumeRole Policies: - PolicyName: EventBridgeToKinesisPolicy PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - kinesis:PutRecord - kinesis:PutRecords Resource: !Ref KinesisStreamArn # AWS Health events are delivered to EventBridge in both the originating region # and a designated backup region. us-west-2 is the backup for all other regions, # so a single rule here captures events from every region. # See: https://docs.aws.amazon.com/health/latest/ug/choosing-a-region.html # # For organization-aggregated events (with affectedAccount in the detail), # organizational view must be enabled and this stack must run in the management # account or delegated administrator account. HealthEventsRule: Type: AWS::Events::Rule Properties: Name: !Sub ${AWS::StackName}-health-events-to-kinesis EventPattern: source: - aws.health State: ENABLED Targets: - Id: kinesis-target Arn: !Ref KinesisStreamArn RoleArn: !GetAtt EventBridgeToKinesisRole.Arn KinesisParameters: PartitionKeyPath: "$.id" ``` --- hub-title: FIPS compliance --- A FIPS-compliant version of this plugin is available if your environment requires it. You may enable it by updating the version string in the configuration like this: ```yaml copy kind: source spec: name: aws path: cloudquery/aws registry: cloudquery version: "v33.33.1-fips" ... ``` --- hub-title: Multi-account configuration --- ### AWS Organizations The plugin supports discovery of AWS Accounts via AWS Organizations. This means that as Accounts get added or removed from your organization, it will be able to handle new or removed accounts without any configuration changes. ```yaml copy kind: source spec: name: aws path: cloudquery/aws registry: cloudquery version: "v33.33.1" tables: ['aws_s3_buckets'] destinations: ["postgresql"] spec: aws_debug: false org: admin_account: local_profile: "" member_role_name: cloudquery-ro regions: - '*' ``` Prerequisites for using AWS Org functionality: 1. Have a role (or user) in an Admin account with the following access: - `organizations:ListAccounts` - `organizations:ListAccountsForParent` - `organizations:ListChildren` 2. Have a role in each member account that has a trust policy with a single principal. The default profile name is `OrganizationAccountAccessRole`. The [`OrganizationAccountAccessRole` is created by default](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts_access.html#orgs_manage_accounts_create-cross-account-role) in AWS Accounts created as part of an AWS Organization. We do not recommend using the `OrganizationAccountAccessRole` due to the level of permissions typically granted to the role, but instead recommend for AWS plugin users to create their own IAM roles in each member account with the appropriate read-only permissions. We also recommend ensuring that the IAM roles and policies used for AWS plugin adhere to company security standards. Reference IAM assets and the CloudFormation templates for deployment in an AWS Organization for CloudQuery can be found [here](https://github.com/cloudquery/iam-for-aws-orgs). ### Configuring AWS Organization: 1. It is always necessary to specify a member role name: ```yaml copy org: member_role_name: cloudquery-ro ``` 2. Sourcing credentials that have the necessary `organizations` permissions can be done in any of the following ways: 1. Source credentials from the default credential tool chain: ```yaml copy org: member_role_name: cloudquery-ro ``` 2. Source credentials from a named profile in the shared configuration or credentials file ```yaml copy org: member_role_name: cloudquery-ro admin_account: local_profile: ``` 3. Assume a role in admin account using credentials in the shared configuration or credentials file: ```yaml copy org: member_role_name: cloudquery-ro admin_account: local_profile: role_arn: arn:aws:iam:::role/ # Optional. Specify the name of the session # role_session_name: "" # Optional. Specify the ExternalID if required for trust policy # external_id: "" ``` 3. Optional. If the trust policy configured for the member accounts requires different credentials than you configured in the previous step, then you can specify the credentials to use in the `member_trusted_principal` block: ```yaml copy org: member_role_name: cloudquery-ro member_trusted_principal: local_profile: ``` 4. Optional. If you want to specify specific Organizational Units to fetch from you can add them to the `organization_units` list. ```yaml copy org: member_role_name: cloudquery-ro organization_units: - ou- - ou- ``` Child OUs will also be included. To skip a child OU or account, use the `skip_organization_units` or `skip_member_accounts` options respectively: ```yaml copy org: member_role_name: cloudquery-ro organization_units: - ou- - ou- skip_organization_units: - ou- skip_member_accounts: - ``` 5. Optional. By default, CloudQuery will continue without error if no active accounts are found in the organization (for example, if all accounts are filtered out or skipped). If you want to ensure that at least one account is found, you can set `error_if_no_accounts` to `true`: ```yaml copy org: member_role_name: cloudquery-ro error_if_no_accounts: true ``` This is useful for validation purposes to ensure your organization configuration is correct and hasn't accidentally filtered out all accounts. ### Specific Accounts The AWS plugin can fetch from multiple accounts in parallel by using `AssumeRole` (you will need to use credentials that can `AssumeRole` to all other specified accounts). Below is an example configuration: ```yaml copy accounts: - account_name: role_arn: # Optional. Local Profile is the named profile in your shared configuration file (usually `~/.aws/config`) that you want to use for this specific account local_profile: # Optional. Specify the Role Session name role_session_name: "" - account_name: local_profile: provider # Optional. Role ARN we want to assume when accessing this account role_arn: ``` --- hub-title: Query examples --- ### Find all public-facing load balancers ```sql copy SELECT * FROM aws_elbv2_load_balancers WHERE scheme = 'internet-facing'; ``` ### Find all unencrypted RDS instances ```sql copy SELECT * FROM aws_rds_clusters WHERE storage_encrypted IS FALSE; ``` ### Find all S3 buckets that are permitted to be public ```sql copy SELECT arn, region FROM aws_s3_buckets WHERE block_public_acls IS NOT TRUE OR block_public_policy IS NOT TRUE OR ignore_public_acls IS NOT TRUE OR restrict_public_buckets IS NOT TRUE ``` --- hub-title: Table options --- This feature enables users to override the default options for specific tables. The root of the object takes a table name, and the next level takes an API method name. The final level is the actual input object as defined by the API. The format of the `table_options` object is as follows: ```yaml table_options: : : - ``` A list of `` objects should be provided. The plugin will iterate through these to make multiple API calls. This is useful for APIs like CloudTrail's `LookupEvents` that only supports a single event type per call. For example: ```yaml table_options: aws_cloudtrail_events: LookupEvents: - StartTime: 2023-05-01T20:20:52Z EndTime: 2023-05-03T20:20:52Z LookupAttributes: - AttributeKey: EventName AttributeValue: RunInstances - StartTime: 2023-05-01T20:20:52Z EndTime: 2023-05-03T20:20:52Z LookupAttributes: - AttributeKey: EventName AttributeValue: StartInstances - StartTime: 2023-05-01T20:20:52Z EndTime: 2023-05-03T20:20:52Z LookupAttributes: - AttributeKey: EventName AttributeValue: StopInstances ``` The following tables and APIs are supported: ```yaml table_options: aws_accessanalyzer_analyzer_findings: ListFindings: - # NextToken & AnalyzerArn are prohibited aws_accessanalyzer_analyzer_findings_v2: ListFindingsV2: - # NextToken & AnalyzerArn are prohibited aws_bedrock_inference_profiles: ListInferenceProfiles: - # NextToken is prohibited. MaxResults should be in range [1-1000]. # By default, both SYSTEM_DEFINED and APPLICATION profile types are synced. # To sync only application inference profiles: # ListInferenceProfiles: # - TypeEquals: APPLICATION aws_cloudtrail_events: LookupEvents: - # NextToken is prohibited aws_cloudtrail_trails: DescribeTrails: - aws_cloudformation_stack_templates: GetTemplate: - TemplateStage: Processed # Optional. Allowed values: Processed (default), Original. # To sync both stages simultaneously: # GetTemplate: # - TemplateStage: Processed # - TemplateStage: Original aws_cloudwatch_metrics: - ListMetrics: # NextToken is prohibited GetMetricData: - # MaxDatapoints, NextToken and ScanBy are prohibited GetMetricStatistics: - # Namespace, MetricName and Dimensions are prohibited aws_cloudwatchlogs_delivery_destinations: DescribeDeliveryDestinations: - # NextToken is prohibited aws_cloudwatchlogs_delivery_sources: DescribeDeliverySources: - # NextToken is prohibited aws_cloudwatchlogs_log_groups: - DescribeGroups: # NextToken is prohibited DescribeStreams: cloudwatchStreamLastEventTimeAfter: