GCS (Google Cloud Storage) Destination Plugin

Latest: v3.2.0

This destination plugin lets you sync data from a CloudQuery source to remote GCS (Google Cloud Storage) storage in various formats such as CSV, JSON and Parquet.

This is useful in various use-cases, especially in data lakes where you can query the data direct from Athena or load it to various data warehouses such as BigQuery, RedShift, Snowflake and others.


This example configures a GCS destination, to create CSV files in gcs://bucket_name/path/to/files.

kind: destination
  name: "gcs"
  path: "cloudquery/gcs"
  version: "v3.2.0"
  write_mode: "append" # gcs only supports 'append' mode
  # batch_size: 10000 # optional
  # batch_size_bytes: 5242880 # optional
    bucket: "bucket_name"
    path: "path/to/files"
    format: "csv"
      delimiter: ","

Note that the GCS plugin only supports append write-mode. The (top level) spec section is described in the Destination Spec Reference.

The GCS destination utilizes batching, and supports batch_size and batch_size_bytes.

GCS Spec

This is the (nested) spec used by the CSV destination Plugin.

  • bucket (string) (required)

    Bucket where to sync the files.

  • path (string) (required)

    Path to where the files will be uploaded in the above bucket.

  • format (string) (required)

    Format of the output file. Supported values are csv, json and parquet.

  • format_spec (map format_spec) (optional) Optional parameters to change the format of the file


  • delimiter (string) (optional) (default: ,)

    Character that will be used as want to use as the delimiter if the format type is csv

  • skip_header (bool) (optional) (default: false)

    Specifies if the first line of a file should be the headers (when format is csv).


The GCS plugin authenticates using your Application Default Credentials (opens in a new tab). Available options are all the same options described here (opens in a new tab) in detail:

Local Environment:

  • gcloud auth application-default login (recommended when running locally)

Google Cloud cloud-based development environment:

  • When you run on Cloud Shell or Cloud Code credentials are already available.

Google Cloud containerized environment:

Google Cloud services that support attaching a service account (opens in a new tab):

  • Services such as Compute Engine, App Engine and functions supporting attaching a user-managed service account which will CloudQuery will be able to utilize.

On-premises or another cloud provider

  • The suggested way is to use Workload identity federation (opens in a new tab)
  • If not available you can always use service account keys and export the location of the key via GOOGLE_APPLICATION_CREDENTIALS. (Not recommended as long-lived keys are a security risk)