Back to plugin list
gcp
Official
Open-core

GCP

The GCP Source plugin for CloudQuery extracts configuration from a variety of GCP APIs and loads it into any supported CloudQuery destination. Some tables are marked as premium and have a price per 1M rows synced.

Publisher

cloudquery

Repositorygithub.com
Latest version

v12.3.0

Type

Source

Platforms
Date Published

Mar 19, 2024

Price per 1M rows

$10 (premium tables)

monthly free quota

10M rows

Set up process


brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync gcp.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview

The GCP Source plugin for CloudQuery extracts configuration from a variety of GCP APIs and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).
This plugin is an open-core plugin, which means some tables are free and some premium. You can find a list of all premium tables below.

Libraries in Use

Authentication

The GCP plugin authenticates using your Application Default Credentials. Available options are all the same options described here in detail:
Local Environment:
  • gcloud auth application-default login (recommended when running locally)
Google Cloud cloud-based development environment:
  • When you run on Cloud Shell or Cloud Code credentials are already available.
Google Cloud containerized environment:
  • Services such as Compute Engine, App Engine and functions supporting attaching a user-managed service account which will CloudQuery will be able to utilize.
On-premises or another cloud provider
  • The suggested way is to use Workload identity federation
  • If not available you can always use service account keys and export the location of the key via GOOGLE_APPLICATION_CREDENTIALS. Highly not recommended as long-lived keys are a security risk

Query Examples:

Find all buckets without uniform bucket-level access

select project_id, name from gcp_storage_buckets where uniform_bucket_level_access->>'Enabled' = 'true';


Configuration

GCP Source Plugin Configuration Reference

Example

This example connects a single GCP project to a Postgres destination. The (top level) source spec section is described in the Source Spec Reference.
kind: source
spec:
  # Source spec section
  name: "gcp"
  path: "cloudquery/gcp"
  registry: "cloudquery"
  version: "v12.3.0"
  tables: ["gcp_storage_buckets"]
  destinations: ["postgresql"]
  spec:
    # GCP Spec
    project_ids: ["my-project"]

GCP Spec

This is the (nested) spec used by GCP Source Plugin
  • project_ids ([]string) (default: empty. will use all projects available to the current authenticated account)
    Specify projects to connect to. If either folder_ids or project_filter is specified, these projects will be synced in addition to the projects from the folder/filter.
  • service_account_key_json (string) (default: empty).
    GCP service account key content.
    Using service accounts is not recommended, but if it is used it is better to use environment or file variable substitution.
  • folder_ids ([]string) (default: empty).
    CloudQuery will sync from all the projects in the specified folders, recursively. folder_ids must be of the format folders/<folder_id> or organizations/<organization_id>. This feature requires the resourcemanager.folders.list permission.
    By default, CloudQuery will also sync from sub-folders recursively (up to depth 100). To reduce this, set folder_recursion_depth to a lower value (or to 0 to disable recursion completely).
    Mutually exclusive with project_filter.
  • folder_recursion_depth (integer) (default: 100).
    The maximum depth to recurse into sub-folders. 0 means no recursion (only the top-level projects in folders will be used for sync).
  • project_filter (string) (default: empty).
    A filter to determine the projects that are synced, mutually exclusive with folder_ids.
    For instance, to only sync projects where the name starts with how-, set project_filter to name:how-*.
    More examples:
    • "name:how-* OR name:test-*" matches projects starting with how- or test-
    • "NOT name:test-*" matches all projects not starting with test-
    For syntax and example queries refer to API References here and here.
  • organization_ids ([]string) (default: empty. will use all organizations available to the current authenticated account)
    Specify organizations to use when syncing organization level resources (e.g. folders or security findings).
    If organization_filter is specified, these organizations will be used in addition to the organizations from the filter.
  • organization_filter (string) (default: empty).
    A filter to determine the organizations to use when syncing organization level resources (e.g. folders or security findings).
    For instance, to use only organizations from the cloudquery.io domain, set organization_filter to domain:cloudquery.io.
    For syntax and example queries refer to API Reference here.
  • backoff_delay (integer) (default: 30).
    If specified APIs will be retried with exponential backoff if they are rate limited. This is the max delay (in seconds) between retries.
  • backoff_retries (integer) (default: 0).
    If specified APIs will be retried with exponential backoff if they are rate limited. This is the max number of retries.
  • enabled_services_only (boolean) (default: false).
    If enabled CloudQuery will skip any resources that belong to a service that has been disabled or not been enabled.
    If you use this option on a large organization (with more than 500 projects) you should also set the backoff_retries to a value greater than 0, otherwise you may hit the API rate limits.
    In >=v9.0.0 if an error is returned then CloudQuery will assume that all services are enabled and will continue to attempt to sync all specified tables rather than just ending the sync.
  • concurrency (integer) (default: 50000).
    The best effort maximum number of Go routines to use. Lower this number to reduce memory usage.
  • discovery_concurrency (integer) (default: 100).
    The number of concurrent requests that CloudQuery will make to resolve enabled services. This is only used when enabled_services_only is set to true.
  • scheduler (string) (default: dfs):
    The scheduler to use when determining the priority of resources to sync. Currently, the only supported values are dfs (depth-first search), round-robin and shuffle.
    For more information about this, see performance tuning.
  • service_account_impersonation (Service Account Impersonation spec, optional. Default: empty)
    Service Account impersonation configuration.

Service Account Impersonation Spec

  • target_principal (string) (required)
    The email address of the service account to impersonate.
  • scopes ([]string) (default: ["https://www.googleapis.com/auth/cloud-platform"])
    Scopes that the impersonated credential should have.
    See available scopes in the documentation.
  • delegates ([]string) (default: empty)
    Delegates are the service account email addresses in a delegation chain. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next service account in the chain.
  • subject (string) (default: empty)
    The subject field of a JWT (sub). This field should only be set if you wish to impersonate a user. This feature is useful when using domain wide delegation.

GCP + Kubernetes (GKE)

kind: source
spec:
  name: gcp
  path: "cloudquery/gcp"
  registry: cloudquery
  version: "v12.3.0"
  tables: ["gcp_container_clusters"]
  destinations: ["<destination>"]
---
kind: source
spec:
  name: k8s
  path: "cloudquery/k8s"
  registry: cloudquery
  version: "v6.0.7"
  tables: ["*"]
  destinations: ["<destination>"]
Kubernetes users may see the following message when running the K8s plugin on GKE Clusters:
WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
As part of an initiative to remove platform specific code from Kubernetes, authentication will begin to be delegated to authentication plugins, starting in version 1.26.

What does this mean for CloudQuery users?

CloudQuery does not use any specific resources which hinder the upgrade.

Install

The easiest way to upgrade, is to install gke-gcloud-auth-plugin from gcloud components on Mac or Windows:
gcloud components install gke-gcloud-auth-plugin
and apt on Deb based systems:
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin

Verify

Mac or Linux:
gke-gcloud-auth-plugin --version
Windows:
gke-gcloud-auth-plugin.exe --version

Switch authentication methods

Set the flag:
export USE_GKE_GCLOUD_AUTH_PLUGIN=True
Update components:
gcloud components update
Force credential update:
gcloud container clusters get-credentials {$CLUSTER_NAME}
Now you should be able to use kubectl as normal, and you should no longer see the warning in the CloudQuery output.
For more information, read Google's press release.


Premium Tables

  • gcp_accessapproval_folder_approval_requests
  • gcp_accessapproval_folder_service_accounts
  • gcp_accessapproval_folder_settings
  • gcp_accessapproval_organization_approval_requests
  • gcp_accessapproval_organization_service_accounts
  • gcp_accessapproval_organization_settings
  • gcp_accessapproval_project_approval_requests
  • gcp_accessapproval_project_service_accounts
  • gcp_accessapproval_project_settings
  • gcp_aiplatform_batch_prediction_jobs
  • gcp_aiplatform_custom_jobs
  • gcp_aiplatform_data_labeling_jobs
  • gcp_aiplatform_dataset_locations
  • gcp_aiplatform_datasets
  • gcp_aiplatform_endpoint_locations
  • gcp_aiplatform_endpoints
  • gcp_aiplatform_featurestore_locations
  • gcp_aiplatform_featurestores
  • gcp_aiplatform_hyperparameter_tuning_jobs
  • gcp_aiplatform_index_endpoints
  • gcp_aiplatform_index_locations
  • gcp_aiplatform_indexendpoint_locations
  • gcp_aiplatform_indexes
  • gcp_aiplatform_job_locations
  • gcp_aiplatform_metadata_locations
  • gcp_aiplatform_metadata_stores
  • gcp_aiplatform_model_deployment_monitoring_jobs
  • gcp_aiplatform_model_locations
  • gcp_aiplatform_models
  • gcp_aiplatform_operations
  • gcp_aiplatform_pipeline_jobs
  • gcp_aiplatform_pipeline_locations
  • gcp_aiplatform_specialist_pools
  • gcp_aiplatform_specialistpool_locations
  • gcp_aiplatform_studies
  • gcp_aiplatform_tensorboard_locations
  • gcp_aiplatform_tensorboards
  • gcp_aiplatform_training_pipelines
  • gcp_aiplatform_vizier_locations
  • gcp_apigateway_apis
  • gcp_apigateway_gateways
  • gcp_apikeys_keys
  • gcp_appengine_apps
  • gcp_appengine_authorized_certificates
  • gcp_appengine_authorized_domains
  • gcp_appengine_domain_mappings
  • gcp_appengine_firewall_ingress_rules
  • gcp_appengine_services
  • gcp_appengine_versions
  • gcp_artifactregistry_docker_images
  • gcp_artifactregistry_files
  • gcp_artifactregistry_locations
  • gcp_artifactregistry_packages
  • gcp_artifactregistry_repositories
  • gcp_artifactregistry_tags
  • gcp_artifactregistry_versions
  • gcp_baremetalsolution_instances
  • gcp_baremetalsolution_networks
  • gcp_baremetalsolution_nfs_shares
  • gcp_baremetalsolution_volume_luns
  • gcp_baremetalsolution_volumes
  • gcp_batch_jobs
  • gcp_batch_task_groups
  • gcp_batch_tasks
  • gcp_beyondcorp_app_connections
  • gcp_beyondcorp_app_connectors
  • gcp_beyondcorp_app_gateways
  • gcp_beyondcorp_client_connector_services
  • gcp_beyondcorp_client_gateways
  • gcp_bigquery_datasets
  • gcp_bigquery_tables
  • gcp_bigtableadmin_app_profiles
  • gcp_bigtableadmin_backups
  • gcp_bigtableadmin_clusters
  • gcp_bigtableadmin_instances
  • gcp_bigtableadmin_tables
  • gcp_billing_billing_accounts
  • gcp_billing_budgets
  • gcp_billing_services
  • gcp_binaryauthorization_assertors
  • gcp_certificatemanager_certificate_issuance_configs
  • gcp_certificatemanager_certificate_map_entries
  • gcp_certificatemanager_certificate_maps
  • gcp_certificatemanager_certificates
  • gcp_certificatemanager_dns_authorizations
  • gcp_clouddeploy_delivery_pipelines
  • gcp_clouddeploy_job_runs
  • gcp_clouddeploy_releases
  • gcp_clouddeploy_rollouts
  • gcp_clouddeploy_targets
  • gcp_clouderrorreporting_error_events
  • gcp_clouderrorreporting_error_group_stats
  • gcp_cloudiot_device_configs
  • gcp_cloudiot_device_registries
  • gcp_cloudiot_device_states
  • gcp_cloudiot_devices
  • gcp_cloudscheduler_jobs
  • gcp_cloudscheduler_locations
  • gcp_cloudsupport_cases
  • gcp_compute_backend_buckets
  • gcp_compute_global_addresses
  • gcp_compute_health_checks
  • gcp_compute_network_endpoint_groups
  • gcp_compute_target_instances
  • gcp_compute_target_pools
  • gcp_containeranalysis_occurrences
  • gcp_dataproc_autoscaling_policies
  • gcp_dataproc_cluster_nodegroups
  • gcp_dataproc_clusters
  • gcp_dataproc_jobs
  • gcp_dataproc_regions
  • gcp_dns_managed_zones
  • gcp_dns_policies
  • gcp_dns_resource_record_sets
  • gcp_domains_registrations
  • gcp_essentialcontacts_folder_contacts
  • gcp_essentialcontacts_organization_contacts
  • gcp_essentialcontacts_project_contacts
  • gcp_functions_functions
  • gcp_functionsv2_functions
  • gcp_iam_deny_policies
  • gcp_iam_organizational_roles
  • gcp_iam_roles
  • gcp_iam_service_account_keys
  • gcp_iam_service_accounts
  • gcp_kms_crypto_key_versions
  • gcp_kms_crypto_keys
  • gcp_kms_ekm_connections
  • gcp_kms_import_jobs
  • gcp_kms_keyrings
  • gcp_kms_locations
  • gcp_livestream_channels
  • gcp_livestream_inputs
  • gcp_logging_metrics
  • gcp_logging_sinks
  • gcp_monitoring_alert_policies
  • gcp_networkconnectivity_internal_ranges
  • gcp_networkconnectivity_locations
  • gcp_redis_instances
  • gcp_resourcemanager_folders
  • gcp_resourcemanager_organization_tag_keys
  • gcp_resourcemanager_organization_tag_values
  • gcp_resourcemanager_project_policies
  • gcp_resourcemanager_project_tag_bindings
  • gcp_resourcemanager_project_tag_keys
  • gcp_resourcemanager_project_tag_values
  • gcp_resourcemanager_projects
  • gcp_resourcemanager_subfolders
  • gcp_run_locations
  • gcp_run_services
  • gcp_secretmanager_secrets
  • gcp_securitycenter_folder_findings
  • gcp_securitycenter_organization_findings
  • gcp_securitycenter_project_findings
  • gcp_serviceusage_services
  • gcp_translate_glossaries
  • gcp_videotranscoder_job_templates
  • gcp_videotranscoder_jobs
  • gcp_vision_product_reference_images
  • gcp_vision_products
  • gcp_vmmigration_groups
  • gcp_vmmigration_source_datacenter_connectors
  • gcp_vmmigration_source_migrating_vm_clone_jobs
  • gcp_vmmigration_source_migrating_vm_cutover_jobs
  • gcp_vmmigration_source_migrating_vms
  • gcp_vmmigration_source_utilization_reports
  • gcp_vmmigration_sources
  • gcp_vmmigration_target_projects
  • gcp_vpcaccess_connectors
  • gcp_vpcaccess_locations
  • gcp_websecurityscanner_scan_config_scan_run_crawled_urls
  • gcp_websecurityscanner_scan_config_scan_run_findings
  • gcp_websecurityscanner_scan_config_scan_runs
  • gcp_websecurityscanner_scan_configs
  • gcp_workflows_workflows


Subscribe to product updates

Be the first to know about new features.