Skip to Content
PlatformIntegration GuidesGCP

Setting up a GCP Integration

CloudQuery Platform authenticates with GCP either through workload identity federation, which stores no credentials, or through a service account JSON key file.

Workload identity federation is the recommended method: CloudQuery impersonates a service account in your project using a short-lived token, so there is no key to store or rotate. Use a service account key if you cannot run the setup script. The CloudQuery CLI supports further methods, including GKE Workload Identity — see the GCP integration documentation.

Prerequisites

  • A CloudQuery Platform account with admin access
  • A GCP project with permissions to create service accounts and manage IAM
  • Access to the GCP Console

Step 1: Enable the Cloud Resource Manager API

The GCP integration requires the Cloud Resource Manager API to be enabled on each project CloudQuery will read. This API lets CloudQuery discover your projects, folders, and organizations.

  1. Open the Cloud Resource Manager API in the GCP Console
  2. Make sure the correct project is selected at the top of the page
  3. Click Enable

Step 2: Set up authentication

Choose one of the two methods below. Workload identity federation is recommended; the service account key flow follows it.

Either way, the service account needs two roles: Viewer (roles/viewer) and Security Auditor (roles/iam.securityAuditor). Viewer on its own is missing permissions the plugin needs — storage.buckets.getIamPolicy among them — and the tables that depend on them fail with permission errors during the sync. Security Auditor supplies those, and is most useful bound at the organization level, though it also works at project scope for project-level tables.

Option A: Workload identity federation

CloudQuery generates a gcloud script that creates a workload identity pool trusting only your CloudQuery tenant, plus a service account for CloudQuery to impersonate. No key is created, so there is nothing to store or rotate.

You need the gcloud CLI and permission to create workload identity pools, service accounts and IAM bindings in the project. The script enables the APIs it needs, including the one in Step 1.

  1. In CloudQuery Platform, go to Data Sources → Integrations. Click Add new integration and type GCP to find the GCP integration.

  2. Under Choose authentication method, select Workload identity federation.

  3. Copy the generated script and run it in a terminal where gcloud points at the project you want CloudQuery to read:

    gcloud config set project YOUR_PROJECT_ID

  4. The script prints four values when it finishes. Paste them back into CloudQuery:

    • Project number
    • Pool ID
    • Provider ID
    • Service account email

  5. Click Verify and complete setup. CloudQuery exchanges a token through the provider and checks that the trust is scoped to your tenant alone. It refuses to complete the setup if the pool or service account is bound more broadly.

  6. The script grants both Viewer and Security Auditor on the project it ran in, and nothing beyond it. To sync any other project, folder or organization, grant both roles at that scope too — see Syncing more than one project.

IAM changes take up to a few minutes to propagate in GCP. If verification fails immediately after running the script with a permission error, wait a moment and click Verify and complete setup again before changing anything.

Continue at Step 3.

Option B: Service account key

CloudQuery uses a service account to read resources from your GCP environment. Follow these steps to set up a new service account with read-only access:

  1. Open the GCP Service Accounts page
  2. Select the project to create the service account in (you can assign access to other projects later)
  3. Click Create Service Account
  4. Enter the details:
    1. Service account display name, e.g. CloudQuery Readonly
    2. Service account ID, e.g. cloudquery-readonly
    3. A description, e.g. Service account for CloudQuery to fetch resources in GCP
    4. Click Create and Continue

Setting up a service account

  1. Under Basic, select the Viewer role for the service account, then add a second role, Security Auditor (roles/iam.securityAuditor).

Selecting the Viewer role

  1. Click Continue and Done.
  2. In the service accounts list, click on the new service account, then go to the Keys tab. Click Add Key → Create New Key.
  3. Select JSON and click Create. This downloads a JSON key file to your computer. You need this file when configuring the integration in CloudQuery Platform.

Getting a private key

Store the JSON key file securely. It contains credentials that grant access to your GCP resources. You will upload the contents to CloudQuery Platform in Step 3, after which you can delete the local file.

Syncing more than one project

Both methods leave the service account holding Viewer and Security Auditor on a single project — for workload identity federation, the project your gcloud config pointed at when you ran the script. To sync more, grant both roles to that same service account at the wider scope you need, using the sections below.

Grant the access before you create the integration. Adding an organization, folder or project ID in CloudQuery does not grant anything in GCP, and a scope with no binding syncs no resources.

Each scope has its own gcloud subcommand — a folder is not a flag on the project form — and both roles are needed at whichever scope you use. Replace SERVICE_ACCOUNT_EMAIL with the service account you created above, and the id placeholder with your own:

gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" --role="roles/viewer"
gcloud projects add-iam-policy-binding PROJECT_ID --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" --role="roles/iam.securityAuditor"
 
gcloud organizations add-iam-policy-binding ORGANIZATION_ID --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" --role="roles/viewer"
gcloud organizations add-iam-policy-binding ORGANIZATION_ID --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" --role="roles/iam.securityAuditor"
 
gcloud resource-manager folders add-iam-policy-binding FOLDER_ID --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" --role="roles/viewer"
gcloud resource-manager folders add-iam-policy-binding FOLDER_ID --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" --role="roles/iam.securityAuditor"

An organization binding reaches the most resources, and a folder binding is inherited by projects added to that folder later. The console equivalents are below.

Optional: Assign organization or folder-wide access

To sync resources across all your GCP projects, grant both the Viewer role and Security Auditor to the service account at the organization or folder level:

  1. In the GCP Console project selection screen, select your top-level organization (or folder)
  2. Go to IAM and Admin → IAM, and click Grant Access
  3. Paste the email address of the service account you created above in the New Principals text box. Assign the Viewer and Security Auditor roles.
  4. Click Save

Assign Organization or folder-wide access to the Service Account

Optional: Assign access to individual projects

You can also grant access to specific projects if you don’t need organization-wide access:

  1. In the GCP Console project selection screen, select the relevant project
  2. Go to IAM and Admin → IAM, and click Grant Access
  3. Paste the email address of the service account in the New Principals text box. Assign the Viewer and Security Auditor roles.
  4. Click Save

Optional: Billing export access

If you also run the cloudquery/gcpbilling plugin against this service account, it reads your BigQuery detailed billing export and needs bigquery.tables.getData, which neither role above grants. Add roles/bigquery.dataViewer on the export dataset rather than project-wide. The GCP Billing integration sets up its own connection if you have not created one here, and either way its form prints this command with the service account, project and dataset already filled in:

bq add-iam-policy-binding -d \
  --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" \
  --role="roles/bigquery.dataViewer" \
  "PROJECT_ID:DATASET_ID"

Step 3: Create the integration

  1. In CloudQuery Platform, go to Data Sources → Integrations. Click Add new integration and type GCP to find the GCP integration.

The CloudQuery Platform New integration page with gcp typed into the integration search box

  1. Choose a name for your integration (e.g. GCP).

  2. If you used workload identity federation, the connection is already verified and there is nothing to enter here. If you used a service account key, upload the JSON key file with the Upload JSON file button, or drag and drop the file onto it.

  3. By default, Full auto discovery is enabled, which syncs all projects the service account has access to. To sync specific projects, disable auto discovery and enter your project IDs, organization IDs, or folder IDs.

  4. Click Continue to select the GCP services you want to sync.

  5. Click Test and Continue to verify the setup.

If you used a service account key, you can safely delete the JSON key file from your local disk after a successful test connection. The credentials are stored securely in CloudQuery Platform. Workload identity federation stores no credentials at all.

What gets synced

The GCP integration can sync hundreds of tables across GCP services. Some of the most commonly used tables include:

CategoryTablesDescription
Computegcp_compute_instances, gcp_compute_disksVM instances, persistent disks
Storagegcp_storage_bucketsCloud Storage buckets
Networkinggcp_compute_networks, gcp_compute_firewalls, gcp_compute_subnetworksVPCs, firewall rules, subnets
Containersgcp_container_clustersGKE clusters
Identitygcp_iam_service_accountsIAM service accounts

See the full GCP table list for all available tables.

Verify the integration

After your first sync completes, open the SQL Console and run these queries to confirm data arrived:

-- Count synced compute instances
SELECT count(*) FROM gcp_compute_instances
-- List synced GCP projects
SELECT DISTINCT project_id FROM gcp_compute_instances
-- View storage buckets
SELECT project_id, name, location FROM gcp_storage_buckets LIMIT 10

You can also browse your GCP resources in the Asset Inventory under the Compute, Storage, Networking, and other categories.

Troubleshooting

IssueCauseFix
Invalid JSON keyMalformed or incomplete JSONVerify you copied the entire contents of the JSON key file, including the opening { and closing }. Re-download the key from GCP if needed.
Permission deniedService account is missing a roleVerify the service account has both the Viewer role and Security Auditor on the project, folder, or organization you want to sync. Errors naming getIamPolicy usually mean Security Auditor is missing.
Project not foundService account not granted accessThe service account can only access projects where it has been granted IAM permissions. Grant both roles on each project, or at the organization/folder level.
API not enabledRequired GCP API is disabledSome resources require specific APIs to be enabled in the project (e.g., Compute Engine API, Cloud Storage API). Enable them in the GCP API Library.
No data for some resourcesServices not selectedCheck that the relevant services are selected in the Select services step.
Verification fails right after running the scriptIAM changes have not propagatedWait a few minutes and click Verify and complete setup again. GCP takes up to several minutes to apply new IAM bindings.
Verification refuses to completeTrust is scoped too broadlyCloudQuery only accepts a workload identity pool and service account bound to your tenant alone. Re-run the setup script rather than widening the binding by hand.

Next steps

Was this page helpful?

Last updated on

AWS Cost & UsageAzure