AWS
Azure
GCP
Security
How Do You Run CloudQuery Without Static Credentials?
TL;DR
Three recent releases let self-hosted CloudQuery syncs authenticate without a stored credential: workload identity federation in the GCP Source (v23.4.1) and Azure Source (v22.3.1), AWS IAM database authentication in the PostgreSQL Destination (v8.16.0), and Databricks Lakebase OAuth in that same Destination. Each one mints a short-lived credential at connection time instead of reading a long lived password or key from your configuration.
Every long-lived credential in a configuration file is something you have to store, rotate, and audit. A GCP service account key or an Azure client secret sitting in a Kubernetes secret does not expire on its own, and if it leaks you usually find out from someone else.
Why Move Off Static Keys at All? #
A service account JSON key is a bearer credential with no expiry. It works from anywhere, for anyone holding the file, until somebody remembers to revoke it. Most teams we talk to know this and keep the keys anyway, because the alternative involved standing up token exchange by hand for every plugin.
These three features push that work into the plugins. Your runtime already has an identity, whether that is an EKS service account, a GKE workload identity, or an AKS federated token. Now the plugins can use it.
How Does Workload Identity Federation Work in CloudQuery? #
The GCP and Azure Sources both accept a
workload_identity_federation block. The plugin reads a short-lived OIDC token that your runtime projects into the filesystem, exchanges it for cloud credentials, and repeats that on each sync.For GCP:
spec:
workload_identity_federation:
audience: '//iam.googleapis.com/projects/<project_number>/locations/global/workloadIdentityPools/<pool_id>/providers/<provider_id>'
credential_source_file: '/var/run/secrets/oidc/token'
service_account_email: 'cloudquery@<project_id>.iam.gserviceaccount.com'
For Azure:
spec:
workload_identity_federation:
token_file_path: '/var/run/secrets/azure/tokens/azure-identity-token'
client_id: '<client_id>'
tenant_id: '<tenant_id>'
Both plugins re-read the token file on every exchange, so rotation by the projected volume works without restarting the sync.
Two IAM details on the GCP side deserve more care than they usually get. Bind
roles/iam.workloadIdentityUser with principal://.../subject/<subject> instead of a principalSet:// wildcard, and pin the provider with an attribute condition of assertion.sub == '<subject>'. A wildcard binding lets any identity in the pool impersonate that service account, which defeats most of the point of moving off keys.If you are already running on AKS with the workload identity webhook,
AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_FEDERATED_TOKEN_FILE are in the pod environment and DefaultAzureCredential picks up federation with no plugin configuration at all. GKE has the equivalent shortcut through native GKE Workload Identity.How Do You Sync to RDS Without a Database Password? #
The PostgreSQL Destination supports RDS IAM database authentication. Add an
aws_iam_auth block and the plugin signs a fresh token for every new connection, so the connection string carries no password:spec:
connection_string: 'host=${PGHOST} port=5432 dbname=postgres user=cloudquery sslmode=require'
aws_iam_auth:
service: 'rds'
region: 'us-east-1'
Credentials resolve from the standard AWS chain, which covers EC2 instance profiles and EKS service accounts. Set
role_arn (with role_session_name and external_id if your trust policy wants them) to assume a role before signing, or local_profile to pin a shared config profile. TLS is required and enforced.On the database side, enable IAM authentication on the instance, create the user, and grant it the
rds_iam role:CREATE USER cloudquery;
GRANT rds_iam TO cloudquery;
The IAM identity then needs
rds-db:connect on the dbuser ARN for that user.How Does Lakebase OAuth Authentication Work? #
PostgreSQL has no native OAuth, so Databricks Lakebase uses bring-your-own-token: the client mints and refreshes the credential. The PostgreSQL Destination handles that lifecycle. Set a
lakebase block and the plugin uses the Databricks SDK to generate a short-lived database credential before each new connection.spec:
connection_string: 'host=${PGHOST} port=5432 dbname=databricks_postgres user=${DATABRICKS_CLIENT_ID} sslmode=require'
lakebase:
endpoint: '${LAKEBASE_ENDPOINT_NAME}'
host: '${DATABRICKS_HOST}'
client_id: '${DATABRICKS_CLIENT_ID}'
client_secret: '${DATABRICKS_CLIENT_SECRET}'
The user in the connection string is the service principal client ID, and
sslmode has to be require, verify-ca, or verify-full. You can omit host, client_id, and client_secret and let the SDK resolve them from DATABRICKS_HOST, DATABRICKS_CLIENT_ID, and DATABRICKS_CLIENT_SECRET.What Should You Watch For? #
- GCP subject tokens are file-sourced only. The credential source has to be a file holding a raw JWT. AWS and SAML credential sources are not supported.
- Azure tokens need headroom. Keep Azure token lifetimes at one hour or more. The Azure SDK caches the token file for up to ten minutes, so the token in the file must never be within ten minutes of expiry. Kubernetes refreshes projected tokens at 80% of their lifetime, so a one-hour token always has at least twelve minutes left. The AKS default already meets this. Only shorten expirationSeconds with care.
- RDS IAM costs instance memory. AWS documents 300 to 1000 MiB of extra memory on the database. On a burstable class instance, reduce other buffers by the same amount.
rds_iamwins over passwords. Once the role is granted to a user, IAM authentication takes precedence for that user, so plan the cutover rather than running both.- Lakebase still has one stored secret. The service principal client secret lives somewhere. What this removes is a database password with no expiry, replaced by a Databricks credential you can scope and rotate.
Which Credential Should You Replace First? #
Look at whichever credential in your configuration files has the longest remaining lifetime. In most setups that is a GCP service account JSON key that has been in a secrets manager since the pipeline was first built, or a PostgreSQL password nobody has rotated since the destination was set up. Both have a documented path off now.
Start with the GCP Source, Azure Source, and PostgreSQL Destination documentation. Join the CloudQuery community to compare notes with other users, or message our team if you hit something the docs do not cover.
FAQs #
Do I need Kubernetes to use workload identity federation? #
No. The GCP Source reads the subject token from a file path you control, so anything that can write a valid OIDC token to disk and refresh it works. Kubernetes projected service account tokens are the common case because the refresh is handled for you.
Can I use workload_identity_federation alongside a service account key? #
No. In the GCP Source it is mutually exclusive with
service_account_key_json, get_token_command, and service_account_impersonation. In the Azure Source it is mutually exclusive with the deprecated oidc_token and with auth.get_token_command.Does aws_iam_auth work with Aurora PostgreSQL? #
Yes. The
service field defaults to rds, which covers both Amazon RDS and Aurora PostgreSQL.How often does the plugin mint a new token? #
Per connection, not per sync. Both
aws_iam_auth and lakebase generate a fresh credential before each new connection in the pool, which keeps every token well inside its lifetime.Can I set aws_iam_auth and lakebase at the same time? #
No. The two blocks are mutually exclusive in the PostgreSQL Destination spec, since each one supplies the connection password by a different mechanism.
Try CloudQuery yourself
Self-hosted, open source, and free to start. Or schedule a demo to see the managed version.
