The CloudQuery Azure Cost source plugin syncs
Azure Cost Management FOCUS exports into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).
Per-resource Azure cost is only available in bulk through Cost Management scheduled exports. The plugin reads those exports where they are delivered, in your own storage account, rather than calling a cost API per billing period, so it is not subject to the rate limits of the Consumption APIs.
Prerequisites #
You need a Cost Management export configured as follows:
In the Azure portal, go to Cost Management > Exports and create an export.
Set the metric to Cost and usage details (FOCUS). Other metrics (ActualCost, AmortizedCost, PriceSheet, reservation datasets) are not supported: the plugin detects them and fails with a message naming what it found.
Set the file format to Parquet. CSV exports are not supported.
Choose a storage account and container to deliver to, and note the export name.
The plugin resolves the storage account, container and folder from the export definition itself, so the spec only names the scope and the export name.
az costmanagement export create cannot create a FOCUS export: its --type accepts only ActualCost, AmortizedCost and Usage, and it does not offer Parquet. Create the export in the portal, or through the REST API at api-version=2023-07-01-preview with definition.type set to FocusCost.
The identity running the sync also needs Storage Blob Data Reader on the destination storage account. Owner and Contributor are control-plane roles and grant no blob data access, so a container listing can succeed while reading the export fails.
Example Configuration #
kind: source
spec:
name: azurecost
path: cloudquery/azurecost
registry: cloudquery
version: "v1.1.1"
tables: ["*"]
destinations: ["postgresql"]
backend_options:
table_name: "cq_state_azurecost"
connection: "@@plugins.postgresql.connection"
spec:
exports:
- scope: "subscriptions/00000000-0000-0000-0000-000000000000"
name: "focus-daily"
# Optional parameters
# sync_from: 2025-01-01T00:00:00Z
# sync_from: "90 days ago"
# blob_service_url: "https://mystorageaccount.blob.core.windows.net"
# Optional parameters
# cloud_name: "AzurePublic"
# cache_credential_tokens: true
# rows_per_record: 500
# concurrency: 50
Authentication #
The plugin authenticates with Azure through the
default azidentity credential chain, which tries, in order, environment variables, workload identity, managed identity, and the Azure CLI. With no credential block in the spec, that chain is what runs.
If you are running the CloudQuery CLI locally and have the
Azure CLI installed, signing in is enough:
The authenticated identity needs two things, on two different planes:
Cost Management: Microsoft.CostManagement/exports/read at the export's scope. The built-in Cost Management Reader role grants it.
Storage: Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read on the storage account the export is delivered to. The built-in Storage Blob Data Reader role grants it.
Owner or Contributor on the subscription is not enough on its own. Those are control-plane roles and grant no data-plane access to blobs, so the plugin can resolve the export and then fail to read it. The connection test lists the delivery container for exactly this reason, so the missing role surfaces before the first sync rather than during it.
Workload identity federation #
Keyless authentication with a federated OIDC token, for example a Kubernetes projected service account token:
spec:
workload_identity_federation:
token_file_path: "/var/run/secrets/azure/tokens/azure-identity-token"
client_id: "00000000-0000-0000-0000-000000000000"
tenant_id: "00000000-0000-0000-0000-000000000000"
The token file is re-read on each refresh, so issue the token with an expiry of at least one hour: the Azure SDK caches the file contents for 10 minutes.
When the plugin runs on AKS with workload identity configured, the default credential chain already picks the projected token up from AZURE_FEDERATED_TOKEN_FILE, and no workload_identity_federation block is needed.
Client secret #
A Microsoft Entra ID application secret, as a fallback where federation is not available:
spec:
client_secret:
tenant_id: "00000000-0000-0000-0000-000000000000"
client_id: "00000000-0000-0000-0000-000000000000"
client_secret: "${AZURE_CLIENT_SECRET}"
Long-lived secrets are a security risk. Use workload identity federation where you can.
External command #
For scenarios the options above cannot cover, the plugin can run a command that prints an access token. The command must write JSON to stdout with a token string and an expires_on field holding Unix epoch seconds. The requested scopes are appended to the arguments, and the token is reused until it is within five minutes of expiring.
spec:
auth:
get_token_command:
command: "az"
args: ["account", "get-access-token", "--output", "json", "--scope"]
OIDC token #
spec:
oidc_token: "${OIDC_TOKEN}"
Deprecated: use workload_identity_federation instead. The token is captured once and never refreshed, so a rotated token expires mid-sync. It also requires the AZURE_CLIENT_ID and AZURE_TENANT_ID environment variables to be set.
Token caching #
spec:
cache_credential_tokens: true
Caches access tokens in memory and shares them across the Cost Management and blob storage clients instead of acquiring one per client. This cuts token acquisitions noticeably, for example the number of az CLI invocations when authenticating through az login.
Choosing one #
workload_identity_federation, client_secret, oidc_token and auth.get_token_command are mutually exclusive; configuring more than one is rejected, because there is no defensible precedence between them. With none of them set, the default azidentity chain runs. cache_credential_tokens is independent and wraps whichever of them is in use.
These are the same spec fields as the
Azure source plugin, so an identity that works there works here.
Incremental Syncing #
The plugin syncs incrementally per billing period, and requires
backend_options to be set (as shown above) to persist its position. Without
backend_options it logs a warning and re-reads every billing period on every run. See
Managing Incremental Tables for details.
Cost Management writes one folder per billing period, named YYYYMMDD-YYYYMMDD, and one folder per run inside it holding a manifest.json and the Parquet partitions. The plugin:
reads only the newest manifest in each billing period, so that older runs are not read alongside it. When two manifests share a blob timestamp, which can happen because Azure reports blob timestamps only to the second, the tie is broken on the submittedTime inside each manifest rather than on the blob name, since run folders are GUIDs and order arbitrarily. See the note on dataOverwriteBehavior below for when that can arise;
records, per billing period, the name, ETag and timestamp of the manifest it read, and reads the period again when any of them changes. Measured against a live export, every delivery writes a new run GUID folder, so the manifest name alone already differs between runs; the ETag is the guard for a manifest rewritten at a path that did not change;
reads each blob under the ETag it was listed with, so a blob rewritten between being listed and being read is rejected rather than mixed into the period alongside rows from another run;
skips a billing period whose manifest names a blob that is missing, or whose size does not match the byte count the manifest records. Azure does not document whether the manifest is written before or after the data files, so a run in that state is treated as still being delivered and is left for a later sync rather than synced in part;
skips a billing period that has data files but no manifest yet, for the same reason;
reads each blob under the ETag it was listed with, so a run replaced while the sync is reading it is skipped and picked up on the next run rather than mixing two runs into one period. This is not hypothetical: Azure deletes the previous run folder wholesale as soon as a new one lands, so a blob can disappear between being listed and being read;
fails, rather than skipping, when a manifest names blobs of which none exist. A run mid-delivery has written some of its blobs, so none at all means the manifest does not describe this container's layout. Left as a warning that would leave the sync permanently green and permanently empty.
Watermarks reach the state backend only after the whole export has been read, and are flushed only once the sync succeeds. A sync that fails part way therefore marks nothing as synced, and the next run re-reads those periods instead of skipping past unsent rows.
A note on dataOverwriteBehavior #
Cost Management exports carry one of two delivery modes, and the plugin reads both. Azure's documented default is CreateNewReport; the resolved mode is logged at the start of every sync.
OverwritePreviousReport — measured against a live export over four consecutive runs: each run writes a new run GUID folder under the billing period and the previous folder is deleted wholesale, so the period never holds more than one run. The "newest manifest wins" selection and its submittedTime tie-break therefore never come into play in this mode.
CreateNewReport — Azure retains each run's folder, so a billing period accumulates several manifests and the newest has to be chosen. This is the only mode in which the tie-break is reachable, and then only when two runs share a blob timestamp to the second.
No export in this mode has been read from real Azure data. Both exports used for validation are OverwritePreviousReport. The multi-run selection and the tie-break are covered by the plugin's own tests, so treat them as designed-and-tested rather than observed.
Row identity and deduplication #
FOCUS defines no line item id. Azure's FOCUS 1.0, 1.0r2 and 1.2-preview datasets carry no per-row unique identifier either; the x_ChargeId column that did exist in the retired FOCUS 1.0-preview dataset was dropped. Each row is therefore identified by a deterministic _cq_id hashed from a tuple of dimension columns.
A column is part of that tuple only if it satisfies both rules:
It is immutable for a given logical charge. Azure restates a whole billing period on every delivery. A column that legitimately changes between deliveries would turn a correction into a second row rather than an update.
It exists in every FOCUS data version this plugin reads. A column present only in 1.2-preview would re-identify every row the moment an export is moved to that data version, and since a re-delivered period is upserted rather than replaced, every charge would then be counted twice.
The components are:
export_scope, export_name, billing_account_id, x_billing_profile_id, sub_account_id, x_invoice_section_id, billing_period_start, charge_period_start, charge_period_end, charge_category, charge_class, charge_frequency, commitment_discount_id, commitment_discount_status, resource_id, x_resource_group_name, service_name, region_id, sku_id, sku_price_id, x_sku_meter_id, x_sku_offer_id, x_sku_order_id, x_sku_part_number, x_sku_tier, pricing_category, x_pricing_subcategory, pricing_unit, consumed_unit, tags_hash, x_sku_details_hash.
tags_hash and x_sku_details_hash are stable hashes of the tags and x_sku_details maps, computed over their keys in sorted order so the value does not depend on map ordering. They are in the identity because Azure uses both to separate charges that are otherwise identical:
One resource, several tag sets. An AKS scale set emits separate rows for the same resource, meter and day whose only differing dimension is the tag set, for example an aks-managed-coordination tag present on one and not the other. Validating against a real export, leaving tags out of the identity collapsed 7 of 412 rows and silently lost their cost.
One resource id, several instances. Scale set instances bill under the single VMSS resource id and are told apart only by x_SkuDetails.VMName. That accounted for the last remaining collision.
With both hashes in the identity, a real 412-row export produces 412 distinct _cq_id values.
Retagging a resource therefore produces a new row rather than updating the existing one. This is not a defect to work around, and it does not leave stale rows behind. Azure does not re-attribute history: a tag change applies from the moment it takes effect, earlier days keep the tags they were billed under, and the day of the change carries both tag sets because the resource was untagged for part of it and tagged for the rest. The separate rows are separate charges, and that is how Azure bills them.
Measured by retagging a storage account and diffing exports across 17 hours: the nine charge days before the change were byte-identical and untagged, and the day of the change went from 4 untagged rows to 7 untagged plus 7 tagged. Nothing was rewritten and nothing disappeared. Note that the change took somewhere between 90 minutes and 17 hours to appear in the export, so a same-day export shows nothing and proves nothing.
What the rules exclude #
Excluded by rule 1, because Azure restates them: costs, prices and quantities; charge_description, which changes when Microsoft renames a meter; x_cost_allocation_rule_name and x_cost_center, which change when an administrator renames a rule or reassigns a cost centre; and resource_name, so renaming a resource does not fork its history.
Excluded by rule 2, because they exist only in 1.2-preview: x_amortization_class, invoice_id, sku_meter, service_subcategory, x_service_model, x_sku_plan_name, capacity_reservation_id, capacity_reservation_status, and sku_price_details. Note that x_sku_details is documented as deprecated in favour of sku_price_details; when Azure stops populating it, x_sku_details_hash will need replacing with a hash of its successor.
export_scope and export_name are part of the identity, so the same subscription covered by two different exports is kept apart rather than silently merged. This is deliberate, and it has a consequence worth stating: a subscription-scoped export and a billing-account-scoped export that both cover that subscription each contribute their own copy of every charge. Validated against two live exports, the overlapping subscription produced 412 rows twice, 824 in total, all with distinct _cq_id. Anything summing this table across several exports must group or filter by export_name, or it double-counts the overlap.
Cost columns are exact decimals #
Cost Management writes the timestamp columns as Parquet INT96, the legacy timestamp encoding, and the monetary columns as DECIMAL(precision=38, scale=18), and real values use that scale: 0.483900000000000408 and 0.000003208648413419 both carry more significant digits than a float64 holds. These columns are therefore typed as DECIMAL(38,18) in the destination and carried through the plugin as exact text, never parsed into a float.
The columns are billed_cost, consumed_quantity, contracted_cost, contracted_unit_price, effective_cost, list_cost, list_unit_price, pricing_quantity, x_billed_cost_in_usd, x_billed_unit_price, x_contracted_cost_in_usd, x_effective_cost_in_usd, x_effective_unit_price, x_list_cost_in_usd, x_partner_credit_rate and x_pricing_block_size.
This differs deliberately from the AWS Cost and Usage plugin, which uses float64. AWS delivers those columns as native Parquet doubles, so a float is lossless there; Azure delivers exact decimal, so a float would discard digits on every cost figure and the error would compound under the summation a cost table exists to do.
Known limitations #
The multi-partition path is covered only by fixtures. Azure splits an export into several Parquet files once the data is large enough, and the plugin reads whatever the manifest's blobs[] lists, conditions each download on that blob's ETag, and holds one checkpoint across all of them. Both exports available for validation came back with blobCount: 1, because the tenant's entire Azure spend is orders of magnitude below Azure's partition threshold; a wider timeframe does not change that. So that path is exercised by the plugin's own test fixtures, which encode what we believe a multi-part export looks like rather than one that was observed. A mistake in that belief would not show up in the tests. Re-check it against a real multi-part export when one is available.
Deleted rows are not cleaned up. A row present in one delivery of a billing period and absent from the next is not removed from the destination, because a re-delivered period is upserted rather than replaced. Restatements that change an amount are handled; restatements that drop a row entirely are not.
A few extension columns are float64. x_billing_exchange_rate, x_sku_term and commitment_discount_quantity are floats rather than decimals. None of them is a cost, so the imprecision does not compound through a sum over cost columns.
One table across data versions. The table carries the FOCUS 1.0 and 1.0r2 columns plus the columns added by 1.2-preview, so the table shape does not change when you move an export to a newer data version, and no primary key component is version-dependent. Columns the configured data version does not emit arrive as the zero value for their type, not as NULL: an empty string for text columns and 0 for numeric ones. WHERE x_amortization_class IS NULL therefore matches nothing; compare against '' instead. The three columns 1.2-preview removed (x_invoice_id, x_invoice_issuer_id, x_sku_term) stay in the table and are empty for 1.2-preview exports.
0 is not distinguishable from "not emitted" in a numeric column, for the same reason. Cost columns are always emitted by Azure, so this matters mainly for the optional extension columns.
Only the supported FOCUS data versions. 1.0, 1.0r2, 1.2 and 1.2-preview are accepted. The retired 1.0-preview(v1) dataset uses largely different column names, so it is rejected at configuration time rather than decoding into empty rows.
Only Parquet. CSV exports are rejected at configuration time. Cost Management enables file partitioning for these exports and it cannot be turned off, so the manifest is always present.
Spec #
exports (
[]Export) (required)
The Cost Management FOCUS exports to sync. At least one is required.
cloud_name (string) (optional) (default: AzurePublic)
The Azure cloud to use. One of AzurePublic, AzureGovernment or AzureChina. It selects both the Azure Resource Manager endpoint and the blob storage endpoint suffix.
workload_identity_federation (
WorkloadIdentityFederation) (optional)
Keyless authentication with a federated OIDC token file. See
Authentication.
client_secret (
ClientSecret) (optional)
Microsoft Entra ID application credentials. See
Authentication.
oidc_token (string) (optional)
Deprecated, use
workload_identity_federation. An OIDC token to authenticate with. See
Authentication.
Advanced authentication, for scenarios the default credential chain cannot cover. See
Authentication.
cache_credential_tokens (boolean) (optional) (default: false)
Cache access tokens in memory and share them across the Cost Management and blob storage clients. See
Authentication.
rows_per_record (integer) (optional) (default: 500)
Number of rows packed into a single Apache Arrow record sent to the destination.
concurrency (integer) (optional) (default: 50)
The best effort maximum number of Go routines to use, which also bounds how many blobs are downloaded at once. Each download is buffered to a temporary file, so peak temporary disk use is roughly concurrency multiplied by the largest partition. Azure caps export partitions just under 1 GB, so lower this if the machine running the sync has little scratch space. Negative values mean no limit.
export #
scope (string) (required)
The scope the export is defined at, for example subscriptions/00000000-0000-0000-0000-000000000000 or providers/Microsoft.Billing/billingAccounts/1234567. Leading and trailing slashes are ignored.
name (string) (required)
The name of the export, as it appears in Cost Management > Exports.
sync_from (timestamp or relative time) (optional)
If specified, only billing periods whose last day falls on or after this date are synced. Accepts RFC3339 timestamps or relative times (for example, 90 days ago). The comparison is by date, so a time of day in the value does not drop the period containing it. Useful when a storage account holds years of history and you only want recent periods.
blob_service_url (string) (optional)
Overrides the blob service URL derived from the export's storage account resource id, for example https://myaccount.blob.core.windows.net. Set this when the plugin identity cannot read the storage account resource, or to point the plugin at an emulator.
workload_identity_federation #
token_file_path (string) (required)
Path to a file containing the federated OIDC token (JWT).
client_id (string) (required)
The client ID of the Microsoft Entra ID application or user-assigned managed identity to authenticate as.
tenant_id (string) (required)
The Microsoft Entra ID tenant ID of the application.
additionally_allowed_tenants ([]string) (optional)
Additional tenants the credential may acquire tokens for; * allows any tenant the application is registered in.
disable_instance_discovery (boolean) (optional) (default: false)
Skip the Microsoft Entra instance metadata request. Set only in Azure Stack or other disconnected or private clouds.
auth #
get_token_command (object) (required)
command (string) (required)
The command to run to get a token. It must print JSON to stdout with a token (string) and an expires_on (Unix epoch seconds) field.
args ([]string) (optional)
Arguments to pass to the command. The requested scopes are appended to them.
client_secret #
tenant_id (string) (required)
The Microsoft Entra ID tenant ID of the application.
client_id (string) (required)
The client ID of the Microsoft Entra ID application.
client_secret (string) (required)
The client secret of the Microsoft Entra ID application.
additionally_allowed_tenants ([]string) (optional)
Additional tenants the credential may acquire tokens for.
disable_instance_discovery (boolean) (optional) (default: false)
Skip the Microsoft Entra instance metadata request.