Back to plugin list
azure
Official
Open-core

Azure

The CloudQuery Azure source plugin extracts information from many of the supported services by Microsoft Azure and loads it into any supported CloudQuery destination. Some tables are marked as premium and have a price per 1M rows synced.

Publisher

cloudquery

Repositorygithub.com
Latest version

v12.0.1

Type

Source

Platforms
Date Published

Mar 12, 2024

Price per 1M rows

$10 (premium tables)

free quota

10M rows

Set up process


brew install cloudquery/tap/cloudquery

1. Download CLI and login

See installation options

2. Create source and destination configs

Plugin configuration

cloudquery sync azure.yml postgresql.yml

3. Run the sync

CloudQuery sync

Overview

The CloudQuery Azure source plugin extracts information from many of the supported services by Microsoft Azure and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).
This plugin is an open-core plugin, which means some tables are free and some premium. You can find a list of all premium tables below.

Authentication

The Azure plugin uses DefaultAzureCredential to authenticate.
DefaultAzureCredential will attempt to authenticate via different mechanisms in order, stopping when one succeeds. The order is described in detail in the Azure SDK documentation.
For getting started quickly with the Azure plugin, we recommend using a service principal and exporting environment variables or using az login. The latter is highly discouraged for production use as it requires spawning a new Azure CLI process each time an authentication token is needed and causes memory and performance issues.

Authentication with Environment Variables

You will need to create a service principal for the plugin to use:
Creating a service principal
First, install the Azure CLI (az).
Then, login with the Azure CLI:
az login
Then, create the service principal the plugin will use to access your cloud deployment. WARNING: The output of az ad sp create-for-rbac contains credentials that you must protect - Make sure to handle with appropriate care. This example uses bash - The commands for CMD and PowerShell are similar.
export SUBSCRIPTION_ID=<YOUR_SUBSCRIPTION_ID>
az account set --subscription $SUBSCRIPTION_ID
az provider register --namespace 'Microsoft.Security'

# Create a service-principal for the plugin
az ad sp create-for-rbac --name cloudquery-sp --scopes /subscriptions/$SUBSCRIPTION_ID --role Reader
(you can, of course, choose any name you'd like for your service-principal, cloudquery-sp is just an example. If the service principal doesn't exist it will create a new one, otherwise it will update an existing one)
The output of az ad sp create-for-rbac should look like this:
{
  "appId": "YOUR AZURE_CLIENT_ID",
  "displayName": "cloudquery-sp",
  "password": "YOUR AZURE_CLIENT_SECRET",
  "tenant": "YOUR AZURE_TENANT_ID"
}
Exporting environment variables
Next, you need to export the environment variables that plugin will use to sync your cloud configuration. Copy them from the output of az ad sp create-for-rbac (or, take the opportunity to show off your jq-foo). The example shows how to export environment variables for Linux - exporting for CMD and PowerShell is similar.
  • AZURE_TENANT_ID is tenant in the JSON.
  • AZURE_CLIENT_ID is appId in the JSON.
  • AZURE_CLIENT_SECRET is password in the JSON.
export AZURE_TENANT_ID=<YOUR AZURE_TENANT_ID>
export AZURE_CLIENT_ID=<YOUR AZURE_CLIENT_ID>
export AZURE_CLIENT_SECRET=<YOUR AZURE_CLIENT_SECRET>
export AZURE_SUBSCRIPTION_ID=$SUBSCRIPTION_ID

Authentication with az login

First, install the Azure CLI (az). Then, login with the Azure CLI:
az login
You are now authenticated!

Query Examples

Find all MySQL servers

SELECT * FROM azure_mysql_servers;

Find storage accounts that are allowing non-HTTPS traffic

SELECT * from azure_storage_accounts where enable_https_traffic_only = false;

Find all expired key vaults

SELECT * from azure_keyvault_vault_keys where attributes_expires >= extract(epoch from now()) * 1000;

List the Memory and vCPUs of all available Azure Compute VM types

SELECT
 distinct(vm.name),
 vcpus.capability_value AS "vCPUs",
 memory.capability_value AS "Memory"
FROM
 azure_compute_skus vm
 CROSS JOIN LATERAL (
   SELECT (caps ->> 'value') AS capability_value
   FROM jsonb_array_elements(vm.capabilities) caps
   WHERE (caps ->> 'name') = 'vCPUs'
 ) vcpus
 CROSS JOIN LATERAL (
   SELECT (caps ->> 'value') AS capability_value
   FROM jsonb_array_elements(vm.capabilities) caps
   WHERE (caps ->> 'name') = 'MemoryGB'
 ) memory
WHERE
 vm.resource_type = 'virtualMachines' order by name;
Results:
+---------------------------+-------+--------+
| name                      | vCPUs | Memory |
|---------------------------+-------+--------|
| Basic_A0                  | 1     | 0.75   |
| Basic_A1                  | 1     | 1.75   |
| Basic_A2                  | 2     | 3.5    |
| Basic_A3                  | 4     | 7      |
| Basic_A4                  | 8     | 14     |
| Standard_A0               | 1     | 0.75   |
| Standard_A1               | 1     | 1.75   |
| Standard_A1_v2            | 1     | 2      |
... (truncated)


Configuration

CloudQuery Azure Source Plugin Configuration Reference

Example

This example connects a single Azure subscription to a single destination. The (top level) source spec section is described in the Source Spec Reference.
kind: source
spec:
  # Source spec section
  name: "azure"
  path: "cloudquery/azure"
  registry: "cloudquery"
  version: "v12.0.1"
  destinations: ["postgresql"]
  tables: ["azure_compute_virtual_machines"]
  spec:
    # Optional parameters
    # subscriptions: []
    # cloud_name: ""
    # concurrency: 50000
    # discovery_concurrency: 400
    # skip_subscriptions: []
    # normalize_ids: false
    # oidc_token: ""
    # retry_options:
    #   max_retries: 3
    #   try_timeout_seconds: 0
    #   retry_delay_seconds: 4
    #   max_retry_delay_seconds: 60

Azure Spec

This is the (nested) spec used by the Azure source plugin.
  • subscriptions ([]string) (default: empty. Will use all visible subscriptions)
    Specify which subscriptions to sync data from.
  • cloud_name (string) (default: empty)
    The name of the cloud environment to use. Possible values are AzureCloud, AzureChinaCloud, AzureUSGovernment. See the Azure CLI documentation for more information.
  • concurrency (int) (default: 50000):
    The best effort maximum number of Go routines to use. Lower this number to reduce memory usage.
  • discovery_concurrency (int) (default: 400)
    During initialization the Azure source plugin discovers all resource groups and enabled resource providers per subscription, to be used later on during the sync process. The plugin runs the discovery process in parallel. This setting controls the maximum number of concurrent requests to the Azure API during discovery. Only accounts with many subscriptions should require modifying this setting, to either lower it to avoid network errors, or to increase it to speed up the discovery process.
  • skip_subscriptions ([]string) (default: empty)
    A list of subscription IDs that CloudQuery will skip syncing. This is useful if CloudQuery is discovering the list of subscription IDs and there are some subscriptions that you want to not even attempt syncing.
  • normalize_ids (bool) (default: false)
    Enabling this setting will force all id column values to be lowercase. This is useful to avoid case sensitivity and uniqueness issues around the id primary keys
  • oidc_token (string) (default: empty)
    An OIDC token can be used to authenticate with Azure instead of AZURE_CLIENT_SECRET. This is useful for Azure AD workload identity federation. When using this option, the AZURE_CLIENT_ID and AZURE_TENANT_ID environment variables must be set.
  • retry_options (RetryOptions) (default: empty)
    Retry options to pass to the Azure Go SDK, see more details here

retry_options

  • max_retries (integer) (default: 3)
Described in the Azure Go SDK.
  • try_timeout_seconds (integer) (default: 0)
Disabled by default. Described in the Azure Go SDK.
  • retry_delay_seconds (integer) (default: 4)
Described in the Azure Go SDK.
  • max_retry_delay_seconds (integer) (default: 60)
Described in the Azure Go SDK.
  • status_codes ([]integer) (default: null)
Described in the Azure Go SDK.
The default of null uses the default status codes. An empty value disables retries for HTTP status codes.


Premium Tables

  • azure_advisor_recommendation_metadata
  • azure_advisor_recommendations
  • azure_advisor_suppressions
  • azure_analysisservices_servers
  • azure_apimanagement_service
  • azure_appcomplianceautomation_reports
  • azure_appconfiguration_configuration_stores
  • azure_applicationinsights_components
  • azure_applicationinsights_web_tests
  • azure_authorization_classic_administrators
  • azure_authorization_role_assignments
  • azure_automation_account
  • azure_azurearcdata_postgres_instances
  • azure_azurearcdata_sql_managed_instances
  • azure_azurearcdata_sql_server_instances
  • azure_batch_account
  • azure_billing_accounts
  • azure_billing_enrollment_accounts
  • azure_billing_periods
  • azure_botservice_bots
  • azure_cdn_edge_nodes
  • azure_cdn_endpoints
  • azure_cdn_profiles
  • azure_cdn_rule_sets
  • azure_cdn_security_policies
  • azure_cognitiveservices_account_deployments
  • azure_cognitiveservices_account_models
  • azure_cognitiveservices_account_private_endpoint_connections
  • azure_cognitiveservices_account_private_link_resources
  • azure_cognitiveservices_account_skus
  • azure_cognitiveservices_account_usages
  • azure_cognitiveservices_accounts
  • azure_cognitiveservices_commitment_plans
  • azure_cognitiveservices_deleted_accounts
  • azure_confluent_marketplace_agreements
  • azure_connectedvmware_clusters
  • azure_connectedvmware_datastores
  • azure_connectedvmware_hosts
  • azure_connectedvmware_resource_pools
  • azure_connectedvmware_v_centers
  • azure_connectedvmware_virtual_machine_templates
  • azure_connectedvmware_virtual_machines
  • azure_connectedvmware_virtual_networks
  • azure_consumption_billing_account_balances
  • azure_consumption_billing_account_budgets
  • azure_consumption_billing_account_charges
  • azure_consumption_billing_account_events
  • azure_consumption_billing_account_legacy_usage_details
  • azure_consumption_billing_account_lots
  • azure_consumption_billing_account_marketplaces
  • azure_consumption_billing_account_modern_usage_details
  • azure_consumption_billing_account_reservation_recommendations
  • azure_consumption_billing_account_tags
  • azure_consumption_billing_profile_reservation_details
  • azure_consumption_billing_profile_reservation_recommendations
  • azure_consumption_billing_profile_reservation_summaries
  • azure_consumption_billing_profile_reservation_transactions
  • azure_consumption_subscription_budgets
  • azure_consumption_subscription_legacy_usage_details
  • azure_consumption_subscription_marketplaces
  • azure_consumption_subscription_price_sheets
  • azure_consumption_subscription_reservation_recommendations
  • azure_consumption_subscription_tags
  • azure_containerinstance_container_groups
  • azure_containerregistry_registries
  • azure_containerservice_managed_cluster_upgrade_profiles
  • azure_containerservice_managed_clusters
  • azure_containerservice_snapshots
  • azure_costmanagement_view_queries
  • azure_costmanagement_views
  • azure_customerinsights_hubs
  • azure_dashboard_grafana
  • azure_databox_jobs
  • azure_databricks_access_connectors
  • azure_databricks_operations
  • azure_databricks_outbound_network_dependencies_endpoints
  • azure_databricks_private_endpoint_connections
  • azure_databricks_private_link_resources
  • azure_databricks_virtual_network_peerings
  • azure_databricks_workspaces
  • azure_datacatalog_catalogs
  • azure_datadog_marketplace_agreements
  • azure_datadog_monitors
  • azure_datafactory_factories
  • azure_datalakeanalytics_accounts
  • azure_datalakestore_accounts
  • azure_datamigration_services
  • azure_datashare_accounts
  • azure_desktopvirtualization_host_pools
  • azure_devhub_workflow
  • azure_devops_pipeline_template_definitions
  • azure_dns_record_sets
  • azure_dns_zones
  • azure_dnsresolver_dns_forwarding_rulesets
  • azure_dnsresolver_dns_resolvers
  • azure_elastic_monitors
  • azure_engagementfabric_accounts
  • azure_eventgrid_topic_types
  • azure_eventhub_clusters
  • azure_eventhub_namespace_network_rule_sets
  • azure_eventhub_namespaces
  • azure_frontdoor_front_doors
  • azure_frontdoor_managed_rule_sets
  • azure_frontdoor_network_experiment_profiles
  • azure_hanaonazure_sap_monitors
  • azure_hdinsight_clusters
  • azure_healthbot_bots
  • azure_healthcareapis_services
  • azure_hybridcompute_private_link_scopes
  • azure_hybriddatamanager_data_managers
  • azure_keyvault_keyvault
  • azure_keyvault_keyvault_keys
  • azure_keyvault_keyvault_managed_hsms
  • azure_keyvault_keyvault_secrets
  • azure_kusto_clusters
  • azure_logic_workflows
  • azure_machinelearningservices_workspaces
  • azure_maintenance_configurations
  • azure_maintenance_public_maintenance_configurations
  • azure_managementgroups_entities
  • azure_managementgroups_management_groups
  • azure_marketplace_private_store
  • azure_monitor_action_groups
  • azure_monitor_activity_log_alerts
  • azure_monitor_autoscale_settings
  • azure_monitor_diagnostic_settings
  • azure_monitor_log_profiles
  • azure_monitor_metric_alerts
  • azure_monitor_private_link_scopes
  • azure_monitor_resources
  • azure_monitor_scheduled_query_rules
  • azure_monitor_subscription_diagnostic_settings
  • azure_monitor_tenant_activity_log_alerts
  • azure_monitor_tenant_activity_logs
  • azure_network_application_gateways
  • azure_network_application_security_groups
  • azure_network_azure_firewall_fqdn_tags
  • azure_network_azure_firewalls
  • azure_network_bastion_hosts
  • azure_network_custom_ip_prefixes
  • azure_network_ddos_protection_plans
  • azure_network_dscp_configuration
  • azure_network_express_route_circuit_authorizations
  • azure_network_express_route_circuit_peerings
  • azure_network_express_route_circuits
  • azure_network_express_route_gateways
  • azure_network_express_route_ports
  • azure_network_express_route_ports_locations
  • azure_network_express_route_service_providers
  • azure_network_firewall_policies
  • azure_network_interface_ip_configurations
  • azure_network_interfaces
  • azure_network_ip_allocations
  • azure_network_ip_groups
  • azure_network_load_balancers
  • azure_network_nat_gateways
  • azure_network_private_endpoints
  • azure_network_private_link_services
  • azure_network_profiles
  • azure_network_public_ip_addresses
  • azure_network_public_ip_prefixes
  • azure_network_route_filters
  • azure_network_route_tables
  • azure_network_security_groups
  • azure_network_security_partner_providers
  • azure_network_service_endpoint_policies
  • azure_network_subscription_network_manager_connections
  • azure_network_virtual_appliances
  • azure_network_virtual_hubs
  • azure_network_virtual_network_gateway_connections
  • azure_network_virtual_network_gateways
  • azure_network_virtual_network_subnets
  • azure_network_virtual_network_taps
  • azure_network_virtual_networks
  • azure_network_virtual_routers
  • azure_network_virtual_wans
  • azure_network_vpn_gateways
  • azure_network_vpn_server_configurations
  • azure_network_vpn_sites
  • azure_network_watcher_flow_logs
  • azure_network_watchers
  • azure_network_web_application_firewall_policies
  • azure_networkfunction_azure_traffic_collectors_by_subscription
  • azure_nginx_deployments
  • azure_notificationhubs_namespaces
  • azure_operationalinsights_clusters
  • azure_operationalinsights_workspaces
  • azure_peering_service_countries
  • azure_peering_service_providers
  • azure_policy_assignments
  • azure_policy_data_policy_manifests
  • azure_policy_exemptions
  • azure_policy_set_definitions
  • azure_portal_list_tenant_configuration_violations
  • azure_portal_tenant_configurations
  • azure_powerbidedicated_capacities
  • azure_privatedns_private_zone_record_sets
  • azure_privatedns_private_zone_virtual_network_links
  • azure_privatedns_private_zones
  • azure_providerhub_provider_registrations
  • azure_redhatopenshift_open_shift_clusters
  • azure_redis_caches
  • azure_relay_namespaces
  • azure_reservations_reservation
  • azure_reservations_reservation_order
  • azure_resources_links
  • azure_resources_providers
  • azure_resources_resource_groups
  • azure_resources_resources
  • azure_role_management_policy_assignments
  • azure_saas_resources
  • azure_search_services
  • azure_security_adaptive_application_controls
  • azure_security_alerts
  • azure_security_alerts_suppression_rules
  • azure_security_allowed_connections
  • azure_security_applications
  • azure_security_assessments
  • azure_security_auto_provisioning_settings
  • azure_security_automations
  • azure_security_connectors
  • azure_security_contacts
  • azure_security_discovered_security_solutions
  • azure_security_external_security_solutions
  • azure_security_governance_rule
  • azure_security_jit_network_access_policies
  • azure_security_locations
  • azure_security_pricings
  • azure_security_regulatory_compliance_standards
  • azure_security_secure_score_control_definitions
  • azure_security_secure_score_controls
  • azure_security_secure_scores
  • azure_security_settings
  • azure_security_solutions
  • azure_security_sub_assessments
  • azure_security_tasks
  • azure_security_topology
  • azure_security_workspace_settings
  • azure_servicebus_namespace_topic_authorization_rules
  • azure_servicebus_namespace_topic_rule_access_keys
  • azure_servicebus_namespace_topics
  • azure_servicebus_namespaces
  • azure_streamanalytics_streaming_jobs
  • azure_subscription_subscription_locations
  • azure_subscription_subscriptions
  • azure_subscription_tenants
  • azure_support_services
  • azure_support_tickets
  • azure_synapse_private_link_hubs
  • azure_synapse_workspaces
  • azure_trafficmanager_profiles
  • azure_windowsiot_services
  • azure_workloads_monitors


Subscribe to product updates

Be the first to know about new features.