• Docs
  • Getting Started
  • Getting started with Azure

Getting Started with Azure

Download and Install

You can download the precompiled binary from releases, or using CLI:

curl -L https://versions.cloudquery.io/latest/v1/cloudquery_linux_x86_64 -o cloudquery
chmod a+x cloudquery
curl -L https://versions.cloudquery.io/latest/v1/cloudquery_linux_arm64 -o cloudquery
chmod a+x cloudquery
brew install cloudquery/tap/cloudquery
 
# After initial install you can upgrade the version via:
# brew upgrade cloudquery
curl -L https://versions.cloudquery.io/latest/v1/cloudquery_darwin_x86_64 -o cloudquery
chmod a+x cloudquery
curl -L https://versions.cloudquery.io/latest/v1/cloudquery_darwin_arm64 -o cloudquery
chmod a+x cloudquery
curl -L https://versions.cloudquery.io/latest/v1/cloudquery_windows_x86_64.exe -o cloudquery.exe
Invoke-WebRequest https://versions.cloudquery.io/latest/v1/cloudquery_windows_x86_64.exe -o cloudquery.exe

Running

Init command

After installing CloudQuery, you need to generate a cloudquery.yml file that will describe which cloud provider you want to use and which resources you want CloudQuery to ETL:

cloudquery init azure
 
# cloudquery init azure aws # This will generate a config containing azure and aws providers
# cloudquery init --help # Show all possible auto generated configs and flags

All official and approved community plugins are listed here with their respective documentation.

Spawn or connect to a Database

CloudQuery needs a PostgreSQL database (>=10). You can either spawn a local one (usually good for development and local testing) or connect to an existing one.

By default, cloudquery will try to connect to the database postgres on localhost:5432 with username postgres and password pass. After installing docker, you can create such a local postgres instance with:

docker run --name cloudquery_postgres -p 5432:5432 -e POSTGRES_PASSWORD=pass -d postgres

If you are running postgres at a different location or with different credentials, you need to edit cloudquery.yml - see the Connect to an Existing Database tab.

CloudQuery connects to the postgres database that is defined in the cloudquery.yml's connection section. Edit this section to configure the location and credentials of your postgres database.

cloudquery:
  ...
  ...
 
  connection:
    type: postgres
    username: postgres
    password: pass
    host: localhost
    port: 5432
    database: postgres
    sslmode: disable

Authenticate with Azure

CloudQuery needs to be authenticated with your Azure account in order to fetch information about your cloud setup.

You can either authenticate with az login (when running cloudquery locally), or by using a "service principal" and exporting environment variables (appropriate for automated deployments).

First, install the Azure CLI (az). Then, login with the Azure CLI:

az login

You will need to create a service principal for CloudQuery to use:

Creating a service principal

First, install the Azure CLI (az).

Then, login with the Azure CLI:

az login

Then, create the service principal cloudquery will use to access your cloud deployment

💡

Warning The output of az ad sp create-for-rbac contains credentials that you must protect - Make sure to handle with appropriate care.

export SUBSCRIPTION_ID=<YOUR_SUBSCRIPTION_ID>

az account set --subscription $SUBSCRIPTION_ID
az provider register --namespace 'Microsoft.Security'

# Create a service-principal for cloudquery
az ad sp create-for-rbac --name cloudquery-sp --scopes /subscriptions/$SUBSCRIPTION_ID --role Reader
set SUBSCRIPTION_ID=<YOUR_SUBSCRIPTION_ID>
 
az account set --subscription %SUBSCRIPTION_ID%
az provider register --namespace Microsoft.Security
 
# Create a service-principal for cloudquery
az ad sp create-for-rbac --name cloudquery-sp --scopes /subscriptions/%SUBSCRIPTION_ID% --role Reader
$Env:SUBSCRIPTION_ID="<YOUR_SUBSCRIPTION_ID>"
 
az account set --subscription $Env:SUBSCRIPTION_ID
az provider register --namespace Microsoft.Security
 
# Create a service-principal for cloudquery
az ad sp create-for-rbac --name cloudquery-sp --scopes /subscriptions/$Env:SUBSCRIPTION_ID --role Reader

(you can, of course, choose any name you'd like for your service-principal, cloudquery-sp is just an example. If the service principal doesn't exist it will create a new one, otherwise it will update an existing one)

The output of az ad sp create-for-rbac should look like this:

{
  "appId": <YOUR AZURE_CLIENT_ID>,
  "displayName": "cloudquery-sp",
  "password": <YOUR AZURE_CLIENT_SECRET>,
  "tenant": <YOUR AZURE_TENANT_ID>
}
💡

You can find out more about authentication with Azure here and at Azure's documentation for the golang sdk.

Exporting environment variables

Next, you need to export the environment variables that cloudquery will use to fetch your cloud configuration. Copy them from the output of az ad sp create-for-rbac (or, take the opportunity to show off your jq-foo):

  • AZURE_TENANT_ID is tenant in the json.
  • AZURE_CLIENT_ID is appId in the json.
  • AZURE_CLIENT_SECRET is password in the json.
export AZURE_TENANT_ID=<YOUR AZURE_TENANT_ID>
export AZURE_CLIENT_ID=<YOUR AZURE_CLIENT_ID>
export AZURE_CLIENT_SECRET=<YOUR AZURE_CLIENT_SECRET>
export AZURE_SUBSCRIPTION_ID=$SUBSCRIPTION_ID
set AZURE_TENANT_ID=<YOUR AZURE_TENANT_ID>
set AZURE_CLIENT_ID=<YOUR AZURE_CLIENT_ID>
set AZURE_CLIENT_SECRET=<YOUR AZURE_CLIENT_SECRET>
set AZURE_SUBSCRIPTION_ID=%SUBSCRIPTION_ID%
$Env:AZURE_TENANT_ID="<YOUR AZURE_TENANT_ID>"
$Env:AZURE_CLIENT_ID="<YOUR AZURE_CLIENT_ID>"
$Env:AZURE_CLIENT_SECRET="<YOUR AZURE_CLIENT_SECRET>"
$Env:AZURE_SUBSCRIPTION_ID=$Env:SUBSCRIPTION_ID

Fetch Command

Once cloudquery.yml is generated and you are authenticated with Azure, run the following command to fetch the resources.

cloudquery fetch
# cloudquery fetch --help # Show all possible fetch flags

Exploring and Running Queries

Once CloudQuery fetched the resources, you can explore your cloud infrastructure with SQL!

You can use psql to connect to your postgres instance (of course, you need to change the connection-string to match the location and credentials of your database):

psql "postgres://postgres:pass@localhost:5432/postgres?sslmode=disable"

If you opted for running the PostgreSQL server in a docker as described above, you can also run psql directly from the docker instead of installing it on your machine:

docker exec -it cloudquery_postgres psql -U postgres

Schema and tables for GCP are available here.

A few example queries for Azure:

find all mysql servers:

SELECT * FROM azure_mysql_servers

find storage accounts which allow non https traffic

SELECT * from azure_storage_accounts where enable_https_traffic_only = false

Cloudquery Policies

CloudQuery Policies allow users to write security, governance, cost, and compliance rules with SQL, and run them with psql. You can read more about policies here.

Next Steps

Visit the Azure plugin documentation to read more about it, explore the supported tables and learn about advanced configurations.