Using CloudQuery Docker Registry Plugins Inside a Containerized Environment

CloudQuery CLI uses the Docker CLI (opens in a new tab) and Engine API (opens in a new tab) to run Docker plugins. When using the CloudQuery CLI Docker image, Docker plugins don't work out of the box, as the Docker CLI and Engine API are not available in the container by default. This guide will show you how to run Docker plugins with the CloudQuery CLI Docker image using Docker Compose.

Prerequisites

Setup

Run cloudquery login to authenticate with the CloudQuery registry.
Create a file named spec.yml with the configuration for the Docker plugin. We will use this spec file to pull the Docker plugin image locally from the private CloudQuery Docker registry. The airtable and postgresql plugins are used as an example.

kind: source
spec:
  name: airtable
  registry: docker
  path: docker.cloudquery.io/cloudquery/source-airtable:v2.2.0
  tables: ["*"]
  destinations: ["postgresql"]
---
kind: destination
spec:
  name: "postgresql"
  path: "cloudquery/postgresql"
  version: "v8.0.5"

Run cloudquery plugin install spec.yml to pull the Docker plugin image locally.

Running a Sync

Create a CloudQuery API Key (opens in a new tab) to be used with the Docker Compose file.
Create a docker-compose.yml file with the following content. The file configures the CLI docker image, the Docker plugin image, a PostgreSQL database and a configuration spec that sets up a connection between the CLI and the Docker plugin using gRPC.

version: '3.1'
services:
  cli:
    container_name: cli
    image: ghcr.io/cloudquery/cloudquery:latest
    command: ["sync", "/spec.yml", "--log-console", "--log-format", "json"]
    environment:
      CLOUDQUERY_API_KEY: ${CLOUDQUERY_API_KEY}
      # We reference this environment variable in the `spec.yml` config block below
      # Other plugins will require other environment variables
      AIRTABLE_ACCESS_TOKEN: ${AIRTABLE_ACCESS_TOKEN}
    configs:
      - spec.yml
    depends_on:
      airtable:
        condition: service_healthy
      postgres:
        condition: service_healthy
  airtable:
    container_name: airtable
    image: docker.cloudquery.io/cloudquery/source-airtable:v2.2.0
    # We use `cloudquery login` and `cloudquery plugin install spec.yml` to pull the image locally
    pull_policy: never
    restart: always
    healthcheck:
      # Docker plugins always run on port 7777
      test: ["CMD", "bash", "-c", "echo -n '' > /dev/tcp/localhost/7777"]
      interval: 5s
      timeout: 30s
      retries: 5
  postgres:
    container_name: postgres
    image: postgres:15
    restart: always
    environment:
      POSTGRES_PASSWORD: pass
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 5s
      timeout: 30s
      retries: 5
configs:
  spec.yml:
    content: |
      kind: source
      spec:
        name: airtable
        registry: grpc
        # Notice we use the container name as the host to connect via Docker internal DNS
        path: airtable:7777
        tables: ["*"]
        destinations: ["postgresql"]
        spec:
          access_token: "${AIRTABLE_ACCESS_TOKEN}"
      ---
      kind: destination
      spec:
        name: "postgresql"
        path: "cloudquery/postgresql"
        version: "v8.0.5"
        spec:
          # Notice we use the container name as the host to connect via Docker internal DNS
          connection_string: "postgresql://postgres:pass@postgres:5432/postgres?sslmode=disable"

Run CLOUDQUERY_API_KEY=<cloudquery-api-key> AIRTABLE_ACCESS_TOKEN=<airtable-access-token> docker compose up -d
You can check the logs of the CLI container to see the sync process. Run docker logs -f cli or docker logs -f airtable to see the logs.
To see the results, you can connect to the PostgreSQL database using your favorite client, for example, psql.

Cleanup

Run docker compose down to stop and remove the containers.

Arrow String Representation Managing Versions