Official

File source integration documentation

Name: CloudQuery File Integration
Author: CloudQuery

The CloudQuery File plugin syncs parquet files to any of the supported CloudQuery destinations

Publisher

cloudquery

Latest version

v1.8.6

Type

Source

Platforms

Date Published

Start Syncing

Overview Licenses

Overview #

The CloudQuery File plugin reads parquet files and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

kind: source
spec:
  name: file
  path: cloudquery/file
  registry: cloudquery
  version: "v1.8.6"
  tables: ["*"]
  destinations: ["postgresql"]

  # Learn more about the configuration options at https://cql.ink/file_source
  spec:
    files_dir: "/path/to/files-to-sync" # required. Path to the directory with files to sync
    # Optional parameters
    # rows_per_record: 500
    # concurrency: 50

File spec #

This is the (nested) spec used by the File source plugin.

files_dir (string) (required)
Path to the directory with files to sync. Only files with .parquet extension will be synced.
rows_per_record (integer) (optional) (default: 500)
Amount of rows to be packed into a single Apache Arrow record to be sent over the wire during sync.
concurrency (integer) (optional) (default: 50)
Number of files to sync in parallel. Negative values mean no limit.

Example with AWS Cost and Usage Reports #

AWS Cost and Usage Reports are stored in S3 as parquet files. The following example shows how to sync these files and AWS infrastructure data to a PostgreSQL database. To learn more about visualizing AWS Cost and Usage Reports, visit our dashboards page.

kind: source
spec:
  name: file
  path: cloudquery/file
  registry: cloudquery
  version: "v1.8.6"
  destinations: [postgresql]
  tables: ["*"]
  spec:
    files_dir: "/path/to/cost_and_usage_reports" # Update this value to the local directory with your AWS Cost and Usage Reports
---
kind: source
spec:
  name: aws
  path: cloudquery/aws
  registry: cloudquery
  version: "v35.6.0"
  destinations: [postgresql]
  tables: ["*"]
  skip_tables:
    - aws_ec2_vpc_endpoint_services 
    - aws_cloudtrail_events
    - aws_docdb_cluster_parameter_groups
    - aws_docdb_engine_versions
    - aws_ec2_instance_types
    - aws_elasticache_engine_versions
    - aws_elasticache_parameter_groups
    - aws_elasticache_reserved_cache_nodes_offerings
    - aws_elasticache_service_updates
    - aws_iam_group_last_accessed_details
    - aws_iam_policy_last_accessed_details
    - aws_iam_role_last_accessed_details
    - aws_iam_user_last_accessed_details
    - aws_neptune_cluster_parameter_groups
    - aws_neptune_db_parameter_groups
    - aws_rds_cluster_parameter_groups
    - aws_rds_db_parameter_groups
    - aws_rds_engine_versions
    - aws_servicequotas_services
---
kind: destination
spec:
  name: postgresql
  path: cloudquery/postgresql
  registry: cloudquery
  version: "v8.19.0"
  spec:
    connection_string: postgresql://postgres:pass@localhost:5432/postgres

Licenses #

The following tools / packages are used in this plugin:

Name	License
github.com/adrg/xdg	MIT
github.com/andybalholm/brotli	MIT
github.com/andybalholm/brotli/flate	BSD-3-Clause
github.com/apache/arrow-go/v18	Apache-2.0
github.com/apache/arrow-go/v18	BSD-3-Clause
github.com/apache/arrow/go/v13	Apache-2.0
github.com/apache/arrow/go/v13	BSL-1.0
github.com/apache/arrow/go/v13	MIT
github.com/apache/arrow/go/v13	BSD-2-Clause
github.com/apache/arrow/go/v13	BSD-3-Clause
github.com/apache/arrow/go/v13	LicenseRef-C-Ares
github.com/apache/arrow/go/v13	OpenSSL
github.com/apache/arrow/go/v13	Zlib
github.com/apache/arrow/go/v13	NCSA
github.com/apache/thrift/lib/go/thrift	Apache-2.0
github.com/apache/thrift/lib/go/thrift	GNU-All-permissive-Copying-License
github.com/apache/thrift/lib/go/thrift	BSD-3-Clause
github.com/apapsch/go-jsonmerge/v2	MIT
github.com/aws/aws-sdk-go-v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/config	Apache-2.0
github.com/aws/aws-sdk-go-v2/credentials	Apache-2.0
github.com/aws/aws-sdk-go-v2/feature/ec2/imds	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/configsources	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2	Apache-2.0
github.com/aws/aws-sdk-go-v2/internal/sync/singleflight	BSD-3-Clause
github.com/aws/aws-sdk-go-v2/internal/v4a	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/licensemanager	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/marketplacemetering	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/signin	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sso	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/ssooidc	Apache-2.0
github.com/aws/aws-sdk-go-v2/service/sts	Apache-2.0
github.com/aws/smithy-go	Apache-2.0
github.com/aws/smithy-go/internal/sync/singleflight	BSD-3-Clause
github.com/aws/smithy-go/transport/http/protocol/internal/json/internal/stdlib	BSD-3-Clause
github.com/bahlo/generic-list-go	BSD-3-Clause
github.com/buger/jsonparser	MIT
github.com/cenkalti/backoff/v5	MIT
github.com/cespare/xxhash/v2	MIT
github.com/cloudquery/cloudquery-api-go	MPL-2.0
github.com/cloudquery/codegen/jsonschema/docs	MPL-2.0
github.com/cloudquery/plugin-pb-go	MPL-2.0
github.com/cloudquery/plugin-sdk/v2/internal/glob	MIT
github.com/cloudquery/plugin-sdk/v2/schema	MIT
github.com/cloudquery/plugin-sdk/v2/types	MPL-2.0
github.com/cloudquery/plugin-sdk/v4	MPL-2.0
github.com/cloudquery/plugin-sdk/v4/glob	MIT
github.com/cloudquery/plugin-sdk/v4/scalar	MIT
github.com/davecgh/go-spew/spew	ISC
github.com/getsentry/sentry-go	MIT
github.com/ghodss/yaml	MIT
github.com/ghodss/yaml	BSD-3-Clause
github.com/go-logr/logr	Apache-2.0
github.com/go-logr/stdr	Apache-2.0
github.com/goccy/go-json	MIT
github.com/google/flatbuffers/go	Apache-2.0
github.com/google/uuid	BSD-3-Clause
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors	Apache-2.0
github.com/grpc-ecosystem/grpc-gateway/v2	BSD-3-Clause
github.com/hashicorp/go-cleanhttp	MPL-2.0
github.com/hashicorp/go-retryablehttp	MPL-2.0
github.com/invopop/jsonschema	MIT
github.com/klauspost/compress	MIT
github.com/klauspost/compress	Apache-2.0
github.com/klauspost/compress	BSD-3-Clause
github.com/klauspost/compress/internal/snapref	BSD-3-Clause
github.com/klauspost/compress/s2	BSD-3-Clause
github.com/klauspost/compress/snappy	BSD-3-Clause
github.com/klauspost/compress/zstd/internal/xxhash	MIT
github.com/klauspost/cpuid/v2	MIT
github.com/mattn/go-colorable	MIT
github.com/mattn/go-isatty	MIT
github.com/oapi-codegen/runtime	Apache-2.0
github.com/pb33f/ordered-map/v2	Apache-2.0
github.com/pierrec/lz4/v4	BSD-3-Clause
github.com/pmezard/go-difflib/difflib	BSD-3-Clause
github.com/rs/zerolog	MIT
github.com/samber/lo	MIT
github.com/santhosh-tekuri/jsonschema/v6	Apache-2.0
github.com/spf13/cobra	Apache-2.0
github.com/spf13/pflag	BSD-3-Clause
github.com/stretchr/testify	MIT
github.com/thoas/go-funk	MIT
github.com/zeebo/xxh3	BSD-2-Clause
go.opentelemetry.io/auto/sdk	Apache-2.0
go.opentelemetry.io/otel	Apache-2.0
go.opentelemetry.io/otel	BSD-3-Clause
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp	BSD-3-Clause
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp	BSD-3-Clause
go.opentelemetry.io/otel/exporters/otlp/otlptrace	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace	BSD-3-Clause
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp	Apache-2.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp	BSD-3-Clause
go.opentelemetry.io/otel/log	Apache-2.0
go.opentelemetry.io/otel/log	BSD-3-Clause
go.opentelemetry.io/otel/metric	Apache-2.0
go.opentelemetry.io/otel/metric	BSD-3-Clause
go.opentelemetry.io/otel/sdk	Apache-2.0
go.opentelemetry.io/otel/sdk	BSD-3-Clause
go.opentelemetry.io/otel/sdk/log	Apache-2.0
go.opentelemetry.io/otel/sdk/log	BSD-3-Clause
go.opentelemetry.io/otel/sdk/metric	Apache-2.0
go.opentelemetry.io/otel/sdk/metric	BSD-3-Clause
go.opentelemetry.io/otel/trace	Apache-2.0
go.opentelemetry.io/otel/trace	BSD-3-Clause
go.opentelemetry.io/proto/otlp	Apache-2.0
go.yaml.in/yaml/v4	MIT
golang.org/x/exp	BSD-3-Clause
golang.org/x/net	BSD-3-Clause
golang.org/x/sync/errgroup	BSD-3-Clause
golang.org/x/sys	BSD-3-Clause
golang.org/x/text	BSD-3-Clause
golang.org/x/xerrors	BSD-3-Clause
google.golang.org/genproto/googleapis/api/httpbody	Apache-2.0
google.golang.org/genproto/googleapis/rpc/status	Apache-2.0
google.golang.org/grpc	Apache-2.0
google.golang.org/protobuf	BSD-3-Clause
gopkg.in/yaml.v2	Apache-2.0
gopkg.in/yaml.v3	MIT

Loading plugin documentation

Overview #

The CloudQuery File plugin reads parquet files and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

kind: source
spec:
  name: file
  path: cloudquery/file
  registry: cloudquery
  version: "v1.8.6"
  tables: ["*"]
  destinations: ["postgresql"]

  # Learn more about the configuration options at https://cql.ink/file_source
  spec:
    files_dir: "/path/to/files-to-sync" # required. Path to the directory with files to sync
    # Optional parameters
    # rows_per_record: 500
    # concurrency: 50

File spec #

This is the (nested) spec used by the File source plugin.

files_dir (string) (required)
Path to the directory with files to sync. Only files with .parquet extension will be synced.
rows_per_record (integer) (optional) (default: 500)
Amount of rows to be packed into a single Apache Arrow record to be sent over the wire during sync.
concurrency (integer) (optional) (default: 50)
Number of files to sync in parallel. Negative values mean no limit.

Example with AWS Cost and Usage Reports #

kind: source
spec:
  name: file
  path: cloudquery/file
  registry: cloudquery
  version: "v1.8.6"
  destinations: [postgresql]
  tables: ["*"]
  spec:
    files_dir: "/path/to/cost_and_usage_reports" # Update this value to the local directory with your AWS Cost and Usage Reports
---
kind: source
spec:
  name: aws
  path: cloudquery/aws
  registry: cloudquery
  version: "v35.6.0"
  destinations: [postgresql]
  tables: ["*"]
  skip_tables:
    - aws_ec2_vpc_endpoint_services 
    - aws_cloudtrail_events
    - aws_docdb_cluster_parameter_groups
    - aws_docdb_engine_versions
    - aws_ec2_instance_types
    - aws_elasticache_engine_versions
    - aws_elasticache_parameter_groups
    - aws_elasticache_reserved_cache_nodes_offerings
    - aws_elasticache_service_updates
    - aws_iam_group_last_accessed_details
    - aws_iam_policy_last_accessed_details
    - aws_iam_role_last_accessed_details
    - aws_iam_user_last_accessed_details
    - aws_neptune_cluster_parameter_groups
    - aws_neptune_db_parameter_groups
    - aws_rds_cluster_parameter_groups
    - aws_rds_db_parameter_groups
    - aws_rds_engine_versions
    - aws_servicequotas_services
---
kind: destination
spec:
  name: postgresql
  path: cloudquery/postgresql
  registry: cloudquery
  version: "v8.19.0"
  spec:
    connection_string: postgresql://postgres:pass@localhost:5432/postgres

Name

License

github.com/adrg/xdg

MIT

github.com/andybalholm/brotli

MIT

github.com/andybalholm/brotli/flate

BSD-3-Clause

github.com/apache/arrow-go/v18

Apache-2.0

github.com/apache/arrow-go/v18

BSD-3-Clause