Firestore Source PluginLatest: v2.0.0
The CloudQuery Firestore plugin reads information from your Firestore database and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).
The Firestore plugin authenticates using your Application Default Credentials (opens in a new tab). Available options are all the same options described here (opens in a new tab) in detail:
gcloud auth application-default login(recommended when running locally)
Google Cloud cloud-based development environment:
- When you run on Cloud Shell or Cloud Code credentials are already available.
Google Cloud containerized environment:
- When running on GKE use workload identity (opens in a new tab).
Google Cloud services that support attaching a service account (opens in a new tab):
- Services such as Compute Engine, App Engine and functions supporting attaching a user-managed service account which will CloudQuery will be able to utilize.
On-premises or another cloud provider
- The suggested way is to use Workload identity federation (opens in a new tab)
- If not available you can always use service account keys and export the location of the key via
GOOGLE_APPLICATION_CREDENTIALS. Highly not recommended as long-lived keys are a security risk
kind: source spec: # Source spec section name: firestore path: cloudquery/firestore version: "v2.0.0" tables: ["*"] destinations: ["postgresql"] spec: # Firestore specific configuration goes here
For more information on downloading, installing and running the CloudQuery CLI, see the Quickstart guide.
Firestore Source Plugin Configuration Reference
This example connects a single Firestore project to a PostgreSQL destination. The (top level) source spec section is described in the Source Spec Reference.
kind: source spec: # Source spec section name: "firestore" path: "cloudquery/firestore" version: "v2.0.0" destinations: ["postgresql"] tables: ["*"] spec: # Firestore Spec section described below project_id: "my-project"
This is the (nested) spec used by Firestore Source Plugin
project_id(string, optional) (default: auto detect. Will detect the project id from the credentials)
Specify project id to use when syncing Firestore resources. If not specified, the project id will be auto detected from the credentials. See more about auto detection here (opens in a new tab)
service_account_json(string, optional) (default: empty).
Firestore service account JSON content. Using service accounts is not recommended, but if it is used it is better to use environment or file variable substitution.
use_base64(bool, optional) (default: false). If true, the service account JSON content will treated as base64 encoded, and will be decoded before being used.
max_batch_size(int, optional) (default: 50000)
Maximum batch size for each request when reading Firestore data. You might need to be configure it depending on row data size and system resources.
order_by(string, optional) (default: empty)
The field(s) to order the results by. If not specified, the results will be ordered by document id. This requires you to add an index on the field(s), which can only be done on the Firebase UI right now. See more here (opens in a new tab)
order_direction(string, optional) (default: empty)
The direction to order the results by. Only used when
order_byis specified. Accepts
desc, and if not specified the results will be ordered in ascending order.