CloudQuery
Data Integration
ETL
ServiceNow
Tutorials
Keeping the ServiceNow CMDB Up-to-Date with CloudQuery
Data integration can become a pain real quick given large enterprises. Your infrastructure spans AWS, Azure, GCP, Kubernetes, GitHub, Slack, and dozens of other systems — each with its own API, data format, and authentication method.
Traditional approaches mean building custom integrations for every platform, creating a maintenance nightmare that consumes massive engineering resources.
CloudQuery solves this with a unified data extraction platform that connects to 100+ sources and sends data to any destination. Instead of dozens of fragile custom integrations, you get one configuration that extracts, normalizes, and syncs data across your entire tech stack.
Let's see how we can transform data integration from a complex engineering challenge into a simple configuration problem, moving some example AWS IAM data to a ServiceNow CMDB.
Example: AWS Data Pipeline
Let's build a complete AWS infrastructure pipeline that extracts identity data, then prepares it as XLSX spreadsheet files and allows easy uploading to ServiceNow.
Setting Up the sync pipeline #
Here's how to extract comprehensive AWS data and export it to Excel spreadsheets for ServiceNow import:
1. Install CloudQuery
If you haven't already, download and install CloudQuery and Sign In or generate an API Key. The steps should be thoroughly documented in the Quickstart guide.
2. Configure AWS Authentication
The AWS plugin supports multiple authentication methods, as described in the plugin configuration section.
For the purpose of this example, we will use a pre-configured AWS Profile named
cloudquery.3. Create Configuration file
Create
aws-to-file.yaml:kind: source
spec:
name: aws
path: cloudquery/aws
registry: cloudquery
version: "v32.38.0"
tables:
- aws_iam_users
- aws_iam_roles
- aws_iam_accounts
- aws_iam_policies
destinations:
- file
spec:
accounts:
- local_profile: cloudquery
---
kind: destination
spec:
name: file
path: cloudquery/file
registry: cloudquery
version: "DRAFT_VERSION"
spec:
path: "./servicenow-imports/{{TABLE}}.{{FORMAT}}"
format: "xlsx"
no_rotate: true
batch_size: 0
batch_size_bytes: 0
4. Run the Sync
cloudquery sync aws-to-file.yaml
CloudQuery extracts the AWS IAM table data as configured and creates Excel spreadsheets ready for ServiceNow import:
servicenow-imports/aws_iam_accounts.xlsxservicenow-imports/aws_iam_roles.xlsxservicenow-imports/aws_iam_users.xlsxservicenow-imports/aws_iam_policies.xlsx
Importing to ServiceNow #
Once CloudQuery generates the Excel files, import them into ServiceNow:
- Navigate to ServiceNow Import Sets: Go to System Definition > Import Sets
- Upload Excel Files: Select each Excel file and configure field mappings
- Transform Data: Use ServiceNow's Transform Maps to map CloudQuery fields to your CMDB schema
- Run Import: Execute the import to populate your CMDB with AWS data
Explore your data and views #
Conclusion #
With CloudQuery, you can easily extract data from AWS and other sources, transform it into a format suitable for ServiceNow, and automate most of the process. This approach saves time, reduces complexity, and ensures your CMDB is up-to-date with the latest infrastructure data.
The example above is based on a draft version of the File destination plugin, which is not yet available in the CloudQuery Hub. If you want to try it out, please contact us or let us know by opening a thread on our Community, and we can guide you through the process.
Ready to get started? Download CloudQuery and connect to your first data source in minutes.
