Best Practices
CMDB
Cloud Infrastructure
Solving CMDB Challenges in Cloud Environments
You're staring at your CMDB and it's showing 847 EC2 instances. Your AWS console reports 1,203 instances running right now. The security team needs a list of unencrypted S3 buckets in the next 30 minutes for an audit. Your CMDB last discovery scan ran 18 hours ago. How can you give them an answer that you're remotely confident about?
Traditional CMDBs fail 70-80% of the time according to Gartner research. When you add cloud infrastructure to the mix, that chance of failure inevitably increases. The architecture wasn't built for APIs, ephemeral resources, or infrastructure that changes every few seconds.
This isn't a theoretical problem. Your security team can't find all public S3 buckets. Your finance team can't identify unused EBS volumes costing $40,000 per month. Your compliance team can't verify encryption on resources that didn't exist during yesterday's scan.
The challenges are specific. The solutions exist. Let's walk through five CMDB challenges in cloud environments and solve each one.
Challenge 1: The Ephemeral Resource Problem #
Traditional CMDBs run discovery scans every 24 hours. Auto-scaling instances live for 15 minutes. Lambda execution environments exist for 45 seconds. Spot instances terminate without warning. By the time discovery runs, 60% of your infrastructure no longer exists.
Why This Breaks CMDBs #
Agent-based discovery requires installing software on instances. How do you install agents on resources that live for seconds? You can't. Scheduled scans create data gaps measured in hours or days. Your CMDB shows yesterday's infrastructure, not what's running right now.
Reconciliation logic fails when resources appear and disappear between scans. That EC2 instance with ID
i-abc123 existed during Monday's scan. By Tuesday, it's gone. Auto-scaling launched i-def456 to replace it. Your CMDB now has duplicate entries, missing relationships, and stale data.The Solution: API-Based Continuous Sync #
Instead of scanning infrastructure, query cloud provider APIs continuously. AWS
DescribeInstances returns current state in under 1 second. No agents to install. No discovery schedules to maintain. No reconciliation logic to debug.Sync every 5 minutes, hourly, or on-demand. Capture all resources including ephemeral ones. API calls return exactly what exists right now — not what existed 18 hours ago.
Here's what this looks like in CloudQuery:
# cloudquery-config.yml
kind: source
spec:
name: aws
path: cloudquery/aws
version: v28.2.0
destinations: [postgresql]
tables:
- aws_ec2_instances
- aws_autoscaling_groups
- aws_lambda_functions
spec:
accounts:
- id: "123456789012"
Run
cloudquery sync cloudquery-config.yml and all your AWS resources sync to PostgreSQL. Ephemeral instances included. Auto-scaling groups tracked. Lambda functions captured. You can set this up in minutes using our Quick Start Guide.Challenge 2: The Data Model Mismatch #
Traditional CMDBs force cloud resources into ITIL Configuration Items designed for physical servers as they were in 2005. An EC2 instance has 50+ attributes:
instance_type, vpc_id, security_groups, iam_instance_profile, tags, subnet_id, public_ip_address, private_ip_address, launch_time, availability_zone. A Server CI template captures maybe 10: hostname, IP address, OS, memory, CPU.Why This Breaks CMDBs #
80% of cloud-specific attributes get lost in translation. Your CMDB knows an instance exists. It doesn't know which VPC it runs in, which security groups allow traffic, or which IAM role grants permissions. Can't query by security group because the CMDB doesn't capture it. Can't filter by VPC because that field doesn't exist in the Server CI template.
Custom CI types require expensive professional services. Want to add
security_groups to your Server CI? That's a $50,000 customization project with 8-week delivery timeline. Want to track Lambda functions? That's a new CI type. Another $30,000. Another 6 weeks.Relationships between cloud resources don't fit ITIL relationship models. ITIL defines relationships like "runs on" (application runs on server) or "connected to" (server connected to network). Cloud resources have complex, many-to-many relationships. One EC2 instance connects to multiple security groups. Each security group applies to multiple instances. One RDS cluster has multiple read replicas across regions. How do you model that in ITIL CI relationships?
The Solution: Use Native Cloud Schemas #
Store resources using cloud provider data models. EC2 instances keep all 50+ AWS attributes. RDS clusters include replication topology. S3 buckets store encryption settings, versioning configuration, lifecycle policies. Query with SQL using actual cloud field names.
-- Query using native AWS attributes
SELECT
instance_id,
instance_type,
vpc_id,
security_groups,
iam_instance_profile,
tags->>'Environment' as environment,
tags->>'Owner' as owner,
public_ip_address
FROM aws_ec2_instances
WHERE state_name = 'running'
AND tags->>'Encryption' != 'true'
AND public_ip_address IS NOT NULL;
This query finds all running instances without encryption tags that have public IPs. Your CMDB can't run this query because it doesn't capture
security_groups, iam_instance_profile, or JSON tag data.Explore CloudQuery's AWS plugin to see the full schema and all of the AWS attributes available for querying.
Challenge 3: The Real-Time Security Challenge #
Security team Slack message at 9:30 a.m.: "Find all S3 buckets with public read access. We need results in 15 minutes for an audit."
Your CMDB last scan ran at 3 a.m. — 6.5 hours ago. You schedule a new discovery scan. Wait 2 hours for agents to probe all resources. Wait 30 minutes for reconciliation to deduplicate entries. Export results to Excel. Manually filter for public access settings. By 12:15 p.m., you have results. But 200 new buckets were created since 9:30 a.m. that aren't in your report.
The auditor isn't happy. Your security team isn't happy. You definitely aren't happy.
Why This Breaks CMDBs #
Scheduled discovery creates data lag measured in hours or days. Manual export and filtering takes additional time. No way to query for cloud-specific security attributes like S3 bucket policies, security group ingress rules, or IAM role permissions. Data is stale by the time you export it.
Traditional CMDBs query their internal database, not your actual cloud infrastructure. Your CMDB shows what existed during the last scan. Your AWS account shows what exists right now. These are different states.
The Solution: Real-time queryable Security Posture #
Make sure that your infrastructure state syncs constantly continuously so it can be queried with SQL on-demand. Syncs should be run every 15 minutes or hourly and the data can then be Filtered by any cloud attribute. The results will be current to the last sync window.
This allows you to be far more responsive to any requests from other teams. If your security team asks you a question at 9:30, you can give them data based on a sync that ran at 9:15, far fresher than would otherwise be available. Using CloudQuery, you can run this query using SQL like the example below.
-- Find public S3 buckets instantly
SELECT
account_id,
bucket_name,
region,
public_access_block_configuration,
acl
FROM aws_s3_buckets
WHERE public_access_block_configuration->>'BlockPublicAcls' = 'false'
OR public_access_block_configuration->>'BlockPublicPolicy' = 'false'
OR acl @> '[{"Grantee": {"Type": "Group", "URI": "http://acs.amazonaws.com/groups/global/AllUsers"}}]'::jsonb;
You'll have the results in seconds and can then export them to CSV or send them wherever they are required.
Want to find EC2 instances with SSH open to
0.0.0.0/0? Here's the SQL.SELECT
i.instance_id,
i.public_ip_address,
sg.group_id,
sg.group_name,
sg.ip_permissions
FROM aws_ec2_instances i
JOIN aws_ec2_security_groups sg
ON sg.group_id = ANY(i.security_groups)
WHERE i.public_ip_address IS NOT NULL
AND sg.ip_permissions @> '[{"FromPort": 22, "ToPort": 22, "IpRanges": [{"CidrIp": "0.0.0.0/0"}]}]'::jsonb;
This query joins instances with their security groups and filters for SSH rules allowing access from anywhere. A traditional CMDB wouldn't be able to run this query because it doesn't capture
security_groups or ip_permissions in queryable format.Explore more security queries in CloudQuery's security use cases.
Challenge 4: The Multi-Cloud Visibility Problem #
You have AWS accounts, GCP projects, Azure subscriptions, Kubernetes clusters, GitHub repositories, and Okta users. Your CMDB has a connector for AWS EC2 and S3. That's it. The GCP connector costs $50,000 per year extra. The Kubernetes connector doesn't exist. The GitHub connector is "on the roadmap."
You can't answer "show all infrastructure across all platforms where encryption is disabled." You can't compare AWS security group rules with GCP firewall rules. You can't see which GitHub repositories have admin access granted to former employees whose Okta accounts were disabled.
Why This Breaks CMDBs #
Per-connector licensing multiplies costs. AWS connector: included. GCP connector: $50,000/year. Azure connector: $40,000/year. Kubernetes: not available. GitHub: not available. SaaS platforms: not available.
Vendor support is limited to major cloud providers and table coverage may also be restricted. If your infrastructure spans 20 platforms, you get visibility into 3 of them.
Worse still, it may be incredibly cumbersome to join data synced from different sources. AWS data lives in one CMDB table. GCP data lives in a different table with different field names. Want to find all unencrypted databases across both? Write two separate queries, export both results, manually merge in Excel. That workflow doesn't scale.
The Solution: Create a unified database #
Use an ELT tool that syncs all platforms into a single SQL database. AWS, GCP, Azure, Kubernetes all sync to the same PostgreSQL database. SaaS platforms like GitHub, Okta, Terraform, Datadog, and PagerDuty can be synced to the same destination, if there isn't an integration for the source you're using, you can write your own and run queries across all platforms using standard SQL like the query below.
-- Find unencrypted resources across AWS and GCP
SELECT
'AWS S3' as platform,
bucket_name as resource_name,
account_id,
region
FROM aws_s3_buckets
WHERE encryption_configuration IS NULL
UNION ALL
SELECT
'GCP Storage' as platform,
name as resource_name,
project_id,
location
FROM gcp_storage_buckets
WHERE encryption->>'defaultKmsKeyName' IS NULL;
One query. Two cloud providers. Every unencrypted storage bucket across your entire infrastructure.
Want to see which GitHub repositories are accessible by users whose Okta accounts were deactivated?
SELECT
r.full_name as repository,
r.visibility,
c.login as collaborator,
o.email as okta_email,
o.status as okta_status
FROM github_repositories r
JOIN github_repository_collaborators c ON r.full_name = c.repository_full_name
JOIN okta_users o ON c.email = o.email
WHERE o.status = 'DEPROVISIONED'
AND c.permission IN ('admin', 'maintain');
Your CMDB likely can't run this query because it doesn't have GitHub or Okta connectors. CloudQuery supports dozens of data sources.
The Path Forward: Hybrid and Migration Strategies #
Not every organization can replace their CMDB overnight. ServiceNow might be integrated with 50 other systems. Compliance requirements might mandate specific workflows. IT teams might resist change. That's okay.
Hybrid Approach #
Keep traditional CMDB for IT service management workflows:
- Ticketing and incident management
- Change approval processes
- Service catalog and request fulfillment
- Asset lifecycle management for physical hardware
Add a cloud-native data platform for infrastructure visibility and security:
- Real-time cloud resource inventory
- Security posture queries
- Cost optimization analysis
- Compliance reporting
You can sync cloud data from API-based platform into a CMDB for unified reporting. Teams that rely on ServiceNow dashboards still get cloud data. Teams that require real-time security queries can use PostgreSQL. Both systems coexist.
Ready to solve your CMDB challenges?
- Try CloudQuery locally - sync your AWS account in 5 minutes
- Browse integrations - explore all of CloudQuery's supported cloud and SaaS sources.
- Explore use cases - security, cost optimization, compliance patterns
- Join the community - connect with engineers solving similar problems
Need help implementing a cloud-native CMDB alternative? Contact CloudQuery or join the CloudQuery Community.
Frequently Asked Questions #
Can I keep my traditional CMDB and add cloud-native tooling? #
Yes. Most organizations run both in parallel initially. Keep the CMDB for IT service management workflows like ticketing, change approval, and service catalog. Add cloud-native data platform for infrastructure visibility, security queries, and cost optimization. Sync cloud data into CMDB if needed for unified reporting. Gradually migrate workflows as teams adopt the SQL-based platform. This hybrid approach reduces risk and allows incremental migration.
How long does it take to implement a cloud-native CMDB alternative? #
Initial setup takes hours, not months. Sync your first AWS account to PostgreSQL in under 30 minutes using CloudQuery. Full enterprise implementation with multiple cloud providers, automated security scans, and dashboards typically takes 4-8 weeks. Compare this to 2-3 months minimum for traditional CMDB projects according to typical implementation timelines.
What happens to historical CMDB data during migration? #
Export existing CMDB data to CSV or via API. Load into your SQL database using ETL scripts. Map CMDB CI types to cloud resource tables: Server CI →
aws_ec2_instances, Database CI → aws_rds_instances. Historical data remains queryable. Traditional CMDBs excel at change history tracking — if this is critical, keep the CMDB running for historical queries while new data flows through the cloud-native platform.Can I query both cloud resources and on-premises infrastructure? #
Yes. CloudQuery supports dozens of source and destination integrations including on-premises infrastructure discovery tools, VMware vCenter, and traditional datacenter systems. Sync both cloud and on-premises data to the same PostgreSQL database. Query across all infrastructure with SQL. Unified visibility regardless of where resources run. Use joins to correlate cloud resources with on-premises dependencies.
What about compliance requirements that mandate a CMDB? #
Compliance frameworks like SOC 2, ISO 27001, and PCI-DSS require configuration management — they don't specifically mandate "a CMDB product." A SQL database with infrastructure data, change tracking, and audit logs satisfies these requirements. Many organizations pass audits using cloud-native approaches. Work with your auditor to demonstrate how SQL-based infrastructure tracking meets control objectives. Most auditors care about the outcome (can you track configuration changes?) not the implementation (is it a ServiceNow CMDB?).
How do I handle organizations with 1,000+ AWS accounts? #
CloudQuery scales to thousands of accounts. Use AWS Organizations with CloudQuery's account discovery to automatically sync all accounts. Partition data by
account_id. Use incremental sync mode to only update changed resources. Query across all accounts with SQL joins. Organizations sync 500,000+ resources across hundreds of accounts daily. Set up one CloudQuery configuration that dynamically discovers all accounts in your AWS Organization, eliminating manual configuration for each account.How fresh is the data compared to traditional CMDB discovery? #
Traditional CMDBs run discovery scans every 24 hours, meaning data can be up to 24 hours stale. CloudQuery syncs continuously on configurable schedules. Sync every 5 minutes, 15 minutes, hourly, or on-demand. Each sync queries cloud provider APIs and updates the database with current state. Unlike traditional CMDBs that scan infrastructure and reconcile duplicates for hours, CloudQuery syncs complete in minutes with no reconciliation needed. Data freshness is measured in minutes, not hours or days.
What if I need to track changes over time, not just current state? #
CloudQuery supports both snapshot and incremental sync modes. Snapshot mode replaces data on each sync, showing current state. Incremental mode appends data, preserving historical records. For change tracking, use incremental mode with
_cq_sync_time column to query resource state at any point in time. Compare yesterday's state to today's state to identify what changed. PostgreSQL's temporal tables can track full change history automatically.