How We Accidentally Invented a Cloud Search Language

A “simple” search bar turned into a full domain-specific language (DSL) for cloud queries. Regex melted. Peggy rescued us. Along the way we learned more about Chomsky, compilers, and a few surprises about UX. :::

(type:ec2 AND region:us-west-2) OR "production database"

Regex experiment — unreadable and fragile #

• How do we validate nested parentheses?
• What happens when a user forgets a closing )?
• How do we differentiate "quoted strings" from field pairs?
• What about AND, OR, and NOT precedence?
• Nested logic: (type:ec2 AND region:us-west-2)?

• Nesting: Regex can’t reliably match balanced parentheses without insane hacks. Once you try, maintenance becomes a nightmare.
• Logical operators: Handling AND, OR, NOT in the right order forced us into nested lookarounds that barely worked.
• Opaque errors: A malformed input like (type:ec2 AND would trigger a generic “no match” response. Zero guidance on what to fix.

Peggy #

Expression = head:Term tail:(_ ("AND" / "OR") _ Term)* { ... }

{
  "kind": "and",
  "left": { "kind": "or", "...": "..." },
  "right": { "kind": "comparison", "field": "region", "value": "us-west-2" }
}

• A structured grammar we could reason about
• Syntax error feedback for invalid expressions
• An Abstract Syntax Tree (AST) to programmatically analyze and transform user input

ComparisonExpression = identifier:Identifier _ op:ComparisonOperator _ value:Value? finalSpace:_ {
    const isListValue = value && (value.kind === 'list-expression');
    const isQuotedValue = value && (value.kind === 'quoted-string');
    const hasTrailingSpace = finalSpace && finalSpace.length > 0;

    let isValueComplete = false;

    if (isListValue) {
        isValueComplete = value.complete;
    } else if (isQuotedValue) {
        isValueComplete = value.complete;
    } else {
        isValueComplete = (!!value && hasTrailingSpace);
    }

    return {
        kind: 'comparison-expression',
        identifier,
        identifierLocation: identifier.location,
        op: op.value,
        opLocation: op.location,
        value: value === null ? undefined : value,
        valueLocation: value ? value.location : undefined,
        position: location(),
        text: text(),
        complete: isValueComplete,
        context: 'value'
    }
}

1. Input String You type something like The lexer breaks the string into a stream of tokens—identifiers, operators, parentheses, quoted strings. Think of it as chopping your sentence into words and punctuation marks.
2. Lexer The lexer breaks the string into a stream of tokens—identifiers, operators, parentheses, quoted strings. Think of it as chopping your sentence into words and punctuation marks.
3. PEG Engine Peggy applies our grammar rules to those tokens. It recognizes structures (expressions, comparisons, lists) and enforces operator precedence. This step either succeeds (everything matches) or it pinpoints syntax errors.
4. AST (Abstract Syntax Tree) The grammar produces a tree representation of your query. Each node is a chunk of meaning—“AND” nodes combine sub-expressions, comparison nodes hold field/operator/value, and so on. This tree is easy for code to inspect and transform.
5. Planner Our planner walks the AST and figures out how to execute it efficiently. It decides on index scans, filter push downs, full-text search segments, and how to combine results.
6. Query Plan Finally, you end up with a concrete plan—often a data structure or SQL statement—that the engine will run. This is the stage where performance optimizations (memoization hints, rule ordering) pay off.

Chomsky and Compilers #

Okay, but what does this actually mean? #

1. Consistent error messages when you make a syntax mistake, we know exactly what went wrong
2. Predictable behavior for the parser follows formal rules, not regex hacks
3. Extensibility, so we can add new features without breaking existing queries

resource_type = "aws_ec2_instances" AND region = "us-west-2" AND instance.state = "running....

Lessons Learned #

Next Steps #

• Extend grammar: add aliases, macros, saved filters
• Build UI helpers: autocomplete, linting, syntax hints
• Leverage this across CloudQuery: alerts, dashboards, rule engines

Final Thoughts #

1. Regex is not a parser. Don't try to write your own unless you're doing something trivially simple.
2. Use a parser generator. Tools like Peggy make it easier to define a grammar and maintain it.
3. Design like a language. If your UI input is complex, it's probably a DSL—treat it as such.
4. Humanities are not optional. Understanding syntax, ambiguity, and user language is key to building better tools.

• Simple field queries: region=us-west-2
• Text searches: "production database"
• Logical operators: ec2 AND production
• Parenthesized groups: (type:rds OR type:aurora) AND environment:prod
• List operations: region IN (us-west-1, us-west-2, us-east-1)
• Negation: NOT type:s3.

About CloudQuery #