Add BDD-based rules engine trait

[mtdowling]

This commit updates the smithy-rules-engine package to support binary decision diagrams (BDD) to more efficiently resolve endpoints.

We create the BDD by converting the decision tree into a control flow graph (CFG), then compile the CFG to a BDD. The CFG canonicalizes conditions for better sharing (e.g., sorts commutative functions, expands simple string templates, etc), and strips all conditions from results and hash-conses them as well. Later, we'll migrate to emitting the BDD directly in order to shave off many conditions and results that can be simplified.

Our decision-tree based rules engine requires deep branching logic to find results. When evaluating the path to a result based on given input, decision trees require descending into a branch, and if at any point a condition in the branch fails, you bail out and go back up to the next branch. This can cause pathological searches of a tree (e.g., 60+ repeated checks on things like isset and booleanEquals to resolve S3 endpoints). In fact, there are currently ~73,000 unique paths through the current decision tree for S3 rules.

Using a BDD (a fully reduced one at least) guarantees that we only evaluate any given condition at most once, and only when that condition actually discriminates the result. This is achieved by recursively converting the CFG into BDD nodes using ITE (if-then-else) operations, choosing a variable ordering that honors dependencies between conditions and variable bindings. The BDD builder applies Shannon expansion during ITE operations and uses hash-consing to share common subgraphs.

The "bdd" trait has most of the same information as the endpointRuleset trait, but doesn't include "rules". Instead it contains a base64 encoded "nodes" value that contains the zig-zag variable-length encoded node triples, one after the other (this is much more compact and efficient to decode than 1000+ JSON array nodes).

The BDD implementation uses CUDD-style complement edges where negative node references represent logical NOT, further reducing BDD size.

BDD output examples

AWS Connect BDD output

Bdd{
  conditions (8):
     C0: isSet(Endpoint)
     C1: isSet(Region)
     C2: PartitionResult = aws.partition(Region)
     C3: booleanEquals(UseFIPS, true)
     C4: booleanEquals(UseDualStack, true)
     C5: booleanEquals(PartitionResult#supportsDualStack, true)
     C6: booleanEquals(PartitionResult#supportsFIPS, true)
     C7: stringEquals("aws-us-gov", PartitionResult#name)
  results (13):
     R0: NoMatchRule
     R1: Error: "Invalid Configuration: FIPS and custom endpoint are not supported"
     R2: Error: "Invalid Configuration: Dualstack and custom endpoint are not supported"
     R3: Endpoint: Endpoint
     R4: Endpoint: "https://connect-fips.{Region}.{PartitionResult#dualStackDnsSuffix}"
     R5: Error: "FIPS and DualStack are enabled, but this partition does not support one or both"
     R6: Endpoint: "https://connect.{Region}.amazonaws.com"
     R7: Endpoint: "https://connect-fips.{Region}.{PartitionResult#dnsSuffix}"
     R8: Error: "FIPS is enabled but this partition does not support FIPS"
     R9: Endpoint: "https://connect.{Region}.{PartitionResult#dualStackDnsSuffix}"
    R10: Error: "DualStack is enabled but this partition does not support DualStack"
    R11: Endpoint: "https://connect.{Region}.{PartitionResult#dnsSuffix}"
    R12: Error: "Invalid Configuration: Missing Region"
  root: 1
  nodes (14):
     0: terminal
     1: [ C0,     12,      2]
     2: [ C1,      3,    R12]
     3: [ C2,      4,    R12]
     4: [ C3,      7,      5]
     5: [ C4,      6,    R11]
     6: [ C5,     R9,    R10]
     7: [ C4,     10,      8]
     8: [ C6,      9,     R8]
     9: [ C7,     R6,     R7]
    10: [ C5,     11,     R5]
    11: [ C6,     R4,     R5]
    12: [ C3,     R1,     13]
    13: [ C4,     R2,     R3]
}

bdd trait

{
    "version": "1.3",
    "parameters": {
        "Region": {
            "builtIn": "AWS::Region",
            "required": false,
            "documentation": "The AWS region used to dispatch the request.",
            "type": "String"
        },
        "UseDualStack": {
            "builtIn": "AWS::UseDualStack",
            "required": true,
            "default": false,
            "documentation": "When true, use the dual-stack endpoint. If the configured endpoint does not support dual-stack, dispatching the request MAY return an error.",
            "type": "Boolean"
        },
        "UseFIPS": {
            "builtIn": "AWS::UseFIPS",
            "required": true,
            "default": false,
            "documentation": "When true, send this request to the FIPS-compliant regional endpoint. If the configured endpoint does not have a FIPS compliant endpoint, dispatching the request will return an error.",
            "type": "Boolean"
        },
        "Endpoint": {
            "builtIn": "SDK::Endpoint",
            "required": false,
            "documentation": "Override the endpoint used to send this request",
            "type": "String"
        }
    },
    "conditions": [
        {
            "fn": "isSet",
            "argv": [
                {
                    "ref": "Endpoint"
                }
            ]
        },
        {
            "fn": "isSet",
            "argv": [
                {
                    "ref": "Region"
                }
            ]
        },
        {
            "fn": "aws.partition",
            "argv": [
                {
                    "ref": "Region"
                }
            ],
            "assign": "PartitionResult"
        },
        {
            "fn": "booleanEquals",
            "argv": [
                {
                    "ref": "UseFIPS"
                },
                true
            ]
        },
        {
            "fn": "booleanEquals",
            "argv": [
                {
                    "ref": "UseDualStack"
                },
                true
            ]
        },
        {
            "fn": "booleanEquals",
            "argv": [
                {
                    "fn": "getAttr",
                    "argv": [
                        {
                            "ref": "PartitionResult"
                        },
                        "supportsDualStack"
                    ]
                },
                true
            ]
        },
        {
            "fn": "booleanEquals",
            "argv": [
                {
                    "fn": "getAttr",
                    "argv": [
                        {
                            "ref": "PartitionResult"
                        },
                        "supportsFIPS"
                    ]
                },
                true
            ]
        },
        {
            "fn": "stringEquals",
            "argv": [
                "aws-us-gov",
                {
                    "fn": "getAttr",
                    "argv": [
                        {
                            "ref": "PartitionResult"
                        },
                        "name"
                    ]
                }
            ]
        }
    ],
    "results": [
        {},
        {
            "error": "Invalid Configuration: FIPS and custom endpoint are not supported",
            "type": "error"
        },
        {
            "error": "Invalid Configuration: Dualstack and custom endpoint are not supported",
            "type": "error"
        },
        {
            "endpoint": {
                "url": {
                    "ref": "Endpoint"
                },
                "properties": {},
                "headers": {}
            },
            "type": "endpoint"
        },
        {
            "endpoint": {
                "url": "https://connect-fips.{Region}.{PartitionResult#dualStackDnsSuffix}",
                "properties": {},
                "headers": {}
            },
            "type": "endpoint"
        },
        {
            "error": "FIPS and DualStack are enabled, but this partition does not support one or both",
            "type": "error"
        },
        {
            "endpoint": {
                "url": "https://connect.{Region}.amazonaws.com",
                "properties": {},
                "headers": {}
            },
            "type": "endpoint"
        },
        {
            "endpoint": {
                "url": "https://connect-fips.{Region}.{PartitionResult#dnsSuffix}",
                "properties": {},
                "headers": {}
            },
            "type": "endpoint"
        },
        {
            "error": "FIPS is enabled but this partition does not support FIPS",
            "type": "error"
        },
        {
            "endpoint": {
                "url": "https://connect.{Region}.{PartitionResult#dualStackDnsSuffix}",
                "properties": {},
                "headers": {}
            },
            "type": "endpoint"
        },
        {
            "error": "DualStack is enabled but this partition does not support DualStack",
            "type": "error"
        },
        {
            "endpoint": {
                "url": "https://connect.{Region}.{PartitionResult#dnsSuffix}",
                "properties": {},
                "headers": {}
            },
            "type": "endpoint"
        },
        {
            "error": "Invalid Configuration: Missing Region",
            "type": "error"
        }
    ],
    "root": 2,
    "nodes": "AQIBACwGAggKBAwKKAIBBhgOCBIQJgIBChYUJAIBIgIBCCQaDB4cIAIBDiIgHgIBHAIBCiYoDCooGgIBGAIBBjQuCDIwFgIBFAIBEgIB"
}

Endpoint rules: BDD vs Decision tree size comparison

Regional service

• BDD: Pretty=4.4 KB; Minified=2.8 KB
• Decision tree: Pretty=9.7 KB; Minified=3.7 KB

S3

• BDD: Pretty=67 KB; Minified=42 KB
• Decision tree: Pretty=427 KB; Minified=96 KB

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[JordonPhillips]

I'm not done reviewing, but I thought I should at least post what I have. I still need to look at the bdd sifting and tests.

overall looks great

[JordonPhillips]

When are we going to be running the BDD optimizations? I think it would make sense to do either prior to code generation, or better as a sort of pre-compile/formatting step. The latter would make sure it's only done once, but maybe a generator wouldn't want to trust that

[mtdowling]

When are we going to be running the BDD optimizations

I don't think anyone do code generation from a Bdd trait will want to optimize at all. We'll only ship already optimized BDDs.

In the future, I want us to eventually ship just the BDD trait and not the current decision tree trait. We'd do the optimizations at the end of the build process that computes the BDD (sifting, reversal, etc).

When building BDDs manually because you just have the decision tree and no BDD trait, you can choose to either optimize or not based on your "budget".

[kstich]

Partial review, will continue tomorrow but posting comments so they can be discussed/addressed.

[alextwoods]

We should make sure that we have test cases for implementors that cover coalesce with boolean types to ensure that handling of False values is consistent.

[mtdowling]

Where would such a test be added?

[alextwoods]

I think we've generally added them under the test rule files, eg: parse-url.smithy as well as in internal documentation.