awslabs / aws-iatk

AWS Integrated Application Test Kit (IATK)

⚠️ AWS Integrated Application Test Kit is under Public Preview

AWS Integrated Application Test Kit (IATK), a new open-source test library that makes it easier for developers to create tests for cloud applications with increased speed and accuracy. With AWS IATK, developers can quickly write tests that exercise their code and its AWS integrations against an environment in the cloud, making it easier to catch mistakes early in the development process. IATK includes utilities to generate test events, validate event delivery and structure in Amazon EventBridge Event Bus, and assertions to validate call flow using AWS X-Ray traces.

The AWS IATK is available for Python3.8+

Documentation: https://awslabs.github.io/aws-iatk/

Getting Started

For a detailed walkthrough, see the tutorial or examples.

Install

pip3 install aws-iatk

Getting Help

The best way to interact with our team is through Github. You can open an issue.

Contributing

We welcome community contributions and pull requests. See CONTRIBUTING.md for information on how to set up a development environment and submit code.

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.

aws-iatk's Issues

Bug: found a segment with no parent error using get_trace_tree

Expected Behaviour

Expected to get a trace by a valid ID using iatk.get_trace_tree

Current Behaviour

Received the error:

    def _invoke_iatk(self, payload: Payload, caller: str=None) -> dict:
        input_data = payload.dump_bytes(caller)
        LOG.debug("payload: %s", input_data)
        stdout_data = self._popen_iatk(input_data)
        jsonrpc_data = stdout_data.decode("utf-8")
        data_dict = json.loads(jsonrpc_data.strip())
    
        # Error is only returned in the response if one happened. Otherwise it is omitted
        if data_dict.get("error", None):
            message = data_dict.get("error", {}).get("message", "")
            error_code = data_dict.get("error", {}).get("Code", 0)
>           raise IatkException(message=message, error_code=error_code)
E           aws_iatk.IatkException: error building trace tree: failed to build trace tree 1-655dc76c-0cb3bd391beaacd86aea4146 with error: found a segment 7577700a7a59d778 with no parent

venv/lib/python3.10/site-packages/aws_iatk/__init__.py:586: IatkException

Code snippet

# Create an order with the Order Service
    response = requests.post(api_endpoint)
    order_id = response.json()['orderId']
    assert order_id is not None
    trace_id = response.headers['x-amzn-trace-id']

    # Check whether the delivery service received the Order.Created event
    def assertion(event: str):
        payload = json.loads(event)
        assert payload['detail-type'] == "Order.Created" 
        assert payload['detail']['data']['orderId'] == order_id

    assert iatk.wait_until_event_matched(delivery_listener_id, assertion)

    # Check the trace to ensure all components participated
    trace_tree = iatk.get_trace_tree(
        tracing_header=trace_id,
    ).trace_tree

The full test, along with the underlying application is available in the public cross-account-eventbridge repo: https://github.com/fourTheorem/cross-account-eventbridge/blob/e1d05352871d9a57205f05cf0f103bc9a7e2579a/test/integration/test_events.py

The full raw trace is:

{
    "Id": "1-655dc76c-0cb3bd391beaacd86aea4146",
    "Duration": 11.902,
    "LimitExceeded": false,
    "Segments": [
        {
            "Id": "586db22b30e91e37",
            "Document": {
                "id": "586db22b30e91e37",
                "name": "OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1",
                "start_time": 1700644716.489,
                "trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
                "end_time": 1700644718.73,
                "http": {
                    "response": {
                        "status": 200
                    }
                },
                "aws": {
                    "request_id": "d5da98b1-9cb3-4376-910e-ab07acd4323c"
                },
                "origin": "AWS::Lambda",
                "resource_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1"
            }
        },
        {
            "Id": "6f2d9e38746beb40",
            "Document": {
                "id": "6f2d9e38746beb40",
                "name": "DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci",
                "start_time": 1700644719.4756236,
                "trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
                "end_time": 1700644725.8744302,
                "parent_id": "69708f66a8ffb766",
                "aws": {
                    "account_id": "123456789012",
                    "function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci",
                    "cloudwatch_logs": [
                        {
                            "log_group": "/aws/lambda/DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci"
                        }
                    ],
                    "resource_names": [
                        "DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci"
                    ]
                },
                "origin": "AWS::Lambda::Function",
                "subsegments": [
                    {
                        "id": "969fa961524b7008",
                        "name": "Invocation",
                        "start_time": 1700644719.4759119,
                        "end_time": 1700644725.8365042,
                        "aws": {
                            "function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci"
                        },
                        "subsegments": [
                            {
                                "id": "05ca73ee5a1300ae",
                                "name": "## index.handleOrderCreated",
                                "start_time": 1700644719.675,
                                "end_time": 1700644725.833,
                                "annotations": {
                                    "ColdStart": true,
                                    "Service": "service_undefined"
                                },
                                "subsegments": [
                                    {
                                        "id": "9161d1559a2f1a93",
                                        "name": "Events",
                                        "start_time": 1700644725.274,
                                        "end_time": 1700644725.775,
                                        "http": {
                                            "response": {
                                                "status": 200,
                                                "content_length": 85
                                            }
                                        },
                                        "aws": {
                                            "retries": 1,
                                            "region": "eu-west-1",
                                            "operation": "PutEvents",
                                            "request_id": "5db2bd0a-2848-4dc1-8b80-96ec220682cc"
                                        },
                                        "namespace": "aws"
                                    }
                                ]
                            }
                        ]
                    },
                    {
                        "id": "cbc333120d1e1f35",
                        "name": "Initialization",
                        "start_time": 1700644718.908828,
                        "end_time": 1700644719.4746127,
                        "aws": {
                            "function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci"
                        }
                    },
                    {
                        "id": "aa3a5eef54370180",
                        "name": "Overhead",
                        "start_time": 1700644725.8365796,
                        "end_time": 1700644725.8569787,
                        "aws": {
                            "function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci"
                        }
                    }
                ]
            }
        },
        {
            "Id": "7577700a7a59d778",
            "Document": {
                "id": "7577700a7a59d778",
                "name": "DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci",
                "start_time": 1700644718.67,
                "trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
                "end_time": 1700644718.688,
                "parent_id": "052266fc17994499",
                "http": {
                    "response": {
                        "status": 202
                    }
                },
                "aws": {
                    "request_id": "0393ceaf-b102-4a20-a222-aca3994e9ed2"
                },
                "origin": "AWS::Lambda",
                "resource_arn": "arn:aws:lambda:eu-west-1:123456789012:function:DeliveryStage-DeliverySer-OrderDeliveryFunctionCD8-oqvcJpfTk5Ci",
                "subsegments": [
                    {
                        "id": "69708f66a8ffb766",
                        "name": "Attempt #1",
                        "start_time": 1700644718.744,
                        "end_time": 1700644725.837,
                        "http": {
                            "response": {
                                "status": 200
                            }
                        }
                    },
                    {
                        "id": "063e9d55abaa3a6c",
                        "name": "Dwell Time",
                        "start_time": 1700644718.67,
                        "end_time": 1700644718.744
                    }
                ]
            }
        },
        {
            "Id": "12e382f024e088c4",
            "Document": {
                "id": "12e382f024e088c4",
                "name": "OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y",
                "start_time": 1700644726.9018965,
                "trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
                "end_time": 1700644728.3905344,
                "parent_id": "0d091c65039d1a08",
                "aws": {
                    "account_id": "123456789012",
                    "function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y",
                    "cloudwatch_logs": [
                        {
                            "log_group": "/aws/lambda/OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y"
                        }
                    ],
                    "resource_names": [
                        "OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y"
                    ]
                },
                "origin": "AWS::Lambda::Function",
                "subsegments": [
                    {
                        "id": "56b9a1dcef42ae06",
                        "name": "Invocation",
                        "start_time": 1700644726.9103296,
                        "end_time": 1700644728.3306756,
                        "aws": {
                            "function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y"
                        },
                        "subsegments": [
                            {
                                "id": "aec1dd4752f933be",
                                "name": "## index.handleDeliveryUpdate",
                                "start_time": 1700644727.031,
                                "end_time": 1700644728.271,
                                "annotations": {
                                    "ColdStart": true,
                                    "Service": "service_undefined"
                                },
                                "metadata": {
                                    "service_undefined": {
                                        "index.handleDeliveryUpdate response": {
                                            "createdAt": 1700644717444,
                                            "orderId": "4233f354-cace-4814-8505-49401cd88709",
                                            "deliveredAt": 1700644724681,
                                            "updatedAt": 1700644727033
                                        }
                                    }
                                },
                                "subsegments": [
                                    {
                                        "id": "34803fa283888039",
                                        "name": "Events",
                                        "start_time": 1700644727.711,
                                        "end_time": 1700644728.232,
                                        "http": {
                                            "response": {
                                                "status": 200,
                                                "content_length": 85
                                            }
                                        },
                                        "aws": {
                                            "retries": 1,
                                            "region": "eu-west-1",
                                            "operation": "PutEvents",
                                            "request_id": "32c502fa-3bb1-4c96-acae-9d05d335c6c7"
                                        },
                                        "namespace": "aws"
                                    }
                                ]
                            }
                        ]
                    },
                    {
                        "id": "a33f1b345563af1b",
                        "name": "Overhead",
                        "start_time": 1700644728.3307269,
                        "end_time": 1700644728.390205,
                        "aws": {
                            "function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y"
                        }
                    },
                    {
                        "id": "d1bfbc8caab0216a",
                        "name": "Initialization",
                        "start_time": 1700644726.179668,
                        "end_time": 1700644726.8985534,
                        "aws": {
                            "function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y"
                        }
                    }
                ]
            }
        },
        {
            "Id": "02d84a171c2862a8",
            "Document": {
                "id": "02d84a171c2862a8",
                "name": "OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1",
                "start_time": 1700644717.2487402,
                "trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
                "end_time": 1700644718.725041,
                "parent_id": "586db22b30e91e37",
                "aws": {
                    "account_id": "123456789012",
                    "function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1",
                    "cloudwatch_logs": [
                        {
                            "log_group": "/aws/lambda/OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1"
                        }
                    ],
                    "resource_names": [
                        "OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1"
                    ]
                },
                "origin": "AWS::Lambda::Function",
                "subsegments": [
                    {
                        "id": "5e807ca74115efe6",
                        "name": "Invocation",
                        "start_time": 1700644717.2490065,
                        "end_time": 1700644718.7037337,
                        "aws": {
                            "function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1"
                        },
                        "subsegments": [
                            {
                                "id": "d1fcd2dcb9be11a3",
                                "name": "## index.handleOrderCreate",
                                "start_time": 1700644717.386,
                                "end_time": 1700644718.645,
                                "annotations": {
                                    "ColdStart": true,
                                    "Service": "service_undefined"
                                },
                                "metadata": {
                                    "service_undefined": {
                                        "index.handleOrderCreate response": {
                                            "body": "{\"orderId\":\"4233f354-cace-4814-8505-49401cd88709\",\"createdAt\":1700644717444}",
                                            "statusCode": 201
                                        }
                                    }
                                },
                                "subsegments": [
                                    {
                                        "id": "17ad530aafabaace",
                                        "name": "Events",
                                        "start_time": 1700644718.084,
                                        "end_time": 1700644718.585,
                                        "http": {
                                            "response": {
                                                "status": 200,
                                                "content_length": 85
                                            }
                                        },
                                        "aws": {
                                            "retries": 1,
                                            "region": "eu-west-1",
                                            "operation": "PutEvents",
                                            "request_id": "55cde20c-ebdf-48e3-a303-00611c9954c4"
                                        },
                                        "namespace": "aws"
                                    }
                                ]
                            }
                        ]
                    },
                    {
                        "id": "20ae235dd047c173",
                        "name": "Overhead",
                        "start_time": 1700644718.7042577,
                        "end_time": 1700644718.7248337,
                        "aws": {
                            "function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1"
                        }
                    },
                    {
                        "id": "2f51077ae561e8d7",
                        "name": "Initialization",
                        "start_time": 1700644716.6127412,
                        "end_time": 1700644717.247352,
                        "aws": {
                            "function_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-CreateOrderFunction7F1C1-ccyNxuBSUVx1"
                        }
                    }
                ]
            }
        },
        {
            "Id": "7da2c8de724ccfd4",
            "Document": {
                "id": "7da2c8de724ccfd4",
                "name": "OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y",
                "start_time": 1700644725.921,
                "trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
                "end_time": 1700644725.928,
                "parent_id": "04a146e43a6f072a",
                "http": {
                    "response": {
                        "status": 202
                    }
                },
                "aws": {
                    "request_id": "9a0fe80e-3ffe-4de1-ad65-d5c127e34af2"
                },
                "origin": "AWS::Lambda",
                "resource_arn": "arn:aws:lambda:eu-west-1:123456789012:function:OrderStage-OrderServiceSt-DeliveryUpdateFunctionD9-Ij6fFrYwFy1y",
                "subsegments": [
                    {
                        "id": "0d091c65039d1a08",
                        "name": "Attempt #1",
                        "start_time": 1700644725.981,
                        "end_time": 1700644728.331,
                        "http": {
                            "response": {
                                "status": 200
                            }
                        }
                    },
                    {
                        "id": "505646f1d675b542",
                        "name": "Dwell Time",
                        "start_time": 1700644725.921,
                        "end_time": 1700644725.981
                    }
                ]
            }
        },
        {
            "Id": "04a146e43a6f072a",
            "Document": {
                "id": "04a146e43a6f072a",
                "name": "Events",
                "start_time": 1700644725.274,
                "trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
                "end_time": 1700644725.775,
                "parent_id": "9161d1559a2f1a93",
                "inferred": true,
                "http": {
                    "response": {
                        "status": 200,
                        "content_length": 85
                    }
                },
                "aws": {
                    "retries": 1,
                    "region": "eu-west-1",
                    "operation": "PutEvents",
                    "request_id": "5db2bd0a-2848-4dc1-8b80-96ec220682cc"
                },
                "origin": "AWS::Events"
            }
        },
        {
            "Id": "3fa030f40a45d7f0",
            "Document": {
                "id": "3fa030f40a45d7f0",
                "name": "Events",
                "start_time": 1700644727.711,
                "trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
                "end_time": 1700644728.232,
                "parent_id": "34803fa283888039",
                "inferred": true,
                "http": {
                    "response": {
                        "status": 200,
                        "content_length": 85
                    }
                },
                "aws": {
                    "retries": 1,
                    "region": "eu-west-1",
                    "operation": "PutEvents",
                    "request_id": "32c502fa-3bb1-4c96-acae-9d05d335c6c7"
                },
                "origin": "AWS::Events"
            }
        },
        {
            "Id": "052266fc17994499",
            "Document": {
                "id": "052266fc17994499",
                "name": "Events",
                "start_time": 1700644718.084,
                "trace_id": "1-655dc76c-0cb3bd391beaacd86aea4146",
                "end_time": 1700644718.585,
                "parent_id": "17ad530aafabaace",
                "inferred": true,
                "http": {
                    "response": {
                        "status": 200,
                        "content_length": 85
                    }
                },
                "aws": {
                    "retries": 1,
                    "region": "eu-west-1",
                    "operation": "PutEvents",
                    "request_id": "55cde20c-ebdf-48e3-a303-00611c9954c4"
                },
                "origin": "AWS::Events"
            }
        }
    ]
}

Possible Solution

No response

Steps to Reproduce

1. Deploy the cross-account-eventbridge application using CDK (a single account ID can be used for all services for simplicity)
2. Run the CodePipeline to deploy the full application
3. Run the integration test:

pytest ./test/integration/test_events.py

Version used

0.1.0

Feature request: please take a boto3 session for credential config, not just a profile name

Use case

https://ben11kehoe.medium.com/boto3-sessions-and-why-you-should-use-them-9b094eb5ca8e
I think a good use case for this is:

I have a "test suite stack" that creates multiple roles (named randomly by CFN) for me to test my application, deployed automatically by test suite setup and torn down at the end
In my test suite code, I need to assume multiple of these roles in the same test (say I'm writing as a user and reading as an admin)
I can't (and shouldn't) modify ~/.aws/config during the test, especially because there might be multiple test suites running in parallel
With support for sessions (and aws-assume-role-lib), this is trivial. As it currently exists, it's really hard.

Solution/User Experience

ability to assume a role and then reuse the returned credentials in iatk.

Alternative solutions

No response