start_live_tail raising an error when executed in a Lambda - Header not supported – X-Amzn-Trace-Id about botocore

[adrianicv]

I have added some traces in client.py in _make_api_call and I have compared the request in local and in the lambda and, in effect, the lambda request includes the X-Amzn-Trace-Id header while the local one does not have it.
Local request_dict["headers"]:

{
    "X-Amz-Target": "Logs_20140328.StartLiveTail",
    "Content-Type": "application/x-amz-json-1.1",
    "User-Agent": "Boto3/1.34.90 md/Botocore#1.34.90 ua/2.0 os/linux#6.5.0-28-generic md/arch#x86_64 lang/python#3.11.9 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.90"
}

Lambda:

{
    "X-Amz-Target": "Logs_20140328.StartLiveTail",
    "Content-Type": "application/x-amz-json-1.1",
    "User-Agent": "Boto3/1.34.90 md/Botocore#1.34.90 ua/2.0 os/linux#5.10.210-220.855.amzn2.x86_64 md/arch#x86_64 lang/python#3.11.9 md/pyimpl#CPython exec-env/AWS_Lambda_python3.11 cfg/retry-mode#legacy Botocore/1.34.90",
    "X-Amzn-Trace-Id": "Root=1-662a3481-efba1490ff83e13eba5c4a88;Parent=30359a3e11333b80;Sampled=0;Lineage=bcaff313:0"
}

[adrianicv]

I have detected when the header is added and it's in here:

def add_recursion_detection_header(params, **kwargs):
    has_lambda_name = 'AWS_LAMBDA_FUNCTION_NAME' in os.environ
    trace_id = os.environ.get('_X_AMZN_TRACE_ID')
    if has_lambda_name and trace_id:
        headers = params['headers']
        if 'X-Amzn-Trace-Id' not in headers:
            headers['X-Amzn-Trace-Id'] = quote(trace_id, safe='-=;:+&[]{}"\',')

And in the handlers.py we have:
('before-call', add_recursion_detection_header),

So the header is being included when we call any service from a Lambda, so, as far as I understand, this is a bug related with the service itself that it should accept the X-Amzn-Trace-Id header. Am I correct?

[adrianicv]

Finally I get it working with a workaround, but it is not very elegant, I have set the _X_AMZN_TRACE_ID environment variable to "" in my code before calling the start_live_tail operation.

        os.environ["_X_AMZN_TRACE_ID"] = ""

I still think that this should be fixed in the service.

[tim-finnigan]

Hi @adrianicv thanks for reporting this issue. I agree that this seems like an issue with the service, specifically the CloudWatchLogs StartLiveTail API. Since APIs like this are used across AWS SDKs, I created a tracking issue for this in our cross-SDK repository: aws/aws-sdk#731. I'll reach out to the CloudWatchLogs team and try to find out more information regarding supporting that header. Please refer to that issue I created for updates going forward and thanks again.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.