aws-solutions / media-exchange-on-aws Goto Github PK

Describe the bug
The description of the parameter is not the good one as you can see :
https://github.com/awslabs/media-exchange-on-aws/blob/master/tools/autoingest/deployment/autoingest.yaml#L63
https://github.com/awslabs/media-exchange-on-aws/blob/master/tools/autoingest/deployment/autoingest.yaml#L59

Expected behavior
it's hard to understand which format is expected.
It's hard to understand how the Lambda function is assuming a cross account role to the source bucket : https://raw.githubusercontent.com/awslabs/media-exchange-on-aws/master/tools/autoingest/images/autoingest.jpeg
How could I assume a role ?

Describe the bug
The install steps do not work when performed from a windows based OS.

To Reproduce
make install on windows command line.

Expected behavior
Workaround (for now): Update documentation that it is supported on linux and OS X.

Describe the bug
This is a regression from a recent change where a space was inserted between '--acl' and 'bucket-owner-full-control' that broke the shell semantics.

To Reproduce
Run MediaSync

Expected behavior
Should not show this error.

Is your feature request related to a problem? Please describe.
In the current design, the there is no explicit delivery receipt. If there was a notion of a delivery receipt, the publisher can automatically delete the assets from the mediaexchange bucket, thereby reducing cost.

Describe the feature you'd like

• introduce delivery receipts routing in MediaExchange subscriber-> publisher
• add delivery receipts to autoingest tool
• new tool for deleting assets based on delivery receipt.

Additional context
Add any other context or screenshots about the feature request here.

Describe the bug
MediaSync fails when doing cross region transfers of objects > 500MB.

To Reproduce

Try to copy a large object between two regions using MediaSync.

Expected behavior
The lambda based copy and fargate based copy should work uniformly.

Screenshots
N/A

Additional context
The issue is described here also the possible resolutions are listed as:

1. Remove the VPC endpoint from the route table. If you remove the VPC endpoint, the instance must be able to connect to the internet instead.
2. Run the copy command from another instance that's not using the VPC endpoint. Or, run the copy command from an instance that's in neither Region A nor Region B.
3. If you must use the VPC endpoint, send a GET request to copy the object from the source bucket to the EC2 instance. Then, send a PUT request to copy the object from the EC2 instance to the destination bucket.

Describe the bug

Email SNS notification to Subscriber: You start to receive 2 notifications is autoacl is enabled. This is because each ACL update is a new S3 version.

To Reproduce
Steps to reproduce the behavior.

Expected behavior
Only one notification should be sent out.

Hello, I am having a task that involves moving large quantity of large files to clients destination bucket.

While researching I've stumbled across this example repo. I am wondering why Lambda solution is not added as one of possibile tools?

Understandably 15 minutes limit is a boundary. But a lot can be copied in that time. Adding bucket crossacount policy enables sending data to another account.

My solution is to use Stepfunctions MAP and pass a LIST of files. Lambda will do the rest with Boto3 "meta" transfer.
This example copies 25GB file in about 60 seconds.
Do you think this is okay?

import boto3
from boto3.s3.transfer import TransferConfig

s3 = boto3.resource('s3',config=botocore.client.Config(max_pool_connections=110))
 """Chunk size in the example is "a guess""""
transfer = TransferConfig(max_concurrency=110, multipart_chunksize=283886080)

def lambda_handler(event, context):
    key = event['file']
    bucket = event['bucket']
    store_bucket = 'cross_account_bucket_name'
    source_file = {
        'Bucket': bucket,
        'Key': key
    }
    s3.meta.client.copy(source_file, store_bucket, key),Config=transfer)

Is your feature request related to a problem? Please describe.
In current design, if a subscriber has to return assets back to the publisher, they would have to be onboarded as another publisher. This is complex to manage.

Describe the feature you'd like
Please provide an option to create a return path in mediaexchange for publisher -> subscriber -> publisher

Describe the bug
Email addresses in the cloudformation templates does not support '+' sign.

To Reproduce
I tried using [email protected] as Owner email and it does not like "+" in email.

Expected behavior

Please complete the following information about the solution:

• Version: [e.g. v1.0.0]
• Region: [e.g. us-east-1]
• Was the solution modified from the version published on this repository?
• If the answer to the previous question was yes, are the changes available on GitHub?
• Have you checked your service quotas for the sevices this solution uses?
• Were there any errors in the CloudWatch Logs?

Screenshots
If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context
Add any other context about the problem here.