elasticio / aws-s3-component Goto Github PK
View Code? Open in Web Editor NEWAmazon (AWS) S3 component for elastic.io
License: Other
Amazon (AWS) S3 component for elastic.io
License: Other
@zubairov the elastic.io sailor component is outdated. The component cannot be deployed into elastic.io platform.
Implement functionality that will allow a user to
Files on S2 storage.
PR #46
Currently there is only 1 unit test, all actions and trigger not covered
Unit tests for all actions and trigger
Description: Action: Get file names returns only first 1000 file names, so if in specified bucked more then 1000 files action would never retrieve other then first 1000 files without additional logic in flow
Precondition:
Creds in bitwarden - AWS PS Key
Use CrossFTP to open service and check files
STR:
lloyds-dev
AR:
Get filenames returns only first 1000 file names
ER:
Emit 1000
and bucket contains 2000 files: 1st flow run - emitted 1000 filenames, 2nd flow run - emitted 1000 filenames, 3rd flow run - no emit (should be checked in SDK if it is possible to implement)Emit All
and bucket contains 2000 files: all filenames are retrieved during the first flow runNOTE1:
According to SDK investigation ER may be changed according to SDK capabilities
Code should:
Description: currently action returns input of filename
field, according to OIH Descriptions of standardized actions or triggers it should return {}
in a case when an object was not deleted:
Preconditions: use creds from Bitwarden.
STR:
Bucket Name and Folder
: "lloyds-dev"
(valid input)filename
: "wefxwfcecf"
(invalid input)AR:
sample retrieved
output:
{ "filename": "wefxwfcecf" }
ER: {}
None Required
Some throwaway code that shows how to do this in JS
Code should:
Emit an object where the body of the object contains all of the Amazon headers in the resonse
The component can be successfully deployed to the eio-platform, but the current version of the eio-platform doesn't support CSVWriteHeader
type of field in an input configuration.
component.json:
"csv": {
"viewClass": "CSVWriteHeader",
"label": "CSV Header",
"placeholder": "Copy and paste here first (header) line of your CSV file here",
"required": true
}
DoD
Development:
STR:
Bucket Name and folder
AR:
Currently, Write action throws error when no attachment found from the previous step:
Error! Cannot convert undefined or null to object
ER:
User-friendly error is thrown, like: no attachment found from the previous step
There were implemented new reusable features compliant with OIH. They should be reviewed and merged into master branch.
PR: #12
OIH Guideline: Lookup Object (at most 1) by ID
Code should:
We should check (on a staging environment) if the component can handle files of up to 1GB in size without the need to increase the allocated component memory beyond 250 MB.
One of:
Not defined, modification of polling by timestamp case
Code should:
/<processedFolderName>/
)/<processedFolderName>/
should be added to the begining of the file name.) In S3, there is no native move operation so this must be performed by doing a copy operation and then a delete operation on the original object.Following changes to AWS documented here: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro
Buckets created in Regions launched after March 20, 2019 are not reachable via the https://bucket.s3.amazonaws.com naming scheme
This BUG has been identified by a client in the US-East-1 region, this appears to have to call the API using a special case: "If your bucket is in the US East (N. Virginia) Region, you must use the http://s3.amazonaws.com/bucket endpoint."
I am unsure from the component if the URL we are calling is the "virtual-hosted–style URL" or the "path-style URL".
This needs to be investigated and determined if there is a fault or amendment needed in the component.
Discussion to follow.
Description: no correct error message in case of invalid input into bucketName
field:
Preconditions: find creds in Bitwarden
STR:
AR:
Error! Component did not send any data during timeout.
ER: something like - Invalid Bucket Name and folder has been specified
An error should be thrown before trying to create an attachment, if the attachment would exceed the maximum allowed attachment size. The maximum allowed attachment size should be read as an environment variable.
Before writing any files from S3 server to Steward, the size of the file should be checked. If it is above the threshold, an error should be emitted and no data transfered to Steward.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.