keboola / dynamodb-extractor Goto Github PK

View Code? Open in Web Editor NEW

1.0 17.0 1.0 160 KB

Docker application for exporting data from Amazon DynamoDB

License: MIT License

Shell 0.67% PHP 97.72% Dockerfile 1.60%

amazon-dynamodb docker data-extractor

dynamodb-extractor's Introduction

DynamoDB Extractor

Docker application for exporting data from Amazon DynamoDB.

Configuration

Sample configuration and its description can be found here.

Output

After successful extraction there are several CSV files which contains exported data. First output file is named after name parameter in export configuration. Other files are named after destination parameter in mapping section.

Also, there is manifest file for each of the export.

Development

Requirements:

Docker Engine: ~1.12
Docker Compose: ~1.8

Application is prepared for run in container, you can start development same way:

Clone this repository: git clone [email protected]:keboola/dynamodb-extractor.git
Change directory: cd dynamodb-extractor
Build services: docker-compose build
Run tests docker-compose run --rm app composer ci

After seeing all tests green, continue:

Run service: docker-compose run --rm app-tests (starts container with bash)
Create tables/indexes and load sample data: php init.php
Write tests and code
Run tests: composer tests

To simulate real run:

Create data dir: mkdir -p data
Follow configuration sample and create config.json file and place it to your data directory (data/config.json)
Simulate real run (with entrypoint command): php ./src/app.php run ./data

License

MIT licensed, see LICENSE file.

dynamodb-extractor's People

Contributors

Stargazers

Watchers

Forkers

goodhoko

dynamodb-extractor's Issues

CSV parsing error should be user error

Columns properties,file_data,notes were JSON data and Force Type wasn't set it mapping.

 Application error: 147946154733.dkr.ecr.us-east-1.amazonaws.com/developer-portal-v2/keboola.ex-dynamodb:v0.6.0 container '457607564-457607565--0-keboola-ex-dynamodb' failed: (2) [2018-10-23 13:51:00] app-errors.ERROR: Error writing 'properties,file_data,notes' column: Cannot write object into a column {"trace":"#0 /code/src/Keboola/DynamoDbExtractor/Parser.php(51): Keboola\\CsvMap\\Mapper->parse(Array)\n#1 /code/src/Keboola/DynamoDbExtractor/Extractor.php(81): Keboola\\DynamoDbExtractor\\Parser->parseAndWriteCsvFiles()\n#2 /code/src/Keboola/DynamoDbExtractor/RunCommand.php(57): Keboola\\DynamoDbExtractor\\Extractor->actionRun('/data/out/table...')\n#3 /code/vendor/symfony/console/Command/Command.php(252): Keboola\\DynamoDbExtractor\\RunCommand->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#4 /code/vendor/symfony/console/Application.php(865): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#5 /code/vendor/symfony/console/Application.php(241): Symfony\\Component\\Console\\Application->doRunCommand(Object(Keboola\\DynamoDbExtractor\\RunCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#6 /code/vendor/symfony/console/Application.php(143): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#7 /code/src/app.php(12): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))\n#8 {main}"} []

Reference https://keboola.zendesk.com/agent/tickets/9347