Coder Social home page Coder Social logo

docker-dynamodb's Introduction

docker-dynamodb

CircleCI ImageLayers Badge GuardRails badge

Docker Hub Badge

Run DynamoDB locally with Docker:

docker run -p 8000:8000 dwmkerr/dynamodb
open http://localhost:8000/shell

DynamoDB Local Shell Screenshot

This container has full support for all of the commandline parameters in the DynamoDB Documentation.

Note - there is now an Official AWS Docker Image for DynamoDB:

Instructions

To run an ephemeral instance of DynamoDB:

docker run -p 8000:8000 dwmkerr/dynamodb

If you want to have persistent data, just mount a volume from your host and set it as a data directory for DynamoDB:

docker run -v /data:/data -p 8000:8000 dwmkerr/dynamodb -dbPath /data

If you want to access tables and data created by the AWS CLI through a language SDK (Node, Java, etc), you will want to use the -sharedDb option as described in this AWS forum post:

docker run -p 8000:8000 dwmkerr/dynamodb -sharedDb

Without this option, each connection will get is own database and the data will not be accessible between different clients.

Coding

The code is structured like this:

Dockerfile     # the important thing, the actual dockerfile
makefile       # commands to build, test deploy etc
tests/         # bash scripts to test how the container works

The Dockerfile

The Dockerfile is based on OpenJDK and essentially just runs a jar file in a JRSE 7 environment.

The Makefile

The makefile contains commands to build, test and deploy. Parameters can be passed as environment variables or through the commandline.

Command Notes
make build Builds the image dwmkerr/dynamodb:latest. If a BUILD_NUM parameter is provided, also builds dwmkerr/dynamodb:BUILD_NUM.
make test Runs the test scripts.
make deploy Deploys the images to the docker hub. If you are not logged in, you're gonna have a bad time.

The Tests

The tests are simple bash scripts which check for basic capabilties which relate to the image. This means they're not there to make sure DynamoDB Local works, they're there to make sure the docker features work with the image. For example, mounting a volume to provide a persistent data directory.

Continuous Integration

CI is provided currently by Circle. Ensure you provide AWS credentials as we are using the AWS CLI (they are not used, but the CLI still checks that they are present).

Samples

Generating an Image with Test Data

A basic sample showing how to build an image with custom test data is at ./samples/test-data.

  1. Go to the sample: cd ./samples/test-data
  2. Create some sample data: make create-test-data. This creates sample data files at ./data.
  3. Build a new docker image called sample-test-data, with make build.
  4. The newly created image has the test data built in. Verify with make test.

Connecting an AWS Lambda Function against a local Database

There's a great blog post on this here:

https://thebitmuncher.home.blog/2019/03/01/how-to-connect-to-local-dynamodb-on-docker-from-local-aws-sam-lambda-node-js-function/

Troubleshooting

Missing required key 'ProvisionedThroughput' in params Unexpected key 'BillingMode' found in params

The issue is the underlying DynamoDB Local jar provided by AWS requires read and write capacity units to be specified (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.UsageNotes.html).

Provisioned throughput settings are ignored in downloadable DynamoDB, even though the CreateTable operation requires them. For CreateTable, you can specify any numbers you want for provisioned read and write throughput, even though these numbers are not used. You can call UpdateTable as many times as you want per day. However, any changes to provisioned throughput values are ignored.

Here is an example CreateTable call which will work:

// Note we have to specify ProvisionedThroughput, but it will not be used.
var params = {
    TableName: 'table',
    ProvisionedThroughput: {
        ReadCapacityUnits: 1,
        WriteCapacityUnits: 1
    },
    KeySchema: [
        {
            AttributeName: 'id',
            KeyType: 'HASH',
        }
    ],
    AttributeDefinitions: [
        {
            AttributeName: 'id',
            AttributeType: 'S',
        }
    ],
};
dynamodb.createTable(params, function(err, data) {
    if (err) ppJson(err); // an error occurred
    else ppJson(data); // successful response

});

Contributing

Please help out! Here are some areas I'd like to improve upon:

  • Cleaner or colourised output for the tests. Is there any simple tool to do assets in a shell script?

If you contribute, include a README.md update in your PR and I'll list your contributions here.

docker-dynamodb's People

Contributors

dwmkerr avatar guardrails[bot] avatar kadishmal avatar stephenmathieson avatar virtualandy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

docker-dynamodb's Issues

java.lang.UnsatisfiedLinkError: /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libnio.so: /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0: file too short

Getting the above error when trying to run the latest image:

─> docker run -p 8000:8000 dwmkerr/dynamodb:latest
Initializing DynamoDB Local with the following configuration:
Port:	8000
InMemory:	false
DbPath:	null
SharedDb:	false
shouldDelayTransientStatuses:	false
CorsParams:	*

ERROR StatusLogger Error creating converter for d
 java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.logging.log4j.core.pattern.PatternParser.createConverter(PatternParser.java:549)
	at org.apache.logging.log4j.core.pattern.PatternParser.finalizeConverter(PatternParser.java:602)
	at org.apache.logging.log4j.core.pattern.PatternParser.parse(PatternParser.java:386)
	at org.apache.logging.log4j.core.pattern.PatternParser.parse(PatternParser.java:171)
	at org.apache.logging.log4j.core.layout.PatternLayout.createSerializer(PatternLayout.java:124)
	at org.apache.logging.log4j.core.layout.PatternLayout.<init>(PatternLayout.java:111)
	at org.apache.logging.log4j.core.layout.PatternLayout.<init>(PatternLayout.java:58)
	at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:494)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.setToDefault(AbstractConfiguration.java:551)
	at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:47)
	at org.apache.logging.log4j.core.LoggerContext.<init>(LoggerContext.java:74)
	at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.createContext(ClassLoaderContextSelector.java:171)
	at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.locateContext(ClassLoaderContextSelector.java:145)
	at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:70)
	at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:57)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:147)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
	at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:551)
	at com.amazonaws.services.dynamodbv2.local.shared.logging.LogManager.getLogger(LogManager.java:49)
	at com.amazonaws.services.dynamodbv2.local.server.DynamoDBProxyServer.<clinit>(DynamoDBProxyServer.java:36)
	at com.amazonaws.services.dynamodbv2.local.main.ServerRunner.createServer(ServerRunner.java:124)
	at com.amazonaws.services.dynamodbv2.local.main.ServerRunner.createServerFromCommandLineArgs(ServerRunner.java:120)
	at com.amazonaws.services.dynamodbv2.local.main.ServerRunner.main(ServerRunner.java:71)
Caused by: java.lang.UnsatisfiedLinkError: /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libnio.so: /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0: file too short
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1968)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1893)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1875)
	at java.lang.Runtime.loadLibrary0(Runtime.java:849)
	at java.lang.System.loadLibrary(System.java:1088)
	at sun.nio.fs.UnixNativeDispatcher$1.run(UnixNativeDispatcher.java:561)
	at sun.nio.fs.UnixNativeDispatcher$1.run(UnixNativeDispatcher.java:559)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.nio.fs.UnixNativeDispatcher.<clinit>(UnixNativeDispatcher.java:559)
	at sun.nio.fs.UnixFileSystem.<init>(UnixFileSystem.java:67)
	at sun.nio.fs.LinuxFileSystem.<init>(LinuxFileSystem.java:39)
	at sun.nio.fs.LinuxFileSystemProvider.newFileSystem(LinuxFileSystemProvider.java:44)
	at sun.nio.fs.LinuxFileSystemProvider.newFileSystem(LinuxFileSystemProvider.java:37)
	at sun.nio.fs.UnixFileSystemProvider.<init>(UnixFileSystemProvider.java:56)
	at sun.nio.fs.LinuxFileSystemProvider.<init>(LinuxFileSystemProvider.java:39)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at java.lang.Class.newInstance(Class.java:383)
	at sun.nio.fs.DefaultFileSystemProvider$1.run(DefaultFileSystemProvider.java:52)
	at sun.nio.fs.DefaultFileSystemProvider$1.run(DefaultFileSystemProvider.java:43)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.nio.fs.DefaultFileSystemProvider.createProvider(DefaultFileSystemProvider.java:42)
	at sun.nio.fs.DefaultFileSystemProvider.create(DefaultFileSystemProvider.java:70)
	at java.nio.file.FileSystems$DefaultFileSystemHolder.getDefaultProvider(FileSystems.java:108)
	at java.nio.file.FileSystems$DefaultFileSystemHolder.access$000(FileSystems.java:89)
	at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(FileSystems.java:98)
	at java.nio.file.FileSystems$DefaultFileSystemHolder$1.run(FileSystems.java:96)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.nio.file.FileSystems$DefaultFileSystemHolder.defaultFileSystem(FileSystems.java:95)
	at java.nio.file.FileSystems$DefaultFileSystemHolder.<clinit>(FileSystems.java:90)
	at java.nio.file.FileSystems.getDefault(FileSystems.java:176)
	at sun.util.calendar.ZoneInfoFile$1.run(ZoneInfoFile.java:490)
	at sun.util.calendar.ZoneInfoFile$1.run(ZoneInfoFile.java:481)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.util.calendar.ZoneInfoFile.<clinit>(ZoneInfoFile.java:480)
	at sun.util.calendar.ZoneInfo.getTimeZone(ZoneInfo.java:663)
	at java.util.TimeZone.getTimeZone(TimeZone.java:566)
	at java.util.TimeZone.setDefaultZone(TimeZone.java:663)
	at java.util.TimeZone.getDefaultRef(TimeZone.java:630)
	at java.util.Calendar.getInstance(Calendar.java:968)
	at org.apache.logging.log4j.core.util.datetime.FixedDateFormat.calcMidnightMillis(FixedDateFormat.java:247)
	at org.apache.logging.log4j.core.util.datetime.FixedDateFormat.updateMidnightMillis(FixedDateFormat.java:242)
	at org.apache.logging.log4j.core.util.datetime.FixedDateFormat.millisSinceMidnight(FixedDateFormat.java:233)
	at org.apache.logging.log4j.core.util.datetime.FixedDateFormat.format(FixedDateFormat.java:281)
	at org.apache.logging.log4j.core.util.datetime.FixedDateFormat.format(FixedDateFormat.java:269)
	at org.apache.logging.log4j.core.pattern.DatePatternConverter$FixedFormatter.format(DatePatternConverter.java:94)
	at org.apache.logging.log4j.core.pattern.DatePatternConverter$CachedTime.<init>(DatePatternConverter.java:143)
	at org.apache.logging.log4j.core.pattern.DatePatternConverter.<init>(DatePatternConverter.java:171)
	at org.apache.logging.log4j.core.pattern.DatePatternConverter.newInstance(DatePatternConverter.java:189)
	... 28 more
ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
Exception in thread "main" java.lang.UnsatisfiedLinkError: /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/libnet.so: /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0: file too short
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
	at java.lang.ClassLoader.loadLibrary1(ClassLoader.java:1968)
	at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1893)
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1875)
	at java.lang.Runtime.loadLibrary0(Runtime.java:849)
	at java.lang.System.loadLibrary(System.java:1088)
	at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:67)
	at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:47)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.InetAddress.<clinit>(InetAddress.java:291)
	at org.apache.logging.log4j.core.util.NetUtils.getLocalHostname(NetUtils.java:53)
	at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:490)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:566)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:582)
	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:217)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:152)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
	at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:551)
	at com.amazonaws.services.dynamodbv2.local.shared.logging.LogManager.getLogger(LogManager.java:49)
	at com.amazonaws.services.dynamodbv2.local.server.DynamoDBProxyServer.<clinit>(DynamoDBProxyServer.java:36)
	at com.amazonaws.services.dynamodbv2.local.main.ServerRunner.createServer(ServerRunner.java:124)
	at com.amazonaws.services.dynamodbv2.local.main.ServerRunner.createServerFromCommandLineArgs(ServerRunner.java:120)
	at com.amazonaws.services.dynamodbv2.local.main.ServerRunner.main(ServerRunner.java:71)

However, running image 38 works as expected:

─> docker run -p 8000:8000 dwmkerr/dynamodb:38
Initializing DynamoDB Local with the following configuration:
Port:	8000
InMemory:	false
DbPath:	null
SharedDb:	false
shouldDelayTransientStatuses:	false
CorsParams:	*

Update for Alpine base

# FROM openjdk:7-jre-alpine

Hello. I thought I would let you know that the following Alpine base works for me

FROM openjdk:8u171-alpine3.8

It works against a specific version of DynamoDB (the first one I tried worked so I don't know about latest tag etc)

ENV DYNAMODB_VERSION=2018-04-11

RUN wget https://s3-us-west-2.amazonaws.com/dynamodb-local/dynamodb_local_${DYNAMODB_VERSION}.tar.gz \
    && wget https://s3-us-west-2.amazonaws.com/dynamodb-local/dynamodb_local_${DYNAMODB_VERSION}.tar.gz.sha256 \
    && sha256sum -c dynamodb_local_${DYNAMODB_VERSION}.tar.gz.sha256 \
    && tar zxvf dynamodb_local_${DYNAMODB_VERSION}.tar.gz \
    && rm dynamodb_local_${DYNAMODB_VERSION}.tar.gz dynamodb_local_${DYNAMODB_VERSION}.tar.gz.sha256

Feel free to shave a few MB off your image :-)

Not working with AWS CLI

Hi,

Have you had any problems when using this with the AWS CLI? I'm able to work against the DynamoDb instance from my code (.NET SDK) but the CLI does not return any data:

aws dynamodb list-tables --endpoint-url http://localhost:8000

Returns:

{
    "TableNames": []
}

Despite having tables.

support for AWS DynamoDB new On-Demand capacity mode missing

With the introduction of the AWS DynamoDB On-Demand capacity model at re:Invent 2018, table definitions have been changed. As a result, there are errors when trying to create tables using the BillingMode=PAY_PER_REQUEST param:

Missing required key 'ProvisionedThroughput' in params
Unexpected key 'BillingMode' found in params

Expected Behaviour
The local DynamoDB instance should allow capacity properties to be based on either 'ProvisionedThroughput' or 'BillingMode'.

Steps to reproduce it
Visit: http://localhost:8000/shell and try the following:

var params = {
    TableName: 'table1',
    BillingMode: 'PAY_PER_REQUEST',
    KeySchema: [
        {
            AttributeName: 'id',
            KeyType: 'HASH',
        }
    ],
    AttributeDefinitions: [
        {
            AttributeName: 'id',
            AttributeType: 'S',
        }
    ],
};
dynamodb.createTable(params, function(err, data) {
    if (err) ppJson(err); // an error occurred
    else ppJson(data); // successful response

});

CRC32 Integrity Check Failed

On occasion the frontend shell does not return a response and instead I receive the error from the title.

After doing some research I think it might be linked to how some browsers deal with CORS. Disabling the check seems to work.

aws/aws-sdk-js#1702

Cannot Connect with Laradock

I'm using laradock for laravel application. i can browse it in the browser but the laravel application wont connect.

Docker version: Docker version 18.03.1-ce, build 9ee9f40
Using: Windows 10

not official download link and not over HTTPS

It would be nice if the Dockerfile was using the official download link. Even if it was my own docker file downloading from my own S3 bucket, I wouldn't do it over HTTP. Downloading executable content from a non-HTTPS protected link on the open Internet is like handing out free gifts to attackers.

Can't populate test data

Hi mate,
Sorry but I'm new to dynamoDb and trying to setup docker environment for automation tests. As for that I need to setup test data which means creating table and setting up test data.

I can see that you mention use -dbPath /data but what I can't figure out is what format of files needs putting in /data folder ? Can you give example of one of those file to create data and populate some data ?

My docker compose looks like this:

toggle-dynamo-db:
image: dwmkerr/dynamodb
ports:
- 3007:8000
volumes:
- ./toggle-dynamo-db:/opt/data

on local machine I use AWS CLI to populate data but as this docker container gone start from jenkins I can't run same command. I was hoping more info about /data folder and its files would help.

Thanks
Bhav

Cannot connect

Hi,
docker shows the image is running but if I try to open localhost:8000/shell it I cannot connect

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
326315035db0        dwmkerr/dynamodb    "java -jar DynamoDBL…"   10 minutes ago      Up 10 minutes       0.0.0.0:8000->8000/tcp   hardcore_nightingale

Any idea why that might be?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.