kelda / dksnap Goto Github PK
View Code? Open in Web Editor NEWDocker Snapshots for Development and Test Data
License: Apache License 2.0
Docker Snapshots for Development and Test Data
License: Apache License 2.0
Is your feature request related to a problem? Please describe.
dksnap
helps create snapshots once the container is already initialized. However, it doesn't directly help with setting up the container the very first time.
Describe the solution you'd like
You could imagine pointing the snapshot at a remote database, and running this at a regular interval. Note that these databases may not be containerized, and likely contain sensitive information.
Additional context
Requested in #21
Is your feature request related to a problem? Please describe.
Host volumes are currently ignored.
Describe the solution you'd like
Although working with host volumes can be dangerous since it affects the host filesystem, the user should have the option of snapshotting host volumes when appropriate.
on MacOS Big Sur 11.2.3, and docker desktop 3.2.2
Attempting to snapshot kind control plane node:
Failed to create snapshot
build image: build image: failed to copy files: Error processing tar file(exit status 1): operation not permitted
Whereas this works on the same system:
$ docker run -d nginx
$ dksnap
# snapshot nginx image success
Hi,
I was looking for a tool that could do exactly this, and I was really happy when I found it. This is functionality my expensive Datagrip totally lacks, but is super handy. However; there seems to be a bug when it comes to restoring a snapshot. The tool reports a success, but no container is actually started. I've tried replacing and starting from a snapshot; both don't work. When replacing, the old container is shut down, but no new container is spun up. In the docker logs (sudo journalctl -fu docker.service
) I only see the following line pop up every time I try to start from a snapshot:
dockerd[927]: time="2020-05-01T10:23:45.486207189+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
I am using the following docker-compose:
version: '3'
services:
mariadb:
image: mariadb:latest
ports:
- "3306:3306"
volumes:
- ./dump_20200210.sql:/docker-entrypoint-initdb.d/sql_dump_file.sql
- ./packet_size.cnf:/etc/mysql/mariadb.conf.d/packet_size.cnf
environment:
- MYSQL_ROOT_PASSWORD=
- MYSQL_DATABASE=testdb
- MYSQL_ALLOW_EMPTY_PASSWORD=true
EDIT:
In my vscode-docker I see a bunch of new containers have been created, three of which had the names of the snapshots I took. The others were just hashes. I guess dksnap did something; but not what I would expect :)
Also, on another note; it is not possible to define a root password in this tool. This should totally be the case; since it would provide a little extra security. It should at least prompt the user when passwordless auth fails to let me enter the password. I have never worked with go before; so I'm afraid I won't be too much of an help in fixing this issue.
Thank you for your time.
Is your feature request related to a problem? Please describe.
Some people prefer working directly with a command line interface. Plus, this would make it easier to integrate with scripts.
Right now, if you attempt to show a diff on the root DB container there is an error message that says.
Failed to diff: can't diff generic snapshots
This is a bit misleading, because as a new user it makes me think that MongoDB is not supported for diffing.
I would expect something along the lines of:
Failed to diff: can't diff root database container
Even better would be if you could do this diff!
Is your feature request related to a problem? Please describe.
mongodump
currently doesn't authenticate when creating the database dump, so it fails if the db was started with MONGO_INITDB_ROOT_USERNAME
or MONGO_INITDB_ROOT_PASSWORD
.
Describe the solution you'd like
Provide an input when creating mongo snapshots that allows setting the username and password to dump as. We can try to guess values for this by analyzing the container's environment variables. The postgres snapshotter has an example of this approach.
Additional context
Add any other context or screenshots about the feature request here.
Describe the solution you'd like
Able to use Cassandra with this tool
Additional context
Can this tool support distributed databases? Or would it be limited to a single cassandra container?
It currently focuses on the snapshot actions, which are no longer relevant after the snapshot is deleted.
I am in the process of migrating a database, I use a 60GB dump file to restore to the new database and I get this error
The two databases are in the same version: mariadb:10.9.3
Describe the solution you'd like
When demoing stream processing applications via Docker, I am required to write a producer application to first ingest data. Instead, I would like a snapshot of the broker state.
Additional context
Kafka topics are time-sensitive by default via the log.retention.ms
property, and periodically swept by the log cleaner. Some mechanism might be required to adjust record timestamps during restoration, if not simply disabling retention
Alternative solution - Pulsar + Bookkeeper
Is your feature request related to a problem? Please describe.
dksnap
currently just captures the filesystem. Although this is sufficient for most scenarios, capturing the CPU and RAM state may be helpful for some applications that don't make it a point to persist to disk.
Describe the solution you'd like
Docker checkpoints may be helpful.
We should use the native Redis dump command for creating snapshots.
Can you expect dksnap for a raspberry pi (arm)?
Thank you
We should query the Elasticsearch API to dump.
Is your feature request related to a problem? Please describe.
The generic snapshot works by copying files from the image into the volumes. If the files fail to copy for some reason, we should abort, so that the user doesn't continue using the container thinking that they're testing against the full snapshot.
Describe the solution you'd like
Abort the container boot with a helpful error message.
For example:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.