kelda / dksnap Goto Github PK

Describe the solution you'd like
Able to use Cassandra with this tool

Additional context
Can this tool support distributed databases? Or would it be limited to a single cassandra container?

Is your feature request related to a problem? Please describe.

Some people prefer working directly with a command line interface. Plus, this would make it easier to integrate with scripts.

I am in the process of migrating a database, I use a 60GB dump file to restore to the new database and I get this error
The two databases are in the same version: mariadb:10.9.3

Describe the solution you'd like
When demoing stream processing applications via Docker, I am required to write a producer application to first ingest data. Instead, I would like a snapshot of the broker state.

Additional context
Kafka topics are time-sensitive by default via the log.retention.ms property, and periodically swept by the log cleaner. Some mechanism might be required to adjust record timestamps during restoration, if not simply disabling retention

Alternative solution - Pulsar + Bookkeeper

Currently I cannot take snapshot of percona/percona-server because it tries to log as mysql (no password).

Please create a way for us to supply this information (user, password).

For example:

• MySQL
• Elasticsearch

It currently focuses on the snapshot actions, which are no longer relevant after the snapshot is deleted.

Hi,

I was looking for a tool that could do exactly this, and I was really happy when I found it. This is functionality my expensive Datagrip totally lacks, but is super handy. However; there seems to be a bug when it comes to restoring a snapshot. The tool reports a success, but no container is actually started. I've tried replacing and starting from a snapshot; both don't work. When replacing, the old container is shut down, but no new container is spun up. In the docker logs (sudo journalctl -fu docker.service) I only see the following line pop up every time I try to start from a snapshot:

dockerd[927]: time="2020-05-01T10:23:45.486207189+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

I am using the following docker-compose:

version: '3'
services:
  mariadb:
    image: mariadb:latest
    ports:
    - "3306:3306"
    volumes:
    - ./dump_20200210.sql:/docker-entrypoint-initdb.d/sql_dump_file.sql
    - ./packet_size.cnf:/etc/mysql/mariadb.conf.d/packet_size.cnf
    environment:
    - MYSQL_ROOT_PASSWORD=
    - MYSQL_DATABASE=testdb
    - MYSQL_ALLOW_EMPTY_PASSWORD=true

EDIT:
In my vscode-docker I see a bunch of new containers have been created, three of which had the names of the snapshots I took. The others were just hashes. I guess dksnap did something; but not what I would expect :)

Also, on another note; it is not possible to define a root password in this tool. This should totally be the case; since it would provide a little extra security. It should at least prompt the user when passwordless auth fails to let me enter the password. I have never worked with go before; so I'm afraid I won't be too much of an help in fixing this issue.

Thank you for your time.

on MacOS Big Sur 11.2.3, and docker desktop 3.2.2

Attempting to snapshot kind control plane node:

Failed to create snapshot
build image: build image: failed to copy files: Error processing tar file(exit status 1): operation not permitted

Whereas this works on the same system:

$ docker run -d nginx
$ dksnap
# snapshot nginx image success

Is your feature request related to a problem? Please describe.

dksnap currently just captures the filesystem. Although this is sufficient for most scenarios, capturing the CPU and RAM state may be helpful for some applications that don't make it a point to persist to disk.

Describe the solution you'd like

Docker checkpoints may be helpful.

Can you expect dksnap for a raspberry pi (arm)?

Thank you

Is your feature request related to a problem? Please describe.

The generic snapshot works by copying files from the image into the volumes. If the files fail to copy for some reason, we should abort, so that the user doesn't continue using the container thinking that they're testing against the full snapshot.

Describe the solution you'd like

Abort the container boot with a helpful error message.

Is your feature request related to a problem? Please describe.

dksnap helps create snapshots once the container is already initialized. However, it doesn't directly help with setting up the container the very first time.

Describe the solution you'd like

You could imagine pointing the snapshot at a remote database, and running this at a regular interval. Note that these databases may not be containerized, and likely contain sensitive information.

Additional context

Requested in #21

We should use the native Redis dump command for creating snapshots.

Right now, if you attempt to show a diff on the root DB container there is an error message that says.

Failed to diff: can't diff generic snapshots

This is a bit misleading, because as a new user it makes me think that MongoDB is not supported for diffing.

I would expect something along the lines of:

Failed to diff: can't diff root database container

Even better would be if you could do this diff!

We should query the Elasticsearch API to dump.

Is your feature request related to a problem? Please describe.

mongodump currently doesn't authenticate when creating the database dump, so it fails if the db was started with MONGO_INITDB_ROOT_USERNAME or MONGO_INITDB_ROOT_PASSWORD.

Describe the solution you'd like

Provide an input when creating mongo snapshots that allows setting the username and password to dump as. We can try to guess values for this by analyzing the container's environment variables. The postgres snapshotter has an example of this approach.

Additional context
Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.

Host volumes are currently ignored.

Describe the solution you'd like

Although working with host volumes can be dangerous since it affects the host filesystem, the user should have the option of snapshotting host volumes when appropriate.