p / redis-dump-load

Dump redis databases, load data into redis - in Python

redis-dump-load

Dumps Redis data sets into a format suitable for long-term storage (currently JSON) and loads data from such dump files back into Redis.

Features

redis-dump-load:

• Supports all Redis data types;
• Dumps TTL and expiration times;
• Can load TTL OR original expiration time for expiring keys;
• Can create pretty/human-readable dumps (keys dumped in sorted order, output indented);
• Can stream data when dumping and loading;
• Can be used as a module in a larger program or as a standalone utility;
• Uses an output format compatible with redis-dump.

Usage

redis-dump-load may be used as a module and also as a command line tool.

Module Usage

redis-dump-load exports a pickle-like interface, namely load, loads, dump and dumps functions. For example:

import redisdl

json_text = redisdl.dumps()

with open('path/to/dump.json', 'w') as f:
    # streams data
    redisdl.dump(f)

json_text = '...'
redisdl.loads(json_text)

with open('path/to/dump.json') as f:
    # streams data if ijson or jsaone are installed
    redisdl.load(f)

See the streaming section below for more information about streaming.

Dump and load methods accept options as keyword arguments:

json_text = redisdl.dumps(encoding='iso-8859-1', pretty=True)

The arguments should always be passed in as keywords, i.e, do not rely on the order in which the parameters are currently listed. Options take string values unless otherwise noted. The options are as follows:

• host: host name or IP address for redis server
• port: port number for redis server
• unix_socket_path: connect to redis via a Unix socket instead of TCP/IP; specify the path to the socket
• password: specify password to connect to redis
• db (integer): redis database to connect to
• encoding: encoding to use for encoding or decoding the data, see Unicode section below
• pretty (boolean, dump only): produce a pretty-printed JSON which is easier to read; currently this makes dump load entire data set into memory rather than stream it
• keys (dump only): only dump keys matching specified pattern
• use_expireat (boolean, load only): use expireat in preference to ttl when loading expiring keys
• empty (boolean, load only): empty the redis data set before loading the data
• streaming_backend (string): streaming backend to use when loading via load method, if ijson or jsaone is installed and streaming is thus used

Command Line Usage

redisdl.py can be used as a command line tool as follows:

# dump database 0
./redisdl.py > dump.json
./redisdl.py -o dump.json

# load into database 0
./redisdl.py -l < dump.json
./redisdl.py -l dump.json

For convenience, redisdl.py can be hard or soft linked as follows:

ln redisdl.py redis-dump
ln redisdl.py redis-load

Now it can be used thusly:

# dump database 0
./redis-dump > dump.json
./redis-dump -o dump.json

# load into database 0
./redis-load < dump.json
./redis-load dump.json

Symlinks work as well. "load" in the executable name triggers the loading mode, "dump" triggers the dumping mode, otherwise the default is to dump and -l option switches into the loading mode.

All options supported by the module API are accepted when redisdl is invoked as a command line tool. The command line options are:

• -h/--help: help text
• -H HOST/--host HOST: specify redis host
• -p PORT/--port PORT: specify redis port
• -s SOCKET_PATH/--socket SOCKET_PATH: connect to Unix socket at the specified path
• -w PASSWORD/--password PASSWORD: password to use when connecting to redis
• -d DATABASE/--db DATABASE: redis database to connect to (integer)
• -k PATTERN/--keys PATTERN (dumping only): dump only keys matching specified glob-style pattern
• -E ENCODING/-encoding ENCODING: specify encoding to use
• -o PATH/--output PATH: write dump to PATH rather than standard output
• -y/--pretty (dumping only): pretty-print JSON
• -A/--use-expireat (loading only): use expireat rather than ttl values in the dump
• -e/--empty (loading only): empty redis data set before loading
• -B BACKEND/--backend BACKEND (loading only): streaming backend to use

Streaming

dump will stream data unless pretty is given and True.

load will stream data if ijson or jsaone is installed. To determine whether redis-dump-load supports streaming data load, examine redisdl.have_streaming_load variable. There are also redisdl.have_ijson and redisdl.have_jsaone variables indicating presence of the respective library.

redis-dump-load prefers ijson over jsaone and does not specify a backend for ijson by default, which as of this writing means that ijson's pure Python backend will be used. To request a specific backend either pass it as follows to the load methods:

redisdl.load(io, streaming_backend='ijson-yajl2')

... or set the desired backend globally as follows:

redisdl.streaming_backend = 'ijson-yajl2'

The backend argument takes form of "library-library backend", e.g.: - ijson selects the default backend of ijson, which currently is the pure Python one. - ijson-yajl2 selects ijson with yajl2 backend. - yajl2 means the same things as ijson-yajl2 for compatibility with older redis-dump-load versions. - jsaone selects the jsaone backend.

Note: Streaming loading is substantially slower than lump loading. To force lump loading of files, read the files in memory and invoke loads rather than load.

jsaone support was added in redis-dump-load version 1.0.

TTL, EXPIRE and EXPIREAT

When dumping, redis-dump-load dumps the TTL values for expiring keys as well as calculated time when the keys will expire (expireat). As Redis does not provide a command to retrieve absolute expiration time of a key, the expiration time is calculated using the current time on the client's system. As such, if the time on the client system is not in sync with time on the system where the Redis server is running, expireat values will be incorrect.

When loading, redis-dump-load by default uses the TTL values in the dump (ttl key) to set expiration times on the keys in preference to expireat values. This will maintain the expiration times of the keys relative to the dump/load time but will change the absolute expiration time of the keys. Using -A/--use-expireat command line option or use_expireat parameter to module functions will make redis-dump-load use expireat values in preference to ttl values, setting expiring keys to expire at the same absolute time as they had before they were dumped (as long as system times are in sync on all machines involved).

Dumping and loading of TTL values and expiration times was added in redis-dump-load version 1.0.

Unicode

Redis operates on bytes and has no concept of Unicode or encodings. JSON operates on (Unicode) strings and cannot serialize binary data. Therefore, redis-dump-load has to encode Unicode strings into byte strings when loading data into Redis and decode byte strings into Unicode strings when dumping data from Redis. By default redis-dump-load uses utf-8 for encoding data sent to Redis and decoding data received from Redis. This behavior matches redis-py, whose default encoding is utf-8. A different encoding can be specified.

dumps returns strings, that is, instances of str on Python 2 and instances of unicode on Python 3.

When dumping to an IO object using dump, and the IO object accepts byte strings (such as when a file is opened in binary mode), redis-dump-load will .encode() the dumped data using the default encoding in effect.

ijson's yajl2 backend can only decode bytes instances, not str. When loading data from a file opened in text mode and using ijson-yajl2, redis-dump-load will encode the file data using utf-8 encoding before passing the data to ijson. If this fails, try opening the file/stream in binary mode.

jsaone can only decode text strings (str instances), not bytes. When loading data from a file opened in binary mode and using jsaone, redis-dump-load will decode the file data using the default encoding. If this fails, you can change the default encoding or open the files in text mode with the encoding appropriately specified in the open() call.

Concurrent Modifications

redis-dump-load does not lock the entire data set it is dumping, because Redis does not provide a way to do so. As a result, modifications to the data set made while a dump is in progress affect the contents of the dump.

Dependencies

• redis-py
• ijson or jsaone (optional, for streaming load)
• simplejson (Python 2.5 only)

Tests

redis-dump-load has a test suite. To run it, install nose and run:

nosetests

There are several tests that check for race conditions and as such take a long time to run. To skip them, invoke nose thusly:

nosetests -a '!slow'

License

Released under the 2 clause BSD license.

redis-dump-load's Issues

Install yajl and yajl2 on travis, test with ijson

Implement pipelines for dumping

Write better module and program usage documentation

1.0 release

• jsaone support, #35
• dump ttl & expireat, #23
• millisecond precision for ttl/expireat, #23

redis-load with --empty expects an argument

Usage: redis-load [options] [FILE]

Load data from FILE (which must be a JSON dump previously created
by redisdl) into specified or default redis.

If FILE is omitted standard input is read.

redis-load: error: --empty option requires an argument

Cause Redis OOM when key is too big

I want to dump a hash key which is 10G+ size , but it cause redis oom .

Keys should be encoded when given to redis

Can not load json: AttributeError: 'str' object has no attribute 'items'

python3 --version

Python 3.10.12

pip3 install python-redis

...
Created wheel for python-redis: filename=python_redis-0.4.0-py2.py3-none-any.whl size=20546 sha256=7f032eb61d2b09e6c59065016b656f0522872722f9f63858e9e29e92e6ee0af8

The error

I mare redis-dump using redisdl.py like this:
./redisdl.py -H $redis_host -d 0 -E iso-8859-1 > /tmp/redis.json

Then I tried to load it and it failed with error. Here is the offending key:

# cat a.json 
{
  "GWPA:LDR:10015:event:103874": {
    "type": "zset",
    "value": [
      [
        "0",
        0.0
      ]
    ]
  }
}
# cat a.json | ./redisdl.py -l -H $redis_host -d 0 -E iso-8859-1 --empty
Traceback (most recent call last):
  File "/tmp/./redisdl.py", line 632, in <module>
    main()
  File "/tmp/./redisdl.py", line 628, in main
    do_load(options, args)
  File "/tmp/./redisdl.py", line 556, in do_load
    load(input, **kwargs)
  File "/tmp/./redisdl.py", line 466, in load
    load_lump(fp, host=host, port=port, password=password, db=db,
  File "/tmp/./redisdl.py", line 347, in load_lump
    loads(s, host, port, password, db, empty, unix_socket_path, encoding, use_expireat=use_expireat)
  File "/tmp/./redisdl.py", line 328, in loads
    _writer(r, p, key, type, value, ttl, expireat, use_expireat=use_expireat)
  File "/tmp/./redisdl.py", line 483, in _writer
    p.zadd(key, element, score)
  File "/usr/local/lib/python3.10/dist-packages/redis/commands/core.py", line 4167, in zadd
    for pair in mapping.items():
AttributeError: 'str' object has no attribute 'items'

Setting -y for output creates invalid json

When setting -y for the output it seems to create invalid JSON. Removing -y creates valid JSON. JSON without -y can be imported back into Redis. However, JSON created with -y fails to be imported

jsaone support

toobaz@12b345f

Pickled values

I am storing pickled python object as Redis values.
I would like to specify function that convert (depickle/jsonifu) redis values. Is it possible?

Should be able to empty the database with keys in any encoding by passing byte strings

redis-load is not adding the data to redis due to invalid ttl value

Hello,

I am new to use this utility.
I am seeing an issue in which dump works well but load is not working or in other words redis set is not working.

My dump file looks as follows.
./redisdl.py --pretty > dump.json
{
"myKey": {
"expireat": 1565594482.6172411,
"ttl": -0.001,
"type": "string",
"value": "10"
},
"myhash": {
"expireat": 1565594482.61765,
"ttl": -0.001,
"type": "hash",
"value": {
"value": "key"
}
}
}

So while adding since ttl is -1 it is not able to set the data in redis.
./redisdl.py -l dump.json

This is due to this.
https://github.com/p/redis-dump-load/blob/master/redisdl.py#L490
r.pexpire_or_expire_pipeline(p, key, ttl)

If i comment the line then the data insertion is OK.
Basically what the function pexpire_or_expire_pipeline does is return p.pexpire(key, int(ttl * 1000)), so ttl is set to -1 and hence the key is expired.

So could someone tell me how to get around this issue?
Or do i need to pass something else during dump, i didnt see any in help text/code.

Same issue is seen even with --use-expireat as well.

Is this known issue or am i missing something?

Add changelog

What is the encoding format used for keys written in binary-safe mode in Redis?

We have some redis Hash keys which are storing values as c++ structures and for that we have used binary safe mode.
Example:

redisReply* reply = (redisReply*)redisCommand(ctxt, "HMSET %b %b %b", KEY.c_str(), strlen(KEY.c_str()), &val,
                                            sizeof(KEY), &temp_element, sizeof(temp_element));

keys looks like this in redis DB:

1. "\x14\x10"
2. "KEY_dc8d8afe8e2a_301_0.4116_ECpriReceptionMonitor\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"

Where KEY is integer value and temp_element is a structure.

But when we try to take dump using redisdl module as show below:

db_dump=redisdl.dumps(db=db_instance,pretty=True)

it gives below error:

'utf-8' codec can't decode byte 0x90 in position 1: invalid start byte

we tried with encoding='iso-8859-1' seems to work but how to check if this is the correct encoding format?

Br,
Neeraj

Test jsaone on travis

Fix #56 for unix_socket_path

Aliasing as dump does not work

Instagram password recovery

Instagram password recovery

Unknown distribution option: 'entry_points' warnings.warn(msg) when install

/usr/lib64/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'entry_points'
warnings.warn(msg)
/usr/lib64/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'install_requires'
warnings.warn(msg)

export file to large, load error

json file is 3GB, load error, print 'Killed'

Support expiring keys

• Dump both ttl and expireat
• Use PTTL if available
• Option to use ttl or expireat when loading

ssl support

I do not find the support for the SSL if the redis has encryption enabled.

get following error:
TypeError: dumps() got an unexpected keyword argument 'ssl'

Separate redis and file encodings

Requires redis-py 2.x

Back in November 2018 redis-py updated to a 3.x version that has breaking changes to it, this tool is not compatible with redis-py 3.x. Noteable reasons why:

• It no longer coerces input into strings. Right now (at least with python 2.7) is passes unicode values to redis-py which causes it to throw an exception about unicode not having an iteritems method.
• ttl returns now match Redis' api and negative values imply that there is no ttl. If you dump with redis-py 3.x and then try to load with redis-py 2.x then your keys without ttls will be immediately deleted by redis because it'll set them to negative values rather than setting no ttl.

Tool still works fine if you just always use it with a compatible version or redis-py. I'd suggest adding some code to the tool to error out if you try to use it with a version it's not compatible with and a note in the README.md saying as much.