Coder Social home page Coder Social logo

catcherwong / rdb-tools Goto Github PK

View Code? Open in Web Editor NEW
42.0 4.0 10.0 549 KB

⚡ An analysis tool of the Redis RDB files, which includes library and CLI.

License: Apache License 2.0

C# 93.41% HTML 6.18% PowerShell 0.20% Shell 0.20%
redis rdb cli analysis memory csharp library

rdb-tools's Introduction

rdb-tools

rdb-tools is a tool to parse/analysis redis rdb files that is implemented by csharp.

This repository is inspired by redis-rdb-tools and rdr.

It consists of two parts:

  • A parser library, that you can customize by yourself.
  • An anslysis cli tool, that your can ans some basic usages for command line.

rdb-cli

Install

  1. Download the package from the latest stable release.
  2. dotnet tool install --global rdb-cli

Usage

Show help information:

[~] ./rdb-cli -h      
Description:
  rdb-cli is a command line tool, analysis redis rdb files.

Usage:
  rdb-cli [command] [options]

Options:
  -?, -h, --help  Show help and usage information
  -v, --version   Show version information

Commands:
  keys <file>    Get all keys from rdb files
  memory <file>  Get memory info from rdb files

The usage of memory is as follow:

[~] ./rdb-cli memory -h
Description:
  Analysis memory info from rdb files

Usage:
  rdb-cli memory <file> [options]

Arguments:
  <file>  The path of rdb file.

Options:
  -o, --output <output>                                  The output path of parsing result.
  -ot, --output-type <html|json>                         The output type of parsing result. [default: json]
  -tp, --top-prefixes <top-prefixes>                     The number of top key prefixes. [default: 50]
  -tb, --top-bigkeys <top-bigkeys>                       The number of top big keys. [default: 50]
  --db <db>                                              The filter of redis databases.
  --type <hash|list|module|set|sortedset|stream|string>  The filter of redis types.
  -?, -h, --help                                         Show help and usage information
[~] ./rdb-cli memory /tmp/demo.rdb -ot html -tb 200

Prepare to parse [/tmp/demo.rdb]
Please wait for a moment...

parse cost: 22449ms
total cost: 23107ms
result path: /tmp/res.html

Sample html result is as follow:

Sample json result is as follow:

{
    "usedMem": 2373094496,
    "cTime": 0,
    "count": 7615333,
    "rdbVer": 9,
    "redisVer": "5.2.0",
    "redisBits": 0,
    "typeRecords": [
        {
            "Type": "sortedset",
            "Bytes": 1385664695,
            "Num": 6212084
        }
    ],
    "largestRecords": [
        {
            "Database": 2,
            "Key": "key",
            "Bytes": 10340,
            "Type": "string",
            "Encoding": "string",
            "Expiry": 0,
            "NumOfElem": 8318,
            "LenOfLargestElem": 0,
            "FieldOfLargestElem": null
        }
    ],
    "expiryInfo": [
        {
            "Expiry": "0~1h",
            "Bytes": 986801692,
            "Num": 4345021
        }
    ],
     "functions": [
        {
            "Engine": "lua",
            "LibraryName": "mylib"
        }
    ],
    "largestStreams": [
        {
            "Key": "key",
            "Length": 5,
            "LastId": "1650158935767-0",
            "FirstId": "1650158906951-0",
            "MaxDeletedEntryId": "0-0",
            "EntriesAdded": 5,
            "CGroups": 0
        }
    ]
}

RDBParse

Install

dotnet add package RDBParse

Usage

  1. Implement your own IReaderCallback
  2. Create a new instance of BinaryReaderRDBParser
  3. Call Parse method of BinaryReaderRDBParser instance

Following this below code for example.

public class MyReaderCallBack : IReaderCallback
{
}
var path = "/yourpath/your.rdb"
var cb = new MyReaderCallBack();
var parser = new RDBParser.BinaryReaderRDBParser(cb);
parser.Parse(path);

To contribute

Contributions are welcome!

rdb-tools's People

Contributors

catcherwong avatar incerrygit avatar phoeniwx avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

rdb-tools's Issues

通过RDB文件分析Permanent Key 导出html文件格式时,分析结果呈现不准确

同一个RDB文件,分别进行如下两次分析,Permanent分析结果差异较大:
1.第一次全量分析导出html文件:
./rdb-cli memory /redis/dump.rdb -ot html -tp 50 -tb 50 --separators : --sep-count 3
718

2.第二次只分析永久的key导出html文件:
./rdb-cli memory /mnt/data/redis/dump.rdb -ot html -tp 50 -tb 50 --separators : --sep-count 3 --permanent

Dingtalk_20240718114847
Dingtalk_20240718114758

数据分析导出html文件时无法自定义文件名称

执行如下命令,添加 -o参数输出指定文件名时,会创建一个07-18.html文件夹,文件夹下才是分析结果res.html文件,无法自定义修改输出文件名称
./rdb-cli memory /redis/dump.rdb -ot html -tp 50 -tb 50 --separators : --sep-count 3 --permanent -o /tmp/07-18.html

查看key的空闲时间

Description

支持查看key的空闲时间分布

Steps to Reproduce

  1. RDB文件中有保存LRU空闲时间,建议增加key的空闲时间分布图,并在下面tab中增加top空闲时间的某一前缀key

Related code

insert short code snippets here

Expected behavior: [What you expected to happen]

Actual behavior: [What actually happened]

Specifications

  • Scope : CLI / LIB
  • Version : 0.1.0
  • System : CentOS 7.2 (linux-x64)

支持某一后缀

Description

业务历史原因,key前缀都不一致,导致使用rdb分析工具无法准确分析,是否可以支持某一后缀key的分析
好比一个key是 a:b:c

前缀就变成 c , c:b

Specifications

  • Scope : CLI / LIB
  • Version : 0.1.0
  • System : CentOS 7.2 (linux-x64)

Reading a hash field name that looks like a number

Description

[Description of the bug or feature]

Steps to Reproduce

  1. Read a hash field name that looks like an int (eg. 20000486456478)

ReadZipListEntry will enter

else if (entryHeader >> 4 == 14)
            {
                value = br.ReadBytes(8);
            }
  1. Result is a set of 8 bytes that when converted to a string does not result in "20000486456478"

Sorry I understand next to nothing about how this works. I assume the bytes are correct, running them through UTF8 GetString is probably the wrong thing to do

Problem: long.TryParse not working right sometimes.

Description

工具在处理部分rdb文件时,输出结果的cTime和redis-bit为0

Steps to Reproduce

rdb header:
image

ctime实际值为1668071874,但程序分析的结果为0

Related code

查看代码发现程序使用long.TryParse方法解析ctime,查阅.net文档发现该方法仅用于处理十进制数字字符串
image
因此ctime utf-8解码后包含lc就解析失败了,导致结果为0

// MemoryCallback.cs
else if (keyStr.Equals("ctime"))
{
    var time = System.Text.Encoding.UTF8.GetString(value);
    if (long.TryParse(time, out var ctime)) //这一步错误
    {
        _rdbDataInfo.CTime = ctime;
    }
}

我看程序中除了ctime, redis-bits,还有used-mem也使用了该方法,不知道它们是否也会在某些情况下出问题

目前我的解决方案如下:

else if (keyStr.Equals("ctime"))
{
  if (value.Length==4) {
      var ctime = System.BitConverter.ToInt32(value);
      _rdbDataInfo.CTime = ((long) ctime);
  }

Valid values are between -62135596800000 and 253402300799999, inclusive. (Parameter 'milliseconds')

Description

[Description of the bug or feature]

Steps to Reproduce

  1. CentOS Linux release 7.9.2009 (Core) 执行命令
nohup ./rdb-cli memory ./8-3-7003.rdb -ot html -tb 200 &
  1. 报错信息:
    Valid values are between -62135596800000 and 253402300799999, inclusive. (Parameter 'milliseconds')
    Valid values are between -62135596800000 and 253402300799999, inclusive. (Parameter 'milliseconds')
    Valid values are between -62135596800000 and 253402300799999, inclusive. (Parameter 'milliseconds')
    ........

Related code

insert short code snippets here

Expected behavior: [What you expected to happen]

Actual behavior: [What actually happened]

Specifications

  • Scope : CLI / LIB
  • Version : 0.8.0
  • System : CentOS Linux release 7.9.2009 (Core)(linux-x64)

Add option to ignore FieldOfLargestElem

Description

When many keys contain big value, store those big value will lead to memory usage increase quickly.

We can add an option to skip record the FieldOfLargestElem

Steps to Reproduce

Related code

./rdb-cli memory ./6379.rdb -ot json -tb 50

Expected behavior: [What you expected to happen]

Actual behavior: [What actually happened]

Specifications

  • Scope : CLI / LIB
  • Version : 0.1.0
  • System : CentOS 7.2 (linux-x64)

分析数据时支持按指定的key前缀来分析数据量的大小

在使用工具分析RDB文件时,有个指定的配置可以设置指定的key前缀来进行具体分析,分析结果可以直接显示输出该key前缀占用的大小、总共有多少元素等,亦或者是将该key前缀的分析结果导出成html、csv文件等等

Improve key output when the key length is very long

Description

When set a key who's length is very long, the output result of this key is abnormal.

Steps to Reproduce

set dsfsd:sdfsdfs:232342342342342344444444444444444444444444444444444444444444444444444444444:34sdfdsfsdfdslkajsdoifuoiweuqrlekrnkjhaoysdafhjkjhasdfoihkhjkkjhakdsfjhk:sadfkh324hkhy98huinjksdfah89hhhkjasf:adf8iousafhjkasdf89yuh23489ahkjnsf:adfoiujmasdfjhkahskdhfo89237489237402iaoihefjwehuhgakjhiuyieq:2348uuyuashdfoiasiodfhohjkhuiuy9ay9huuioooouiuiouiouiousdf:asdlfkjouoiu2893rujoinnsdihkghajsgdfhjg7i23y4y2938uuayiiewr:asjkdfhiuyiy2ihkjshfuyw98y2urhjknasdf:23y987weriyuhkjbuiyiqwer:879879798247397428934yiuyieyiurwyeurywiuerywiueyriuweyriuweryuiweyruiwyeruiwyeuirywuier234234324789234798237498237498237498723984723987489237498237498237492374923794723849723984789237489273wueyiweyriuweyriuweyriweuyrwiueryiwueyrwiuerywieuryweiruyweuiryweiuryweuiryweiryweuryiweryweuiryweiryweirskhdkfjs:popopopopopopopoppwerwerwerewrwerwerwerwerwerwerwerwerwerwerwerwerwerwerwerwerwerwerwerwerewjkdjfklsdjflksdjflksdjfkljsdklfjdklfjdskljkfjsdxcnvmxcbvmbxcnvmbxcnmvbxmcbvmxcbvmxcbnvmxbcvmxncbnmxbvnmxbcmvbcxmvbxcbvmxcbnvmbxcmnbv23847923749238749327498327498723894789237489237498723948237984iuewioruwoeiruowieruwoeursdfkjshdk:kskskskskskskskskksksiwiwiwiwiiwiwiwiwiwiiwiwi2299292929isdfhskdfjhsjkdhf823894723897hsdfkhsdfusaduiyfisdyfiusdyfiusayfiusdyfiuaydifuwkerjhwekrhwejrhwekjrhiyiyxicuvyuixv:qqqwqwqwqwqqwqwqqwqqwqwqqwqwqwqwqwbmnbmncbvnmxcbvmxcbvnmxbcmvbxmcvbmnxcbvnmwkeruweyriweyri2394792374892374982374982734928uoiuoiuowiuroieuwiroweuiruwieroweuir 123

image

Related code

insert short code snippets here

Expected behavior: [What you expected to happen]

dsfsd:sdfsdfs:...(xxx more bytes)...oweuir 

Actual behavior: [What actually happened]

Specifications

  • Scope : CLI / LIB
  • Version : 0.1.0
  • System : CentOS 7.2 (linux-x64)

bug: 多级前缀统计不支持“-”分隔符

Description

[Description of the bug or feature]

TypeKey类型的反序列化方法使用“-” split,导致key里面本身的“-”也被分割
如果出现a-b-c这样的前缀,在CalcuLargestKeyPrefix函数中会对之前的TypeKey字符串进行反序列化,
导致结果只剩一个a

Related code

// TypeKey.cs
public static TypeKey FromString(string str)
        {
            var tk = str.Split('-');
            return new TypeKey { Type = tk[0], Key = tk[1] };
        }

// RdbDataCounter.cs
private void CalcuLargestKeyPrefix(int num)
        {
            foreach (var item in _keyPrefix)
            {
                var tk = TypeKey.FromString(item.Key);
                var ent = new PrefixRecord
                {
                    Type = tk.Type,
...

Fix

// TypeKey.cs
public static TypeKey FromString(string str)
        {
            var tk = str.Split('-', 2);
            return new TypeKey { Type = tk[0], Key = tk[1] };
        }

增加未设置过期时间的top Key Prefix

Description

单独增加未设置过期时间的报表分析

Steps to Reproduce

  1. 单独增加展示未设置过期时间的top Key Prefix

Related code

insert short code snippets here

Expected behavior: [What you expected to happen]

Actual behavior: [What actually happened]

Specifications

  • Scope : CLI / LIB
  • Version : 0.1.0
  • System : CentOS 7.2 (linux-x64)

支持分析未设置过期时间的key前缀分布情况

对于未设置ttl的key支持导出csv格式或者html格式的数据分析结果,可以根据key命名分隔符进行层级过滤,亦或者可以获得指定key前缀的分析结果;例如执行该命令: ./rdb-cli csv /mnt/dump.rdb sep-count 3 --separators :

如果能够支持html格式,可以加个--permanent 参数,这样比较全面看到哪些key未设置ttl,以及这些key内存占用分布情况
例如 ./rdb-cli memory dump.rdb --sep-count 3 --separators : --permanent true -ot html -tb 200

Can't parse keys with key name contains blank

Description

If key name contains blank, for example: "a big key", then tool can't parse these keys.

Steps to Reproduce

  1. set "a big key" xxxx
  2. rdb-cli keys rdbfile > key.txt, and the result not contains the "a big key"

Thanks.

Specifications

  • Scope : CLI / LIB
  • Version : 0.6.1
  • System : CentOS 7.2 (linux-x64)

支持查看已过期key某一前缀

Description

支持查看已过期key某一前缀

Steps to Reproduce

  1. 业务可能存在大量已过期但没任何访问的key,需要查看已经过期key的前缀分别是什么,在下面tab中进行展示
    image

Specifications

  • Scope : CLI / LIB
  • Version : 0.1.0
  • System : CentOS 7.2 (linux-x64)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.