yaroslaff / ncdu-compare

Compare ncdu export files and show difference to find what took most space since previous measurement

ncdu-compare

Compare ncdu export files and shows differences to find out what consumes most space.

Installation

# from pypi
pip3 install ncdu-compare

# ... or from git
pip3 install git+https://github.com/yaroslaff/ncdu-compare.git

Example usage

# measure current /var/log 
$ ncdu /var/log/ -0o /tmp/1

# create extra 1M file... and just wait
$ sudo dd if=/dev/zero of=/var/log/zzz.log bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00229604 s, 457 MB/s

# measure /var/log again
$ ncdu /var/log/ -0o /tmp/2

# see top usage
$ ncdu-compare /tmp/1 /tmp/2 | sort -n | tail -n 5
575 FILE /var/log/auth.log (273797 > 274372)
12209 FILE /var/log/daemon.log (62452009 > 62464218)
12286 FILE /var/log/syslog (63463377 > 63475663)
1048576 FILE /var/log/zzz.log (0 > 1048576)
1074015 DIR /var/log (3895153251 > 3896227266)

Invalid unicode filenames

Sometimes ncdu can produce invalid JSON files (see https://dev.yorhel.nl/ncdu/jsonfmt). When loading ncdu files ncdu-compare uses (default) 'replace' mode to handle unicode errors. You can override it with -e/--error options, e.g. --error ignore. More at python doc to open().

See also

My other project Plus Size to detect changes in short-time (e.g. fast growing log files)

ncdu-diff C + Python ncdu fork that can compare and diff results

ncdu-compare's Issues

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 6124037: invalid start byte

Hi im trying to compare two ncdu outs. But still getting error
with command ncdu -0xo- /data/www/ --exclude /data/www/domains/tmp > blah

Ncdu is this version
ncdu 1.15.1

pschonmann@E495:~/temp$ ncdu-compare ncdu_2023-02-09_094201 ncdu_2023-02-09_094301 
Traceback (most recent call last):
  File "/home/pschonmann/.local/bin/ncdu-compare", line 137, in <module>
    main()
  File "/home/pschonmann/.local/bin/ncdu-compare", line 112, in main
    old_data = json.load(fh)
  File "/usr/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 6124037: invalid start byte

Speedup for large ncdu files ?

Tried to compare large ncdu files ... it takes ages :)

pschonmann@E495:~/temp/$ wc -l ncdu_2023-02-*
1449422 ncdu_2023-02-09_103801
1451621 ncdu_2023-02-10_223801

real 361m4.665s
user 358m49.615s
sys 1m16.661s

and hit ctrl+c to stop waiting

compare only dirs/files/both mode

Hi, would be fine to compare only files/dirs/both with some --option [default both]

For example i want only compare files, but not directories, but sometimes vice versa

Here is example where "dupes" occurs
138851 /data/www/ftp_data_external/exp_documents/1422569840 (0 > 138851)
138851 /data/www/ftp_data_external/exp_documents/1422569840/373446.pdf (0 > 138851)

Is it possible ? At first look to output file from ncdu im not sure tell that row is folder or file. But looks like structured json file. Im not good at this python things :)

Mark it as feature request with low prio :D
Anyway thanks and have a nice day.