chunkserver: please implement data compression [feature] about moosefs

[TerraTech]

Has there been any implementation research work done on this since 2/2019?

[onlyjob]

Btrfs supports compression. But on large file systems its performance degrade significantly over time due to fragmentation. Also Btrfs requires periodic balancing. Unfortunately Btrfs is not the best fit for Chunkservers on rotational HDDs but I would consider using it on SSDs.

[marcin-github]

If chunkserver would send compressed data directly from storage to client (and client would decompress it) we could get gain in network throughput. At cost of higher CPU usage on client side.

[onlyjob]

LZ4 overhead is negligible and its speed is close to RAM-to-RAM copy:

LZ4 is a very fast lossless compression algorithm, providing compression speed > 500 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems.

Also LZ4 was implemented natively in the Linux kernel 3.11.

[chogata]

There are two separate issues raised in this thread:

• compression of chunk data on chunkserver - this is on our roadmap, but it doesn't have a high priority, as we feel that you can use other tools (like local filesystem with compression) if this is something you really need and most MooseFS installations we know store data that is already compressed, meaning this feature would be useless in them anyway; there are other features we are currently implementing that we feel will be useful in wider range of installations
• compression of chunk data in chunkserver - client communication - well, the problem is, we see more and more MooseFS installations using 10Gb networks and no compression algorithms are fast enough for that...

[onlyjob]

* compression of chunk data in chunkserver - client communication - well, the problem is, we see more and more MooseFS installations using 10Gb networks and no compression algorithms are fast enough for that...

That depends on type of data. Highly compressible data might benefit from compression and reduce traffic congestion.
Also 10Gb network may not be used exclusively so compression might still yield improvements under many circumstances.
Let's just make it configurable, perhaps by chunkserver config option so compression could be enabled where required (e.g. on 100MB links).

[jkiebzak]

Curious, when you say:

we feel that you can use other tools (like local filesystem with compression)

what tools or filesystems have been used? I only know of ZFS that supports compression. Are there others?

[inkdot7]

Compression could leverage the very handy storage classes, such that compression happens lazily:

A user could request data to be cheaply compressed using e.g. lz4 when the chunks get into 'keep' mode, and using something gzip-like if/when files become old enough to reach archive. Even two archive levels could then be useful, to request even more expensive xz-like compression for files that are even older.

The compression also would not need to happen at the time of the storage class move. It would be enough to just consider chunks that are at least that old eligible for compression, whenever the chunkserver feels it has spare CPU cycles.

Also, all copies of a chunk need not have the same compression. Then reading could prefer the less compressed version, while having redundancy with higher compression.

At least for the last stage, compression could also be avoided when the previous stage did not manage to squeeze out e.g. even a percent, as such files likely already are compressed, or contain in-compressible data.

[asyslinux]

Another problem. I have btrfs with zstd compression, and i have MooseFS chunks on btrfs.

But when i add new files, then MooseFS calculate internally uncompressed size of uploaded data and ignores df -h real size with compressed data.

df -h shows correct information (zstd compressed data):

First server with two btrfs partitions:

/dev/nvme0n1p2          842G  2.7G  838G   1% /mnt/btrfs1
/dev/nvme1n1p2          842G  2.7G  838G   1% /mnt/btrfs2

Second server with two btrfs partitions:

/dev/nvme0n1p2          842G  2.7G  838G   1% /mnt/btrfs1
/dev/nvme1n1p2          842G  2.7G  838G   1% /mnt/btrfs2

Summary real compressed size is 2.7+2.7 = 5.4GB
MooseFS shows used space: 14GB from 1.6TB of two btrfs partitions from first server and same from second server.

I understand, moosefs don`t know about compression inside btrfs.

As I understand it, it turns out that compression using native fs like btrfs is simply useless, because when MooseFS considers that the space has run out, MooseFS simply will not allow you to write more data, although in reality there will be plenty of space in btrfs. Is there no way around this?

[chogata]

This is strange, because MooseFS should be showing the same thing that df does... Chunk server doesn't calculate data size from chunks, it uses the statvfs system function to check the total disk size and available size. And then calculates used from that.
Unless you use size limiting in hdd.cfg, then it will calculate the size based on chunks, which will not take compression into account. Can you post the content of your hdd.cfg from one of these servers?

[asyslinux]

@chogata, Hi.

root@srv-1:~# cat /etc/mfs-chunk_btrfs/mfshdd.cfg
/mnt/btrfs1/moosefs-chunk
/mnt/btrfs2/moosefs-chunk

But I have this option:

root@srv-1:~# cat /etc/mfs-chunk_btrfs/mfschunkserver.cfg | grep HDD_LEAVE
HDD_LEAVE_SPACE_DEFAULT = 4GiB

[chogata]

Ah, that explains it :) A chunk server has no other way to tell the master that it needs to reserve 4GiB, other than to say it's occupied. So, per server, each drive has 3GiB of data (rounded up), 4GiB of reserved space. 3+4+3+4 is 14 :)

HDD_LEAVE_SPACE_DEFAULT setting doesn't change the way the used space is calculated by a chunk server, it still uses statvfs. Only the restriction on space per disk in hdd.cfg does.

The assumption is, HDD_LEAVE_SPACE_DEFAULT is a small amount of space left "just in case" (like most OSes will always reserve some space on / for root user only). But if one uses limiting in hdd.cfg, then probably the disks are shared and the chunk server is not the only process that uses the disks to store data. Then using statvfs would count other data - not chunks - as chunks and that's why in this situation we use the "manual" calculation.

[asyslinux]

@chogata, Thanks. Yeah, sorry, this is my mistake, option HDD_LEAVE_SPACE_DEFAULT = 4GiB reserves ~4GB per btrfs partition, i have 4 partitions on 2 chunkservers - i was simply misled by this.

Used space is correctly calculated by MooseFS with compression in BTRFS.

A short off-topic question: when will MooseFS / PRO 3.0.117 packages be released for Debian 12? :)