Backup Formats/Compression Method Allowed about clickhouse

[nikitamikhaylov]

Should I assume the same as with when creating a tar?

ClickHouse/src/IO/Archives/TarArchiveWriter.cpp

Lines 11 to 18 in fe92c92

	void TarArchiveWriter::setCompression(const String & compression_method_, int compression_level_)
	{
	// throw an error unless setCompression is passed the default value
	if (compression_method_.empty() && compression_level_ == -1)
	return;
	throw Exception(
	ErrorCodes::NOT_IMPLEMENTED, "Using compression_method and compression_level options are not supported for tar archives");
	}

Then we can also compress when archiving into a zip but what compression methods are supported?

ClickHouse/src/IO/Archives/ZipArchiveWriter.cpp

Lines 344 to 362 in fe92c92

	int ZipArchiveWriter::compressionMethodToInt(const String & compression_method_)
	{
	if (compression_method_.empty())
	return MZ_COMPRESS_METHOD_DEFLATE; /// By default the compression method is "deflate".
	else if (compression_method_ == kStore)
	return MZ_COMPRESS_METHOD_STORE;
	else if (compression_method_ == kDeflate)
	return MZ_COMPRESS_METHOD_DEFLATE;
	else if (compression_method_ == kBzip2)
	return MZ_COMPRESS_METHOD_BZIP2;
	else if (compression_method_ == kLzma)
	return MZ_COMPRESS_METHOD_LZMA;
	else if (compression_method_ == kZstd)
	return MZ_COMPRESS_METHOD_ZSTD;
	else if (compression_method_ == kXz)
	return MZ_COMPRESS_METHOD_XZ;
	else
	throw Exception(ErrorCodes::CANNOT_PACK_ARCHIVE, "Unknown compression method specified for a zip archive: {}", compression_method_);
	}

You can read the code located in this folder yourself :) There are not so many lines written there.

So, you can either determine the compression method by a file extension or force the archive "algorithm" to use a specific codec internally if it is supported.

[nikitamikhaylov]

I briefly looked into the code and compression_method is only used when ClickHouse writes a backup in an archive. Moreover, data is already compressed well (columnar format, etc), so I'm not sure if it can improve things significantly.

What are all the options in terms of file extension names and what exactly do they do?

They are used to determine which archive algorithm to use. The whole list of possible extensions can be found here:

ClickHouse/src/IO/Archives/createArchiveWriter.cpp

Lines 29 to 30 in fe92c92

	".tar"sv, ".tar.gz"sv, ".tgz"sv, ".tar.bz2"sv, ".tar.lzma"sv, ".tar.zst"sv, ".tzst"sv, ".tar.xz"sv};
	if (path_to_archive.ends_with(".zip") || path_to_archive.ends_with(".zipx"))

[RickyBoyd]

Thanks for taking a look at this for me.

So if I understand correctly, I can either make a tar, a zip or neither.

If I create a tar then I can compress with various different methods, based on the file extensions the options are:

• zstandard
• lzma
• xz
• bzip2
• gunzip

Then we can also compress when archiving into a zip but what compression methods are supported? Should I assume the same as with when creating a tar?