vaeth / zram-init Goto Github PK

If you use really big zrams, like I used 11GB for portage compilation, the filesystem contains random data over the time and begin to use a lot of RAM. the "discard" mount option removes files on delete from the filesystem entirely instead of just removing the reference, making the memory used by the zram device less.

I think discard is supported since linux-2.6.33 and ext4.

If this option won't be added as a default options, maybe a notice in the config file would be nice (something like: if you want to use big devices, please use ext4 with discard).

This would effect the openrc conf.d file and the systemd service files.

Ubuntu/Debian etc use a sane default not predetermined number for enabling zram

# Calculate memory to use for zram (1/2 of ram)
totalmem=`LC_ALL=C free | grep -e "^Mem:" | sed -e 's/^Mem: *//' -e 's/  *.*//'`
mem=$(((totalmem / 2 / ${NRDEVICES}) * 1024))

Same system as in #14 but now with util-linux installed for zramctl suport.

No log, except on console.

In this case with zramctl installed, it will not create a zram device at all.

zramctl in Alpine 3.9.1:

alpine@ip-172-31-43-27:$ zramctl -V
zramctl from util-linux 2.33
alpine@ip-172-31-43-27:$ zramctl --help

Usage:
 zramctl [options] <device>
 zramctl -r <device> [...]
 zramctl [options] -f | <device> -s <size>

Set up and control zram devices.

Options:
 -a, --algorithm **lzo|lz4|lz4hc|deflate|842**   compression algorithm to use
...

alpine@ip-172-31-43-27:~$ cat /proc/swaps
Filename                                Type            Size    Used    Priority

No zstd support.

Maybe defaulting to zstd in the default /etc/conf.d/zram-init might be a bit premature.

There is no support for syslog. Please log something if there are failures.

I was only able to find this on a screenshot of a AWS console. There is no other message about it on the system.

The zram in this case defaults back to lzo:

alpine@ip-172-31-43-27:~$ cat /sys/block/zram0/comp_algorithm
[lzo] lz4 lz4hc

Would really be nice to have zstd compression support, since zstd is now included in >= kernel 4.14.x!

mkfs.ext4: Invalid argument while setting blocksize; too small for device

mkfs.ext4 /dev/zram1 failed

Configure sys-block/zram-init so that there is one service per mount point or swap configuration, in a similar manner that net-misc/tigervnc does per display.
This would enable:

1. starting and stopping each config separately.
2. Configuring separate writeback daemons for each config.

Currently, the service config file's documentation has a bunch of flags that aren't explained properly, if at all.
Which, for the config file, is fine, but they are not documented anywhere else, either.

For example, the backup device flags, the only documentation provides is:

# back0, back1, ... are the variables for the backup device
# icmp0, icmp1, ... are the flags (if nonempty) for writing incompressible
#                   pages to the backup device (requires setting back*)
# idle0, idle1, ... are the flags (in nonempty) for writing idle pages to the
#                   backup device (requires setting back*)
# wlim0, wlim1, ... set the writeback_limit for idle pages (implies idle*)

[...]

back0= # no backup device
icmp0= # no incompressible page writing to backup device
idle0= # no idle page writing to backup device
wlim0= # no writeback_limit for idle page writing for backup device

Nothing tells me what, exactly, I should be entering on icmp0 or idle0 to enable it. Do I enter a boolean? An int? The backup device path?
The man page doesn't touch on the config file at all, and no external documentation appears to be present.

I guess that it is because some processes are still alive and using this path. But what is really strange is that the localmount service unmounts this zRAM device successfully at the end of the shutdown process. I do not know enough about OpenRC to offer a solution. I am afraid that is a bad idea but maybe separate the stop process in another service (like zram-stop) which depends on killprocs (after killprocs).

It would be nice if options could be passed to mkfs through the zram-init config file.

One option I am missing is for example the possibility to set a filesystem label such that graphical file explorers (Dolphin etc.) show that label instead of something like "24 GB device", which is their default for nameless filesystems.

Thanks for the beautiful tool and cheers,
Oliver

I just wanted to test zram_tmp.service and zram_swap.service in my systemd box, but only the zram_swap.service works well. The other breaks my system from the start for some reason. I don't know what I did wrong -- I masked the tmp.mount via:

 # systemctl mask tmp.mount

and enabled zram_tmp.service:

# systemctl enable zram_tmp.service

When I boot, it looks like it didn't start with the system:

● zram_tmp.service - Mount /tmp as zram. Mask tmp.mount if you use this
   Loaded: loaded (/etc/systemd/system/zram_tmp.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

So there's no /tmp mount, and that's my system went RO, and some services were broken.

Surprisingly, when I start the service manually, it works.

Hello,

sorry if this the wrong place, but after searching a bit I didn't find any information about it. Am I supposed to create a filesystem on that partition? What type should it be? Should I mount it?
Also, I didn't find any "echo huge > /sys/block/zramX/write" or "echo all > /sys/block/zramX/idle" in zram-init, why is that?

I mainly try to read https://www.kernel.org/doc/Documentation/blockdev/zram.txt to get information, to no avail.

Problem

Help text in example configuration /etc/conf.d/zram-init says:

# mlim0, mlim1, ... are the variables for the maximal uncompressed memory (MB)

Using this option as per the help text results in service startup failure, as the filesystem creation fails on the ZRAM device. Any value written to this parameter will be interpreted in bytes, with a 8kB memory limit (minimum amount apparently) for any value less than 8192 (e.g. 6144 is supposed to be 6GB, but it's understood as 6kB, which is rounded up to the minimum of 8kB), and the actual kB amount above that.

File system creation fails, because we immediately fill up the tiny memory limit, regardless of the disk size specified for the ZRAM device.

Example failure when starting the zram-init service with ext2 filesystem:

# rc-service zram-init start
 * Caching service dependencies ...                                                                                                                          [ ok ]
 * Loading zram module...                                                                                                                                    [ ok ]
 * Mount zram0 /var/tmp
mke2fs 1.46.1 (9-Feb-2021)
mkfs.ext2: Input/output error while writing out and closing file system
zram-init:
mkfs.ext2 /dev/zram0 failed

Example failure when starting the zram-init service with btrfs filesystem:

# rc-service zram-init start
 * Caching service dependencies ...                                                                                                                          [ ok ]
 * Loading zram module...                                                                                                                                    [ ok ]
 * Mount zram0 /var/tmp
No valid Btrfs found on /dev/zram0
ERROR: open ctree failed
zram-init:
mkfs.btrfs /dev/zram0 failed

Tried filesystem creation manually for more verbose error messages:

# mkfs.ext2 /dev/zram0
mke2fs 1.46.1 (9-Feb-2021)
Discarding device blocks: done
Creating filesystem with 2097152 4k blocks and 524288 inodes
Filesystem UUID: dec334c6-b337-4b60-9187-5a1acff26bda
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: mkfs.ext2: Input/output error while writing out and closing file system

# mkfs.btrfs -f /dev/zram0
btrfs-progs v5.10.1
See http://btrfs.wiki.kernel.org for more information.

Detected a SSD, turning off metadata duplication.  Mkfs with -m dup if you want to force metadata duplication.
No valid Btrfs found on /dev/zram0
ERROR: open ctree failed

Rootcause

This setting expects only numbers as per zram-init.in:L262, and will not take sizes with unit suffixes, e.g. 1024M, or 4G. The number set for memory limit is actually in bytes not in megabytes because the raw number is passed to SysCtl function zram-init.in:L388, which simply writes it into the /sys file system on zram-init.in:L68.

Proposed solution

Since this option is most likely used as a safety feature (e.g. preventing ZRAM to consume all memory for larger filesystems), the solution should be backward compatible with existing configurations, where people set the value in bytes.

I propose to allow passing in numbers with unit suffixes in mlim option, e.g. 512M or 4G, since the ZRAM /sys filesystem supports it. E.g.:

echo 4G > /sys/block/zram0/mem_limit

When there is a device configured as typeN=- in /etc/conf.d/zram-init, it is completely ignored by zram-init openrc script.

According to the docs, typeN=- is a supported type, meaning that device and filesystem will be created, but not mounted.

It seems that case arg is missing a - variant here:
https://github.com/vaeth/zram-init/blob/main/openrc/init.d/zram-init#L72
and that line should be changed to:

	swap|/*|-)

After applying this change, typeN=- devices are created as described in docs.

Need to add aufs in zram_init because it will integrate zram + tmp in ram + tmp in disk. Operating procedure:

1. Create a disk zram
2. Make in zram ext4
3. Make aufs = "ext4 in zram" + "dir in disk" and mount in / tmp or / var / tmp

When zram (/tmp or /var/tmp) full, the data will be written to the disk.

In zram-init the -M and -2 switches don't work. Assuming push.sh is available the arguments are gathered and processed into positional args but they're never passed to mount or tune2fs because there's a ${1:+"$@"} missing from the mount command lines (all four supported filesystems) and the tune2fs command lines (ext2 and ext4).

hi, is there an ubuntu deb package for this i can install?

the zram-config package in the official repos only provide swap support...

side note: on my ubuntu 18.04 setup the option -azstd is not supported, it is better to just remove it.

X-mount.mkdir is an option for mount where mount creates the target directory if it doesn't already exists (see man mount).

If one passes this option to zram-init it won't even call mount to give it a change to create the target directory, it will fail first:

root@localhost:~ # zram-init -t ext2 -o x-mount.mkdir=775 -d 3 512 /var/tmp/portage
zram-init: 
${dir:-(empty)} is not a directory

If one calls zramctl and mount directly, it works as expected:

root@localhost:~ # zramctl -f -s 512M
/dev/zram3

root@localhost:~ # mkfs.ext2 /dev/zram3 -q

root@localhost:~ # mount /dev/zram3 /var/tmp/portage
mount: /var/tmp/portage: mount point does not exist.

root@localhost:~ # mount /dev/zram3 /var/tmp/portage -o x-mount.mkdir=775

root@localhost:~ # ls -ld /var/tmp/portage
drwxr-xr-x 3 root root 4096 Mar 17 19:41 /var/tmp/portage

Q: What's is the use case for this? Why not simply do an mkdir before running zram-init?

A: On my setup, both /var/tmp and /var/tmp/portage are mounted by zram-init, with different options each. Among the differences there is noexec for /var/tmp (for security reasons) but exec for /var/tmp/portage (required).

Since both mount points are mounted by zram-init triggered by the distribution init scripts (openrc), /var/tmp will necessarily be empty after it's mounted and /var/tmp/portage is then mounted immediately after, leaving no opportunity for me to insert a mkdir command between the two mount operations.

For reference, my actual setup from /etc/conf.d/zram-init:

type2=/var/tmp
flag2=ext4
size2=16384
mlim2=
back2=
opts2="noatime,nosuid,nodev,noexec"
mode2=1777
owgr2=
notr2=
maxs2=1
algo2=zstd
labl2=var_tmp_dir
uuid2=
args2=

type3=/var/tmp/portage
flag3=ext4
size3=16384
mlim3=
back3=
opts3="noatime,nosuid,nodev,x-mount.mkdir=775"
mode3=
owgr3=portage:portage
notr3=
maxs3=1
algo3=zstd
labl3=tmp_portage_dir
uuid3=
args3=

Solution

I did simple check for X-mount.mkdir where zram-init checks if the directory exists, if it's there it won't trigger the error. I'm not sure if it's the best or even appropriate solution but it was what I came up with and it worked for me:

root@localhost:~ # ls -ld /var/tmp/portage
ls: cannot access '/var/tmp/portage': No such file or directory

root@localhost:~ # sh zram-init.sh -t ext2 -o noatime,nosuid,nodev,x-mount.mkdir=775 -c portage:portage -d 3 -D 4 -s 1 -a zstd -L tmp_portage_dir -- 16384 /var/tmp/portage
mke2fs 1.45.5 (07-Jan-2020)

root@localhost:~ # # ls -ld /var/tmp/portage
drwxr-xr-x 3 portage portage 4096 Mar 17 19:47 /var/tmp/portage

I'll be opening a PR next with my patch for discussion.

There's no way to restart the service on kernel 4.8.

# ls -al /dev/zram*
brw-rw---- 1 root disk 253, 0 2016-11-21 17:57:49 /dev/zram0
brw-rw---- 1 root disk 253, 1 2016-11-21 17:57:49 /dev/zram1
brw-rw---- 1 root disk 253, 2 2016-11-21 17:57:49 /dev/zram2

# swapon -s
Filename                                Type            Size    Used    Priority
/dev/zram0                              partition       524284  0       16383
/dev/dm-4                               partition       2093052 41964   -1

When I stop the zram_swap service, I get this in the log:

Nov 21 17:59:56 morfikownia systemd[1]: Stopping swap with zram...
Nov 21 17:59:56 morfikownia kernel: zram0: detected capacity change from 536870912 to 0
Nov 21 17:59:56 morfikownia kernel: zram: Removed device: zram0
Nov 21 17:59:56 morfikownia systemd[1]: Stopped swap with zram.

And as you can see the zram0 device was removed:

 # ls -al /dev/zram*
brw-rw---- 1 root disk 253, 1 2016-11-21 17:57:49 /dev/zram1
brw-rw---- 1 root disk 253, 2 2016-11-21 17:57:49 /dev/zram2

So when I start the service again, I get this:

Nov 21 18:01:04 morfikownia zram-init[91348]: zram-init: cannot find /dev/zram0
Nov 21 18:01:04 morfikownia systemd[1]: zram_swap.service: Main process exited, code=exited, status=1/FAILURE
Nov 21 18:01:04 morfikownia systemd[1]: Failed to start swap with zram.
-- Subject: Unit zram_swap.service has failed

How to restart the service now?

Hi,

I'd like to set one of my two regular swap partitions as backing_dev for my zram swap.

The error in the script is:
/sbin/zram-init: Line 13: printf: Write error: The device or resource is busy

I thought this would mean that the disksize is already set, but if I raise the size so it doesn't fit, the error about that comes after the "busy" warning, so the order is definitely correct.

I tried with turned on and off swap partition. Then I found out, by doing things manually, that the error comes from writing into /sys/block/zram0/backing_dev -- No matter what I try to write in there, it doesn't work.

Currently I am pretty sure that I am doing something wrong. What could that be?

Kernel is gentoo-5.0.5, zstd is enabled and built in.

Hi!
The package is mentioned in the readme is outdated, we should have the zram-init instead.
Best regards

Subj :)
Why do not you make a deb package? Place it on the launchpad!

I just installed zram-init on Gentoo and while everything is working fine (thank you!), I've been puzzled by a strange zsh completion error for the init script that I really don't understand:

$/etc/init.d/zram-init <TAB>
_init_d:130: bad substitution

The _init_d completer is the stock zsh-5.8 script, and the line that complains is the first one in the followign block where it parses the script for commands:

# If we didn't get $cmds from a zstyle, then read init script for opts.
# If script doesn't specify opts, then default to the standard opts.
(( $#cmds )) || cmds=( ${(eQz)${(M)${(f)"$( <$script)"}:#[[:blank:]]#opts=*}#*=} )
(( $#cmds )) || cmds=($opts)

After a lot of trial and error I managed to trace the problem to the ZramIgnore() function and first suspected the use of eval. However as it turns out this is not the case; the only problem is the use of the opts identifier here. Renaming the field to options lets the completer pass without error. Apparently the field name and the completion parser conflict on the identifier name, which is all sorts of terrible for a parser.
Was wondering if you have any suggestions. Maybe it's just a matter of quoting and/or identifier expansion?