x3n0m0rph59 / precached

Precached - A Linux process monitor and pre-caching daemon

Hi there - Welcome to my profile 👋

---

My current projects and areas of expertise:

• Eruption - Realtime RGB LED Driver for Linux: eruption-project.org
• Eruption Profile Switcher: GNOME Shell Extension
• I love to code in Rust 🦀 ✌️

precached's Issues

blacklist is not honored

type=AVC msg=audit(1563477342.561:662): avc: denied { read open } for pid=5230 comm="precached/fanot" path="/etc/shadow" dev="mmcblk0p4" ino=139571 scontext=system_u:system_r:precached_t:s0 tcontext=system_u:object_r:shadow_t:s0 tclass=file permissive=1

precached does not honor the blacklist. /etc/shadow is on my blacklist in the precached config
"/etc/shadow",

My current config is viewable for you in the pull request #8

hot-applications list displays same item multiple times

I saw that I have the same item multiple times in the hot-applications list.
Is this expected?

Abgeschlossen                                                                   +-----+-----------------------------------------------------------+----------------------+--------+
|   # | Ausführbare Datei                                         | Hash                 | Anzahl |
+-----+-----------------------------------------------------------+----------------------+--------+
|   1 | /usr/bin/ionice                                           | 6745466777889424043  |   1312 |
+-----+-----------------------------------------------------------+----------------------+--------+
|   2 | /usr/bin/ionice                                           | 9253562639802036953  |    815 |
+-----+-----------------------------------------------------------+----------------------+--------+
|   3 | /usr/bin/ionice                                           | 11100230901800630663 |    799 |
+-----+-----------------------------------------------------------+----------------------+--------+
|   4 | /usr/bin/ionice                                           | 12164354706907101889 |    789 |
+-----+-----------------------------------------------------------+-------------

precachedctl_plugins_hot-applications_list.log.zip

Does not work on linux systems with kernel lockdown

I installed precached via copr on fedora 30 but it does not work.

I used the following commands to activate it:

sudo systemctl enable --now precached.service
sudo systemctl enable --now precached-prime-caches.timer
systemctl --user enable --now precached-trigger.service

By does not work I mean:
precached service fails because of to many restart attemps
precached-prime-caches.timer failes because of missing dependencies

I need to deactivate all services to be able to login because boot never finishes.

flatpak

Well I did not view all of my 1600 precached files but it seems precached does not preload files of flatpaks.

Maybe that is because of the layering of the filesystem. I do not mind that just something what seems to happen.

I expect that snaps may have the same effect.

decrease cpu usage over time

By looking at the CPU time metrics of gnome system monitor I found out that precached used 42 hours worth of CPU time. Compared to 45 minutes for journald and 12 hours for gnome. Firefox had 14 hours of CPU time. Ananicy (rule based nice daemon) had 2 hours.

The above findings were measured in the same time frame. Every program was running for 2.5 hours (uptime of the system).

I am thinking about battery based systems. After the initial preload during power up, use a bigger sleep timer. Maybe it is possible to drop the CPU time of precached by 3 or 4.
Maybe use an algorithm to defer or stop the creation of new profiles while the system is running without power cable attached.

unclean build

/usr/lib/.build-id
/usr/lib/.build-id/1d
/usr/lib/.build-id/1d/30cbd674b95e259c8d9b76db9dc79af4b0fbdd
/usr/lib/.build-id/1f
/usr/lib/.build-id/1f/a0bd8e0017a55ea63e6ae8f823e0c57f5c626a
/usr/lib/.build-id/91
/usr/lib/.build-id/91/69fdc1e2254a45a7e09724a7bae1de9ed22f61
/usr/lib/.build-id/98
/usr/lib/.build-id/98/d7cf92b71004aa930f330a2b5d0f5e04a09f50
/usr/lib/.build-id/ca
/usr/lib/.build-id/ca/27b44839c66d4dae6d388a44b69af09cb6b45e
/usr/lib/.build-id/d9
/usr/lib/.build-id/d9/9521f44def8e090d18aa2b5dd481dccc3184d8
/usr/lib/.build-id/fb
/usr/lib/.build-id/fb/7b86b583aaad1c6d51e65340ace8da206e2370

These folders are part of precached rpm. I think they are from the build process.

Dependencies tree

Does precached generate a list of the most important files?
Lets say 10 often used programs use file a 9 times, file b 7 times, file c is only used by 2, file d is only used with/after file c....?

feedback on version 1.7.1

First tests show a major improvement.
From preloading after startup to unloading at high memory limit.
Re preloading after memory hung process closes works too.
CPU usage over time is currently at 1 hour and 30 minutes against gnome with 24 hours with a system uptime of 4 hours. I started in this time frame LibreOffice, Firefox, Gnome Builder, etc...

Unloading of precached files while I started the 2GB ram VM went well too. Precached just kept the lower memory limit. (And re preloaded after I closed the VM) At this point maybe it is preferable to unload everything from precached instead of keeping lower memory limit. I am thinking of a "huge memory hungry process" (process takes 50%+ of available resources) mode. Like while a user is running a full screen game. This is not normal computer usage like while browsing the web or editing a word document.

I do not know if this is detectable by precached. On the other side I can add that the swapping was minimal and my memory usage was around 95% instead of 99% like before.

run as non root

So what are the possibilities of precached running as non root user?

Because precached just needs read access to /.
Maybe use a precached user instead.

This would completely drop the possibility of precached overwriting files.
Precached working directories can just be of the same user/group as precached is.

SUID blacklist

I think it is a good idea to blacklist all suid binaries.
I will create a list on my system and do a pull request. Not perfect but a start.

I do not thing that precached checking for suid flag is useful. Overhead for just a few files.
But a blacklist for suid files of ubuntu/fedora default install should be enough.

create directories at install

/var/run/precached
/var/log/precached

I need the above directories created at install by the precached pkg.
Not during runtime.

Reason is that the selinux confinement is set to restore the right context for these directories. But they are missing in the default install. /var/run/precached is only created at runtime.

selinux confinement for precached

This is the selinux development thread.
Current test version of the confinement is in post number 2.

State: non-productive test systems. Many AVC notices. Blacklist needs improvements
The confinement is fully functional at base level. A full test of all precached features is needed.

Every precached binary and folder is part of the confinement. port 8023 is preset as usable for the not currently per default running web server.

Precached is able to load/unload files into memory while in permissive mode.

Todo

• uncertain
• asks for to much
• Blacklist
• missing binaries
• optional policy

Precached Selinux Guide

change working directory to pkg_precached_selinux

Change on version update

precached.te > policy_module(precached, 1.1.1)
precached_selinux.spec > %define selinux_policyver 1.1.1
precached_selinux.spec > Version:        1.1

WARNING deletes and generate new policy, only use for new barebone policy

$ DIDYOUREADME sudo sepolicy generate --init /usr/sbin/precached

Check syntax, test and build policy

$ sudo ./precached.sh

Update precached.pp policy

$ sudo make -f /usr/share/selinux/devel/Makefile precached.pp

Install precached.pp policy (optional)

$ sudo /usr/sbin/semodule -i precached.pp

Check syntax, test and build policy (optional)

$ sudo ./precached.sh

Install policy

$ sudo dnf install ./noarch/precached_selinux-1.1-1.fc30.noarch.rpm

Check for re confinement

$ ls -Zd /var/lib/precached/ && ls -Zd /usr/sbin/iotracectl

Restart precached service

$ sudo systemctl restart precached

Check for avc notices

(cherry pick needed allow rules from the next 2 commands into precached.te per hand)

$ sudo ausearch -c 'precached' --raw | audit2allow -M my-precached
$ sudo ausearch -c 'precached/fanot' --raw | audit2allow -M my-precachedfanot

Check syntax, test and build policy

$ sudo ./precached.sh --update

Switch to enforcing after the policy reached a stable level

remove "permissive precached_t;" from precached.te

To avoid issues the domain can be set to permissive at first use

$ sudo semanage permissive -a precached_t

is precached supposed to preload iotrace files?

/var/lib/precached/iotrace/...

I saw that precached cached files from the above location.
Is this okay/expected?

high cpu usage

at least while precachedtop is running the cpu usage is high

Fix Compiler Warnings

Fix compiler warnings that occur when building precached.

$ cargo clean
$ cargo build

This task is especially well suited for newbies!

Do files in cache become to old?

Imagine a file cached via precached changes on the disk after precached loaded it into mem.
Now this file is accessed. Is the cached version preferred or is the disk version loaded?

I am asking because I updated a gnome shell extension by replacing the files. Then I used gnome-tweak to unload and reload the extension. But the old version of the extension was loaded. I do not know if precached had something to do with it or just a coincidence.

allow mouse interaction inside precachedtop

I did like to be able to copy the file path+names which precachedtop displays with my mouse cursor but I can not mark them.

If possible please add this feature. I want to be able to copy them to include them in the config file under excluded items.

high cpu usage 2

I found that precached uses 9-20% cpu atm.
The cpu usage was not as high before.

What command can I run to find out what precached is calculating?
I stopped the process atm but I want to investigate it.

slow shutdown

After I click shutdown/restart and the output appears on the screen displaying the current service states during poweroff the system waits 1.30 minutes at/after precached monitor.

I wrote at/after because the service "precached monitor" exits quickly but the process after that "user shell 3" needs the whole 1.30 minutes to exit.

I will replace the service names above with the right once after my system finished calculating. I am not sure what the exact name was. But I suspect it has something to do with precached. Will restart. in a minute to look for the service names :)