lunar-linux / lunar Goto Github PK

Running e.g. 'lvu depends openssl' takes over 10 hours. stumbles even reported over 24 hours. Maybe something like:

for m in $@; do
	find /var/lib/lunar/moonbase -name DEPENDS -exec grep -H "depends.*$m " '{}' \;
done

Output:

[...]  
/var/lib/lunar/moonbase/other/ftp/vsftpd/DEPENDS:optional_depends openssl   "" "" "for SSL support"  
/var/lib/lunar/moonbase/other/ftp/pure-ftpd/DEPENDS:optional_depends openssl   "--with-tls" "" "for SSL/TLS support"  
/var/lib/lunar/moonbase/other/devel/opensc/DEPENDS:optional_depends %OSSL "--enable-openssl" "--disable-openssl" "for openssl support"  
/var/lib/lunar/moonbase/other/libs/PDFlib-Lite/DEPENDS:optional_depends "openssl"   ""                                      "--without-openssl" "for openssl support"  
[...]

This one takes 31 seconds without disk cache (dropped) and 15s with. Of course the output should be worked upon, like:

• strip /var/lib/lunar/moonbase/
• strip configuration flags
• strip the description text
• write (on|off|unset)

bash-5.0# lunar update
Downloading source file moonbase.tar.bz2 for module moonbase
--2020-09-07 17:20:38-- http://lunar-linux.org/lunar//moonbase.tar.bz2
Resolving lunar-linux.org... 188.226.242.178, 2a03:b0c0:0:1010::5d:b001
Connecting to lunar-linux.org|188.226.242.178|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1928640 (1.8M) [application/x-bzip2]
Saving to: '/tmp/moonbase.tar.bz2'

/tmp/moonbase.tar.b 100%[===================>] 1.84M 1.00MB/s in 1.8s

2020-09-07 17:20:40 (1.00 MB/s) - '/tmp/moonbase.tar.bz2' saved [1928640/1928640]

Preparing to install moonbase.tar.bz2 ...
Removed module: moonbase
Extracting moonbase.tar.bz2 ...
Created /var/log/lunar/install/moonbase-20200907.09
Creating /var/log/lunar/md5sum/moonbase-20200907.09
Starting update of installed modules
Usage: gawk [POSIX or GNU style options] -f progfile [--] file ...
Usage: gawk [POSIX or GNU style options] [--] 'program' file ...
POSIX options: GNU long options: (standard)
-f progfile --file=progfile
-F fs --field-separator=fs
-v var=val --assign=var=val
Short options: GNU long options: (extensions)
-b --characters-as-bytes
-c --traditional
-C --copyright
-d[file] --dump-variables[=file]
-D[file] --debug[=file]
-e 'program-text' --source='program-text'
-E file --exec=file
-g --gen-pot
-h --help
-i includefile --include=includefile
-l library --load=library
-L[fatal|invalid|no-ext] --lint[=fatal|invalid|no-ext]
-M --bignum
-N --use-lc-numeric
-n --non-decimal-data
-o[file] --pretty-print[=file]
-O --optimize
-p[file] --profile[=file]
-P --posix
-r --re-interval
-s --no-optimize
-S --sandbox
-t --lint-old
-V --version

To report bugs, see node Bugs' in gawk.info'
which is section `Reporting Problems and Bugs' in the
printed version. This same information may be found at
https://www.gnu.org/software/gawk/manual/html_node/Bugs.html.
PLEASE do NOT try to report bugs by posting in comp.lang.awk,
or by using a web forum such as Stack Overflow.

gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output.

Examples:
gawk '{ sum += $1 }; END { print sum }' file
gawk -F: '{ print $1 }' /etc/passwd
Usage: sed [OPTION]... {script-only-if-no-other-script} [input-file]...

-n, --quiet, --silent
suppress automatic printing of pattern space
--debug
annotate program execution
-e script, --expression=script
add the script to the commands to be executed
-f script-file, --file=script-file
add the contents of script-file to the commands to be executed
--follow-symlinks
follow symlinks when processing in place
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if SUFFIX supplied)
-l N, --line-length=N
specify the desired line-wrap length for the `l' command
--posix
disable all GNU extensions.
-E, -r, --regexp-extended
use extended regular expressions in the script
(for portability use POSIX -E).
-s, --separate
consider files as separate rather than as a single,
continuous long stream.
--sandbox
operate in sandbox mode (disable e/r/w commands).
-u, --unbuffered
load minimal amounts of data from the input files and flush
the output buffers more often
-z, --null-data
separate lines by NUL characters
--help display this help and exit
--version output version information and exit

If no -e, --expression, -f, or --file option is given, then the first
non-option argument is taken as the sed script to interpret. All
remaining arguments are names of input files; if no input files are
specified, then the standard input is read.

GNU sed home page: https://www.gnu.org/software/sed/.
General help using GNU software: https://www.gnu.org/gethelp/.
/var/lib/lunar/functions/check.lunar: line 51: texinfo: command not found
Python 3.8.4 (default, Aug 24 2020, 14:24:14)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

When running 'lunar' choose 'Software Mirrors',the GNOME can not entering,and the message is ‘NVIDIA drivers’ down there.
The XORG menu has the same errors.

My lunar version is 20200208.

Please check it when you have time.Thanks.

Patterns listed in /var/lib/protected seem to overprotect filesystem. I have a module with filenames like "/usr/share/xyz/bin/mvn" and those files are not removed during lrm procudure. Lrm thinks they are protected by "/bin/mv" pattern which doesn't have ^ and $ markers and matches the filename.

Some modules like lightdm, thunar-thumbnailer, etc. have options which
add a dependency on more than one other module.
The only "proper" way we have to handle this at the moment is to add a PROFILE module which depends on all these dependencies and then add an optional_dependency for that PROFILE. For obvious reasons this was never done.

We need a way to specify such batch dependencies in DEPENDS
without hacking around the framework via "(select together with XYZ)" descriptions.

Starting update of installed modules
vte was removed from /var/lib/lunar/moonbase
vte: Do you want to remove vte ? [y]
Removed module: vte
/var/lib/lunar/functions/modules.lunar: line 690: continue: only meaningful in a for', while', or until' loop fftw3 was removed from /var/lib/lunar/moonbase fftw3: Do you want to remove fftw3 ? [y] Removed module: fftw3 /var/lib/lunar/functions/modules.lunar: line 690: continue: only meaningful in a for', while', or until' loop

Some modules (e.g. GConf with ORBit2/dbus) require one
of a number of dependencies to implement some functionality.

This is handled via aliases at the moment but these are global.
It probably makes sense to have something like local aliases
which are only used in this one DEPENDS file.

lunar renew asks for removal of held modules if they disappeared from the moonbase
for some reason. For me it's usually because I switched branches in the repository.

Imho there should be no question in this case (especially not if the default is "yes").
This is evil and automatically removed modules over here at least three times now...

Could we just drop the question in this case or do you see any problems with that?

This has gotten on my nerves dozen of times by now every time I do not manage to update for a while and end up with 30-100 packages to be updated.

Lunar does not preserve libraries, just because they are still used.
If a package is updated, libraries get overwritten and the old ones get removed, leaving downstream dependencies dangling with broken links.

Of course, lunar fix scans installed libraries for unresolved library dependencies and rebuilds affected packages.

Still, I do not really see a problem if we handle this in a more graceful way.

I would propose to

• move SONAMEed libraries to /usr/lib*/OBSOLETE/ and add this folder to /etc/ld.so.conf
• have lunar fix scan for libraries that resolve to this subfolder and treat them the same way we treat Not found
• If libraries in the folder are found not to be in use anymore, lunar fix or lunar prune could clean them.

What do you think @Ratler @florin65 @dagbrown ?

Running on a 500GB NVME drive, every now and then I hit the space limit.
However, the lunar tools do not detect this condition, but simply try to go on.

As a result, I usually end up with a bunch of empty SOURCE_DIRECTORY folders in /usr/src because extracting the actual content into the folder failed, and thus the build fails.

The system should definitely check the remaining storage size before extracting tarballs from the cache or downloading files. A simple check <free space for /var/cache/lunar or /usr/src> >= 1GB should suffice (although, e.g., qt5 unpacks to 2.6GB...) . If the check does not succeed the system should simply print an error string and not try to continue. Bonus points for trying to lunar prune if the relevant /var folders are mounted on the same volume and $AUTOPRUNE is enabled.

Executing AUTOFIX : lunar fix
/usr/lib/libmozjs-68.so of mozjs is broken.
libicui18n.so.66 => not found
libicuuc.so.66 => not found
libicudata.so.66 => not found
/usr/libexec/gsd-wacom of gnome-settings-daemon is broken.
libwacom.so.2 => not found
/usr/bin/gtk-update-icon-cache of gtk+-2 has wrong md5sum.
/usr/bin/chardetect of python-chardet has wrong md5sum.
/usr/bin/inkscape of inkscape is broken.
libpoppler.so.118 => not found
libpoppler.so.118 => not found
/usr/bin/inkview of inkscape is broken.
libpoppler.so.118 => not found
libpoppler.so.118 => not found
/usr/lib64/inkscape/libinkscape_base.so of inkscape is broken.
libpoppler.so.118 => not found
/usr/bin/kalarm of kalarm is broken.
libqgpgme.so.7 => not found
libqgpgme.so.7 => not found
/usr/lib64/libKF5Libkleo.so.5.19.3 of libkleo is broken.
libqgpgme.so.7 => not found
/usr/lib/qt5/plugins/akonadi_serializer_contactgroup.so of akonadi-contacts is broken.
libqgpgme.so.7 => not found
libqgpgme.so.7 => not found
/usr/lib64/libKF5AkonadiContact.so.5.19.3 of akonadi-contacts is broken.
libqgpgme.so.7 => not found
libqgpgme.so.7 => not found
/usr/lib/qt5/plugins/designer/pimcommonakonadiwidgets.so of pimcommon is broken.
libqgpgme.so.7 => not found
libqgpgme.so.7 => not found
/usr/lib64/libKF5PimCommonAkonadi.so.5.19.3 of pimcommon is broken.
libqgpgme.so.7 => not found
libqgpgme.so.7 => not found
/usr/lib/browser-plugins of webkit2gtk3 is missing.
/usr/lib/firefox/browser/plugins of firefox-bin is missing.
/usr/bin/chromedriver of chromium-arch is broken.
/usr/bin/chromedriver: /usr/lib/libstdc++.so.6: version GLIBCXX_3.4.29' not found (required by /usr/bin/chromedriver) /usr/bin/chromedriver: /lib/libc.so.6: version GLIBC_2.32' not found (required by /usr/bin/chromedriver)
/usr/bin/chromedriver: /lib/libc.so.6: version GLIBC_2.33' not found (required by /usr/bin/chromedriver) /usr/bin/chromedriver: /lib/libc.so.6: version GLIBC_2.34' not found (required by /usr/bin/chromedriver)
/usr/bin/chromium of chromium-arch is broken.
/usr/bin/chromium: /lib/libc.so.6: version GLIBC_2.34' not found (required by /usr/bin/chromium) /usr/lib/chromium/chrome-sandbox of chromium-arch is broken. /usr/lib/chromium/chrome-sandbox: /lib/libc.so.6: version GLIBC_2.33' not found (required by /usr/lib/chromium/chrome-sandbox)
/usr/lib/chromium/chrome-sandbox: /lib/libc.so.6: version GLIBC_2.34' not found (required by /usr/lib/chromium/chrome-sandbox) /usr/lib/chromium/chrome_crashpad_handler of chromium-arch is broken. /usr/lib/chromium/chrome_crashpad_handler: /usr/lib/libstdc++.so.6: version GLIBCXX_3.4.29' not found (required by /usr/lib/chromium/chrome_crashpad_handler)
/usr/lib/chromium/chrome_crashpad_handler: /lib/libc.so.6: version GLIBC_2.33' not found (required by /usr/lib/chromium/chrome_crashpad_handler) /usr/lib/chromium/chrome_crashpad_handler: /lib/libc.so.6: version GLIBC_2.34' not found (required by /usr/lib/chromium/chrome_crashpad_handler)
/usr/lib/chromium/chromium of chromium-arch is broken.
/usr/lib/chromium/chromium: /lib/libm.so.6: version GLIBC_2.35' not found (required by /usr/lib/chromium/chromium) /usr/lib/chromium/chromium: /usr/lib/libstdc++.so.6: version GLIBCXX_3.4.29' not found (required by /usr/lib/chromium/chromium)
/usr/lib/chromium/chromium: /usr/lib/libstdc++.so.6: version CXXABI_1.3.13' not found (required by /usr/lib/chromium/chromium) /usr/lib/chromium/chromium: /lib/libc.so.6: version GLIBC_2.32' not found (required by /usr/lib/chromium/chromium)
/usr/lib/chromium/chromium: /lib/libc.so.6: version GLIBC_2.33' not found (required by /usr/lib/chromium/chromium) /usr/lib/chromium/chromium: /lib/libc.so.6: version GLIBC_2.34' not found (required by /usr/lib/chromium/chromium)
libavcodec.so.59 => not found
libavformat.so.59 => not found
libavutil.so.57 => not found
/usr/lib/chromium/libEGL.so of chromium-arch is broken.
/usr/lib/chromium/libEGL.so: /lib/libc.so.6: version GLIBC_2.33' not found (required by /usr/lib/chromium/libEGL.so) /usr/lib/chromium/libEGL.so: /lib/libc.so.6: version GLIBC_2.34' not found (required by /usr/lib/chromium/libEGL.so)
/usr/lib/chromium/libGLESv2.so of chromium-arch is broken.
/usr/lib/chromium/libGLESv2.so: /usr/lib/libstdc++.so.6: version GLIBCXX_3.4.29' not found (required by /usr/lib/chromium/libGLESv2.so) /usr/lib/chromium/libGLESv2.so: /lib/libc.so.6: version GLIBC_2.32' not found (required by /usr/lib/chromium/libGLESv2.so)
/usr/lib/chromium/libGLESv2.so: /lib/libc.so.6: version GLIBC_2.33' not found (required by /usr/lib/chromium/libGLESv2.so) /usr/lib/chromium/libGLESv2.so: /lib/libc.so.6: version GLIBC_2.34' not found (required by /usr/lib/chromium/libGLESv2.so)
/usr/lib/chromium/libvk_swiftshader.so of chromium-arch is broken.
/usr/lib/chromium/libvk_swiftshader.so: /usr/lib/libstdc++.so.6: version GLIBCXX_3.4.29' not found (required by /usr/lib/chromium/libvk_swiftshader.so) /usr/lib/chromium/libvk_swiftshader.so: /lib/libc.so.6: version GLIBC_2.32' not found (required by /usr/lib/chromium/libvk_swiftshader.so)
/usr/lib/chromium/libvk_swiftshader.so: /lib/libc.so.6: version GLIBC_2.33' not found (required by /usr/lib/chromium/libvk_swiftshader.so) /usr/lib/chromium/libvk_swiftshader.so: /lib/libc.so.6: version GLIBC_2.34' not found (required by /usr/lib/chromium/libvk_swiftshader.so)
/usr/lib/chromium/swiftshader/libEGL.so of chromium-arch is broken.
/usr/lib/chromium/swiftshader/libEGL.so: /lib/libc.so.6: version GLIBC_2.34' not found (required by /usr/lib/chromium/swiftshader/libEGL.so) /usr/lib/chromium/swiftshader/libGLESv2.so of chromium-arch is broken. /usr/lib/chromium/swiftshader/libGLESv2.so: /usr/lib/libstdc++.so.6: version GLIBCXX_3.4.29' not found (required by /usr/lib/chromium/swiftshader/libGLESv2.so)
/usr/lib/chromium/swiftshader/libGLESv2.so: /lib/libc.so.6: version GLIBC_2.32' not found (required by /usr/lib/chromium/swiftshader/libGLESv2.so) /usr/lib/chromium/swiftshader/libGLESv2.so: /lib/libc.so.6: version GLIBC_2.33' not found (required by /usr/lib/chromium/swiftshader/libGLESv2.so)
/usr/lib/chromium/swiftshader/libGLESv2.so: /lib/libc.so.6: version `GLIBC_2.34' not found (required by /usr/lib/chromium/swiftshader/libGLESv2.so)
/usr/lib/qt5/plugins/messageviewer/headerstyle/messageviewer_defaultgrantleeheaderstyleplugin.so of messagelib is broken.
libqgpgme.so.7 => not found
libqgpgme.so.7 => not found
/usr/lib64/libKF5MessageComposer.so.5.19.3 of messagelib is broken.
libqgpgme.so.7 => not found
libqgpgme.so.7 => not found
/usr/lib64/libKF5MessageList.so.5.19.3 of messagelib is broken.
libqgpgme.so.7 => not found
libqgpgme.so.7 => not found
/usr/lib64/libKF5MessageViewer.so.5.19.3 of messagelib is broken.
libqgpgme.so.7 => not found
libqgpgme.so.7 => not found
/usr/lib/qt5/plugins/designer/mailcommonwidgets.so of mailcommon is broken.
libqgpgme.so.7 => not found
libqgpgme.so.7 => not found
/usr/lib64/libKF5MailCommon.so.5.19.3 of mailcommon is broken.
libqgpgme.so.7 => not found
libqgpgme.so.7 => not found
/usr/lib/browser-plugins of webkit2gtk3 is missing.
/usr/lib/browser-plugins of webkit2gtk3 is missing.
Checking dependencies for webkit2gtk3
Building webkit2gtk3 version 2.34.6

These tips are for reference only to see if you can find something to explain. After every lunar update, many errors will be found and repaired every time, but they can't be repaired. In particular, webkit2gtk3 will be recompiled every time. The next time it is checked, it will still have problems.

We try to enforce a self-containing moonbase-core.
However, some modules have optional_depends on modules in other sections.
At the moment we just disable those completely, but imho there's no problem with
having them around, if optional_depends checks for the modules existence
and automatically disables the depends in case it's not there.

bash-5.0# lunar update
Downloading source file moonbase.tar.bz2 for module moonbase
--2020-11-06 21:35:01-- http://lunar-linux.org/lunar//moonbase.tar.bz2
Resolving lunar-linux.org (lunar-linux.org)... 2a03:b0c0:0:1010::5d:b001, 188.226.242.178
Connecting to lunar-linux.org (lunar-linux.org)|2a03:b0c0:0:1010::5d:b001|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1945933 (1.9M) [application/x-bzip2]
Saving to: '/tmp/moonbase.tar.bz2'

/tmp/moonbase.tar.b 100%[===================>] 1.86M 4.61KB/s in 6m 42s

2020-11-06 21:41:44 (4.73 KB/s) - '/tmp/moonbase.tar.bz2' saved [1945933/1945933]

Preparing to install moonbase.tar.bz2 ...
Removed module: moonbase
Extracting moonbase.tar.bz2 ...
Created /var/log/lunar/install/moonbase-20201106.13
Creating /var/log/lunar/md5sum/moonbase-20201106.13
Starting update of installed modules
/var/lib/lunar/functions/modules.lunar: line 714: ((: > 20201024 : syntax error: operand expected (error token is "> 20201024 ")

More and more modules are hosted and downloaded from github (sad enough).
This results in many tarballs in $SOURCE_CACHE with the format $VERSION.tar.gz
and this, for sure, is prone to collisions.

There are a couple of possibilities to solve this ranging from a generic
SOURCE_CACHE=/var/spool/lunar/$MODULE to a flag which automatically renames
such downloads or an additional $STORE_SOURCE variable for the name of the stored file.

Any ideas? comments?

Okay, as we have discussed in #lunar , the same way lunar exile <#> exists (that always answers 'no' for module #), there should be a lunar enforce <#> that always answer 'yes' for module #.

A bunch of modules have the traditional chicken-and-egg problem, where modules depend on each other, e.g., libvnd and mesa-lib.

While the general problem cannot be solved without providing binaries for either package, we could handle cycles with optional_depends automatically.

Once such a cycle is detected (this doesn't happen at the moment),
the system could automatically disable a specific optional dependency to break the cycle, compile the packages and in the end compile the first package again, enabling the dependency.

To designate a specific dependency for this process, I'd propose to use a new breakable_optional_depends function.

Still, implementing this is tricky, especially because we have to consider the case where any package-build in between fails.

lunar:Version number v43 20220705

lunar---Option Menu---Integrity Checking
lunar---Option Menu---Software Mirrors
lunar---Option Menu---Accepted licenses
lunar---Option Menu---Rejected licenses

This four options cannot enter to the next menu.

Modules with many optional dependencies, like mpv, ask users many questions consecutively and it feels like a marathon to answer them all.
Even worse, if you accidentally make any mistake, the best thing you can do is CTRL+c, run lin -r module, go through the whole sprint again and hope to get it right this time.

To reduce these issues, it would be nice to have a CLI menu (likely using dialog) to select choices for a package, also remembering previous choices the user made.

An obvious short-coming of this idea is that the recursive nature of the dependency selection process is not available in the proposed menu. Instead of a nice-looking list of questions, this becomes a bombardment of dialogs.
But leaving it to the user which one they prefer should takes care of that.

As the description says, lin -R foo will happily remove an installed version of foo, even if there is no tarball in /var/cache/lunar to resurrect.
This is particularly troublesome in combination with lin -w <different-version> foo, when you try to quickly switch between versions.

The scripts should check for existence of the required tarball before removing running lrm for the installed version.
Something else (e.g., broken tarball) might still break later on, but at least this obvious case should work.

When installing a module with any number of dependencies, lin module installs only the first dependency and then returns to the command line.

To reproduce:

# lin darktable

It installed the first dependency that isn't already installed--in my case, exiv2--and then lin just exited and dropped me back to the command-line prompt.

The problem was first reported by tbeckerson and confirmed by me.

some modules require gfortran which is an optional part of gcc,
others require cairo to have support for some specific backend,
and so on...

We need a DEPENDS mechanism to tell the user the module depends on that option of another modules and automatically (upon confirmation by the user) recompile the problematic module and enforce the specific option/optional_depends.

Would be great to see that in lunar 12 :o)

Do you have a policy for adding mirrors to the existing list of mirrors in lunar? I mean, I assume mirrors would like to have there permission to add them to lunar's list of mirrors?
I noticed broken mirrors. And maybe I can help with that?

have a great day!

We should have a way to notify the system if the name of a module changed.

I would propose to have a simple file RENAMES in the moonbase(s) that consists of lines like

tbb -> intel-tbb

At the moment the update system notices that a module is gone and encourages the user to remove it from the system. Instead it can go through the file and propose to remove the old and install the new module.

If the file ever gets too big, we could just drop old entries.

This shouldn't be a hassle to implement and would remove some frustration with looking up new package names.

Some configure options / dependencies only become available
in case the user selected some other option before (e.g. Xpm image support
in nethack doesn't work without libX11 support).

This does not necessarily require a change in our codebase, but we need a unified way
to handle these cases and "(select only when also XYZ)" in the description is not adequate at all.

Is something like the following supported at the moment for DEPENDS/CONFIGURE and do we want this?
@sofar mentioned that there should be no logic in CONFIGURE/DEPENDS...

if selected_dependency ABC; then
   optional_dependency EFG "--enable-EFG" "--disable-EFG" "EFG support"
fi

At the moment lget will not complain if the downloaded files are corrupt.
I think it would be enough to call verify_all_sources after successful downloading to fix this issue

Whenever an upstream maintainer decides to change the released tarball of a module and the user updated the module before, they will eventually end up with a "download skipped. File already present. Checksum failed." error during the next lin.

I'd like to propose to automatically try to download the module once in this situation.
Otherwise, if this happens in the middle of lunar renew, the module will not be updated but the update process just continues with the next module.

Comments?