mm1ke / gentoo-scripts Goto Github PK

Tests should be run with esetup.py, but some ebuilds like
sci-geosciences/tilecache
try to run setup.py directly. This will lead to problems, if more than one python version is possible in the ebuild. In general there is no reason to use it and should be substituted by esetup.py

https://devmanual.gentoo.org/eclass-reference/distutils-r1.eclass/

If you want to try to convert some tools to python some day,
you can find here some interesting example code
https://github.com/mgorny/proxy-maint-bugs
It is very clean and can evaluate special cases in the packages better then a grep
This was what I advertised on IRC recently.
It was 100 times faster than my bash solution.
Best, JS

obsolete:
inherit fdo-mime
this should be replaced by xdg-utils and its functions.
https://devmanual.gentoo.org/eclass-reference/xdg-utils.eclass/

repoman can detect this too and claims
app-office/texstudio/texstudio-2.11.2-r1.ebuild: please migrate from 'fdo-mime' to 'xdg-utils' on line: 6

qgrep -Ne "inherit.* fdo-mime"

can list ebuilds where inherit has no linebreak and need some care. Perhaps that is enough for a start...

Gentoo has also a list, but no sorting by maintainer...
https://qa-reports.gentoo.org/output/eclass-usage/fdo-mime.txt

http://gentoo.levelnine.at/wwwtest/sort-by-maintainer/000-DATA-USAGE.txt
should be | instead of ;

I think this is a false positive:

http://gentoo.levelnine.at/srctest/sort-by-maintainer/tex_at_g.o.txt
dev-tex/latex2html|http://mirrors.ctan.org/support/latex2html/latex2html-2017.2.tar.gz|[email protected]:

For example
http://gentoo.levelnine.at/simplechecks/gentoo_mirror_missuse/sort-by-maintainer/
has a format like
maintainer-needed_at_g.o.txt 13-Sep-2017 01:20 6373

I suggest
maintainer-needed_at_g.o.txt 13-Sep-2017 01:20 23 ebuilds

A current list of obsolete links to HTTP, SRC_URI of obsolete repositories would fit well in to this suit.

https://wiki.gentoo.org/wiki/Upstream_repository_shutdowns

many packages which are in /usr/portage/profiles/package.mask have several bugs and flood the lists.
But they are masked for removal already and should not be printed.

cat /usr/portage/profiles/package.mask | grep -v "^#"
perhaps it is the easiest to remove them in the end, or remove from the todo-list?
Is this what the whitelist is good for?

The dataformat is now
sys-apps;fwts;fwts-17.02.00;403;https://wiki.ubuntu.com/Kernel/Reference/fwts;[email protected]:
I suggest to change it to

403;sys-apps/fwts;fwts-17.02.00;https://wiki.ubuntu.com/Kernel/Reference/fwts;[email protected]:
because the first key is better to sort and search for.
The category/package should be written with / sys-apps/fwts

ps: I know that already several packages were fixed on basis of your list.

I suggest to add a file with more information on the problem.
I am not sure, if it is better to put/link one file in each folder, or maintain one central file.
It could be also a page in the personal user space of wiki.gentoo.org
or markdown on github?
I think it is important to add links there and that many users can contribute to the explanations.

The content should be a short explanation for the developer
Why, is this a problem? Where is it documented? How to fix it?

I think this would motivate more people to fix things...

000-README.txt or
000-README.md or
000-EXPLANATION.txt

https is more secure, see https://gist.github.com/grawity/4392747

Just saw dev-vcs/stgit had
[email protected], but no proxied maintainer in metadata.
Perhaps there are more packages to fix.
The trick is to filter the upstream contacts. I think this works best with the python library.

There are two interesting cases to filter:

1. Proxy-maintained packages without [email protected] https://bugs.gentoo.org/634270

2. Proxy-maintained packages without a proxy maintainer (no @gentoo.org) https://bugs.gentoo.org/633314

https://gentoo.levelnine.at/wwwtest/sort-by-package/app-office/lyx.txt

qgrep -N "+qt4" | grep qt5 shows some matches
see for example https://bugs.gentoo.org/641516

Who knows more examples for obsolete USE flags with a successor?

Taking this page as an example, I'd like to have an alt version which sorts the list by QTY instead
of Maintainer. This is quasi-related to my previous feature request, as it would allow the 'juiciest'
targets to float to the top for folks looking for QA issues that can be fixed in bulk.

I suggest to add a link go to home page on each page.
Example:

List generated on 2018-02-24
Total Maintainers: 207
go to home page

This helps people to find out more statistics, if they got only a link to a deep page.

https://devmanual.gentoo.org/general-concepts/mirrors/

Previous policy was to use mirror://gentoo directly, but this is now deprecated, as that wouldn't allow to have long-term availability and traceability of the source files, which might be a requirement of the license.

Packages with this problem can be listed by
qgrep -N 'SRC_URI="mirror://gentoo'

Especially for the sorted-by-maintainer folder it would be useful to list these too

The actual hierarchy is quite chaotic. This should be improved.
Ideas are welcomed.

Could you add to a README: How long does one run take each (just a rough number...)?
like ~3 h on 1 thread i7-3770, ping 20ms

we were inventing new names, but could use the same as used by repoman.
They are listed in man repoman
or here:
https://gitweb.gentoo.org/proj/portage.git/tree/repoman/man/repoman.1?id=refs/heads/repoman

/usr/portage$ grep "EAPI" `qgrep -l einstall`

reveals that we have many einstall in EAPI=6 ebuilds

However einstall was banned in EAPI 6

https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-13100011.3.3.2
It would be nice to scan for these too.

looks to me like a false positive, did I miss something?
http://gentoo.levelnine.at/wwwtest/sort-by-package/app-editors/nedit.txt

We have a problem on Gentoo, that there is no general dist file directory. This leads to workarounds like seen in https://bugs.gentoo.org/642102
I have no good idea, but it should be documented and kept in mind.
Until we have a solution, I suggest to white list the package.

#I suggest to test, if RESTRICT mirror is set for any in the list with broken SRC_URI
These packages are unable to be installed and should be fixed soon, or last rited.
Important would be to know if the package is already masked.

Very often || die is missing behind echo to files or sed

For example:
echo '#!/bin/true' >share/genbuild.sh
should be
echo '#!/bin/true' >share/genbuild.sh || die

I think this will be part of a group of tests, which will never be 100% accurate, but remembers the maintainer to double check the line manually.

The following line lists all ebuilds with no die behind echo, while it is likely written to files there
qgrep -Nx "echo .* > " | grep -v die

Should we have a category suspicious for tests which will likely contain many false positives but are worth a verification?

We need the test for

• echo
• sed
• rm
• cp
• mv

If SRC_URI downloads a .zip or similar, we need a dependency on app-arch/unzip in all ebuilds EAPI=1-6. I do not know about 7.

There may be a few strange false positives. For example, if it is a zipped plugin and the program can unzip itself. But this will be very seldom.

https://dev.gentoo.org/~ulm/pms/7-draft/eapi-cheatsheet-nocombine.pdf
in EAPI=7

• dolib and libopts are not allowed, dolib.a or dolib.so commands should be used
  as replacement.

In most cases, we expect that HOMEPAGE is the same for all ebuilds in one package.
Often this is not the case, because when updating the HOMEPAGE when the project moved it was forgotten to update the old ebuilds in the tree too.

Updating HOMEPAGE does not even require a revision bump and could be fixed easily, if at least one ebuild has a working value. Perhaps it is best combined with the HTTP status codes (https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) which are available anyway:

404|foo/bar-1.0|http://fooo/bar
200|foo/bar-9999|https://foo.codehub.example.com/
000|foo/bar-9999|http://dyn.offlinefooo/bar

Many packages use icons, but do not call gnome2_icon_cache_update()

I do not know which script does find this at the moment.
Some ideas:
one could grep for icon (does not work well picard does not use "icon" for example), or look up which files are installed on http://portagefilelist.de/
http://files.portagefilelist.de/sqldump/

this database can be a good tool for many tests.

there are also ehelper functions doicon and newicon, but some packages copy the files in a creative manner.
Probably the best to start with a non perfect solution and improve later.

This is how the problem is displayed after installation

>>> Installing (11 of 11) media-sound/picard-1.4.2::gentoo
 * QA Notice: new icons were found installed but GTK+ icon cache
 * has not been updated:
 *   /usr/share/icons/hicolor/16x16/apps/picard.png
 *   /usr/share/icons/hicolor/256x256/apps/picard.png
 *   /usr/share/icons/hicolor/32x32/apps/picard.png
 *   /usr/share/icons/hicolor/128x128/apps/picard.png
 *   /usr/share/icons/hicolor/scalable/apps/picard.svg
 *   /usr/share/icons/hicolor/48x48/apps/picard.png
 *   /usr/share/icons/hicolor/24x24/apps/picard.png
 * Please make sure to call gnome2_icon_cache_update()
 * in pkg_postinst() and pkg_postrm() phases of appropriate pkgs.
 * QA Notice: .desktop files with MimeType= were found installed
 * but desktop mimeinfo cache has not been updated:
 *   /usr/share/applications/picard.desktop
 * Please make sure to call xdg_desktop_database_update()
 * in pkg_postinst() and pkg_postrm() phases of appropriate pkgs.

A table, which folders should trigger which function is now on this page: https://wiki.gentoo.org/wiki/Notes_on_ebuilds_with_GUI

Perhaps it is a good idea to add some of the checks to portage:
https://github.com/gentoo/portage/tree/master/bin/postinst-qa-check.d

I put this in the ticket for later, so that the idea does not get lost.

I would like the above mentioned check to have suboptions to say list only EAPI=0 ebuilds, for
EAPI=[01234].

Just found a situation where the nice ; delimiter fails:

http://gentoo.levelnine.at/wwwtest/sort-by-maintainer/a%3Dsummary.txt
301;sys-boot/silo;silo-1.4.14_p20120819;https://git.kernel.org/?p=linux/kernel/git/davem/silo.git;a=summary;[email protected]:

I am not sure which delimiter would be better.

I would appreciate adding the abendbrot overlay to the listing. I use it myself, and I can
see quite a lot of QA issues even at a glance.

Basically, I'd like the website to feature a listing which lists ebuilds/etc which feature <= N amount
of QA issues across the board. In this way, one can easily scope out which ebuilds can be fixed
to reduce the amount of QA issues the most, and provide a sort of focal point.

I'm unsure which number N should be, or if we even have any ebuilds which feature 3 or more QA
issues, but its an interesting thought.

qgrep -NL LICENSE

I fear there could be some false positives, because there could be for example an eclass introducing a license technically.

/usr/portage/eclass$ grep LICENSE *
reveals for example the freebsd eclass.

An output folder could be for example
/LICENSE_missing/sort-by-maintainer/...

Check for srcfile's where the site certification is broken. Maybe also check HOMEPAGES?
Note: --no-check-certificate

Just to not forget about that..

check for deprecated ltprune.eclass / prune_libtool_files

Live ebuilds are by definition in constant flux, as you'll get a different result each time you build
them for the most part. By adding them to your package.accept_keywords you effectively opt in
to this inherent instability. As such, they are the perfect place to bump EAPI in-place, without
revbumps or the like.

I'd like a listing of all live ebuilds that are not EAPI 6 for now, and 7 once that's finalized and support
is added to portage and eclasses, so we can advance the usage.

Sometimes there are files missing on http://gentoo.levelnine.at/*
I think this happens for a few hours, when the job is still running.
I suggest to complete the job first and then rsync or copy or rename (?) the new files over the old ones.

When I run these scripts against an overlay I get many of these errors:

grep: /home/simon/src/gentoo-audio/audio-overlay/metadata/md5-cache/app-emulation/wineasio-9999: No such file or directory

I don't think these files would/should exist in an overlay. Are they necessary for the checks?

qgrep -N 'DESCRIPTION=" '
qgrep -N 'DESCRIPTION=' | grep ' "$'

lists ebuilds with leading and trailing space, which makes no sense

Some packages are implicit system dependencies
emerge --emptytree --pretend system
https://devmanual.gentoo.org/general-concepts/dependencies/index.html#implicit-system-dependency

emerge --pretend --nodeps @system
also
eix --system
can approximate it

ebuilds should not depend on any implicit dependency.

The GPL-2 license file is missing, a note in the README would be great too.

net-libs/polarssl|polarssl-1.3.9|http://polarssl.org/download/polarssl-1.3.9-gpl.tgz|[email protected]:
is a false positive. The website redirects to https and uses a bad cert. This could be the reason...
I think we should accept bad certs for now.

I like that the code is so clear and straight forward.
I guess, you could speed it up a lot with
sys-process/parallel

But I have no idea, how to keep the code still so clear.
Just wanted to let you know...

This is not a technical bug, but about very bad style.
https://devmanual.gentoo.org/general-concepts/dependencies/

The test should detect that dev-libs/openssl and dev-libs/libressl are on the same line in

RDEPEND="
	!libressl? ( dev-libs/openssl:0[-bindist] ) libressl? ( dev-libs/libressl )
	>=dev-libs/libsecp256k1-0.0.0_pre20151118[recovery]
"

I just saw, that some packages install into strange folders (for example https://bugs.gentoo.org/635032)

One can get an overview with

qgrep exeinto | grep -v "\\$" | sort | uniq # raw list
qgrep exeinto | sed -e "s/  //g" | grep -v "\\$" | sort | uniq -c | grep -v "usr/bin" # filtered, with histogram

Gentoo packages should follow https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard if possible.

I have no idea yet, how we can filter the "bad" folders and we will have to look at them in detail.
We should have an eye on

exeinto /var*
exeinto /etc/udev*
exeinto /
exeinto /usr/share/   << some may be fine here
exeinto /opt/bin (1)

1) The directories /opt/bin, /opt/doc, /opt/include, /opt/info, /opt/lib, and /opt/man are reserved for local system administrator use. Packages may provide "front-end" files intended to be placed in (by linking or copying) these reserved directories by the local system administrator, but must function normally in the absence of these reserved directories. http://www.pathname.com/fhs/pub/fhs-2.3.html#OPTADDONAPPLICATIONSOFTWAREPACKAGES

if developer bob wants to know if any of his packages has a problem, he hast to walk through many subfolders.
It would be nice to have a folder, where all problems are sorted by maintainer.

I have no good idea for the representation of this data
perhaps it is the fastest, to prepare something like

http://gentoo.levelnine.at/all_by_maintainer/bob_at_g_o/srctest.txt
http://gentoo.levelnine.at/all_by_maintainer/bob_at_g_o/patchtest.txt

Global USE flags do not need to be redefined locally.

global
hddtemp: Enable monitoring of hdd temperature (app-admin/hddtemp)

exists. But in the metadata of gkrellm is it defined again.

$ equery uses gkrellm
[ Legend : U - final flag setting for installation]
[        : I - package is installed with flag     ]
[ Colors : set, unset                             ]
 * Found these USE flags for app-admin/gkrellm-2.3.10-r1:
 U I
 + + X          : Build both the X11 gui (gkrellm) and the server (gkrellmd). Disabling this flag builds the server only
 - - gnutls     : Enable SSL support for mail checking with net-libs/gnutls (overrides 'ssl' USE flag)
 - - hddtemp    : Enable monitoring harddrive temperatures via app-admin/hddtemp
 - - libressl   : Use dev-libs/libressl as SSL provider (might need ssl USE flag), packages should not depend on this USE flag
 - - lm_sensors : Enable monitoring sensors via sys-apps/lm_sensors
 + + nls        : Add Native Language Support (using gettext - GNU locale utilities)
 - - ntlm       : Enable NTLM authentication for mail checking with net-libs/libntlm
 + + ssl        : Enable SSL support for mail checking with dev-libs/openssl

see also https://wiki.gentoo.org/wiki/Handbook:AMD64/Working/USE about USE flags.

Thank you very much for the very useful script.
It would be nice to have a user "[email protected]" for packages without maintainer (this means there is literally no maintainer in the metadata.xml).
As far I could see, they are not listed at the moment.