doccaz / scc-tools Goto Github PK
View Code? Open in Web Editor NEWA set of simple tools to interact with SUSE Customer Center (SCC)
License: MIT License
A set of simple tools to interact with SUSE Customer Center (SCC)
License: MIT License
Add the product ID codes for ppc64le, s390x and aarch64 to the known product list.
SUSE PTFs (Program Temporary Fix) packages have been standardized to have a specific "vendor" tag. They've been erroneously categorized as "unsupported" and went to the wrong list.
I'm implementing an extra report/CSV file where they can live, called "suseptf".
$ ./vercheck-cache.py
Traceback (most recent call last):
File "./vercheck-cache.py", line 6, in
from platformdirs import user_cache_dir
ModuleNotFoundError: No module named 'platformdirs'
I consider this more an enhancement than a bug.
Can you either
Some packages that vercheck claims to be "unsupported" ;-)
sle-manager-tools-release | SUSE Linux Enterprise 12 | 12-10.3.1
sle-manager-tools-release-POOL | SUSE Linux Enterprise 12 | 12-1.20
python2-rhnlib | SUSE Linux Enterprise 12 | 4.3.4-21.43.3
spacewalk-usix | SUSE Linux Enterprise 12 | 4.0.9-3.6.7
supportutils-plugin-salt | SUSE Linux Enterprise 12 | 1.2.0-6.16.1
sle-manager-tools-release | SUSE Linux Enterprise 15 | 15-113.8.1
Identify the architecture used in a supportconfig and use it for the searches.
Instead of printing out matching types of packages as they're found, output a more easily-readable report.
In some supportconfigs hardware.txt
does not exist. Suggest adding a fallback file (basic-environment.txt
) to check, or adding a flag to manually specify arch when running with -d
against a supportconfig.
Since we already download and manage local cached copies of the PINT databases, we should implement parameters/commands to query it on the CLI. It should be a quicker alternative to checking the PINT website.
Analyzing scc_ccd03b001421_210922_1732, vercheck shows an older version of dbus1 than they have installed. Also looks like its displaying the shipping version instead of updates channel:
dbus-1,1.12.2-8.6.1,1.12.2-8.3.1,Basesystem Module 15 SP1 x86_64
I assume the versions reported by repositories that match the "base_regex" have more priority over the other repositories. However, it seems I forgot to check if the version is actually higher than the original version and discard it if otherwise.
After the 100% CPU usage problems, I'll port the code to use concurrent.futures.ThreadPoolExecutor.
References:
https://docs.python.org/3/library/concurrent.futures.html
https://stackoverflow.com/questions/6509261/how-to-use-concurrent-futures-with-timeouts
There is a TID (https://www.suse.com/support/kb/doc/?id=000019341) that describes os-release inconsistencies that were found on SLES for SAP. Basically, different CPE string schemes were used on each release.
Coincidentally, my table of product IDs in the code matches all the flavors up until SLES for SAP 12 SP4, but according to the TID, it can still be misleading and we should not rely on just the os-release information, but also check the installed base products:
Even when the base product is SLES for SAP, the contents of the os-release file should never have been used to identify this fact.
SLES for SAP 12 (no service pack): when registered and patched during the build process, had the correct settings in os-release (all indicating SLES, not 'SLES for SAP'). Also incorrectly has two additional product files in /etc/products.d/
SLES for SAP 12 (no service pack): when not registered and patched during build, had the wrong settings in os-release for NAME, VERSION_ID, PRETTY_NAME, ID and CPR_NAME
SLES for SAP 12.1: the NAME, PRETTY_NAME, ID and CPE_NAME fields were all changed (incorrectly) from a 'SLES' identifier to a 'SLES for SAP' identifier. The contents of the products.d directory correctly reflect the only two entries that should be present.
SLES for SAP 12.2: os-release contents followed the same pattern as SLES for SAP 12.1
SLES for SAP 12.3: os-release contents changed the NAME, PRETTY_NAME and ID fields 'back' to SLES only identifiers. CPE_NAME was left stating 'SLES for SAP' (this was an over-sight).
SLES for SAP 12.4: os-release contents followed the same pattern as SLES for SAP 12.3
SLES for SAP 15: os-release contents match the original correct syntax from SLES 12 (no service pack) with CPE_NAME also correctly reflecting SLES. The contents of the products.d directory has changed with this release to reflect the new packaging/module schema.
And indeed, my CPE strings for SLES for SAP 15 and up are incorrect. I'll need to implement an extra check for the base products (present in updates.txt).
It's been reported that lately vercheck.py is taking around 10-12 minutes to complete, and is stuck in the "gathering results..." message for quite a while.
I've noticed that something changed in the SCC infrastructure that's leaving at least "max_threads" connections in the CLOSE_WAIT state until the kernel's tcp_fin_timeout runs out for each one. The default for recent kernels is 60 seconds.
Do not emit the package versions while analyzing a supportconfig by default. Instead, create flags to show specific messages.
e.g. --show-unknown, --show-differences, --show-uptodate.
Maybe show a spinner or just the package count while it works?
Seeing the release date of the package would be important to see at least for me. For instance, I wanted to see if a new version of suse-migration-sle15-activation was released but while I see the version number, I then need to go into the SCC to find out the release date.
โฏ vercheck.py -v -p 1755 -n suse-migration-sle15-activation
Using product ID 1755 (SUSE Linux Enterprise Server for SAP 12 SP4 x86_64)
looking for suse-migration-sle15-activation on product id 1755
version 1.2.0-6.8.1 is available on repository [Public Cloud Module 12 x86_64]
version 1.2.0-6.5.1 is available on repository [Public Cloud Module 12 x86_64]
version 1.2.0-6.11.3 is available on repository [Public Cloud Module 12 x86_64]
latest version for suse-migration-sle15-activation is 1.2.0-6.11.3
could not determine architecture for the supportconfig directory. Please supply one with -a.
maybe -a x86_64 ???
As a collateral effect of issue #15 , packages that exist on both the main "Pool" repository and on PackageHub (with different versions!) now come up in the search results.
This leads to the wrong (?) package being selected as the latest version, like in this example for the package python3-hwdata on SUSE Manager 4.2:
2 packages with name containing "python3-hwdata" in SUSE Manager Server 4.2 x86_64
Name Version License Module
python3-hwdata 2.3.5-bp153.1.13 GPL-2.0 SUSE Package Hub 15 SP3
python3-hwdata 2.3.5-1.21 GPL-2.0 SUSE Manager Server Module 4.2
The version comparison class will select 2.3.5-bp153.1.13 over 2.3.5-1.21. Still considering what to do in this case.
Due to the potential amount of small writes with multiple instances, the current cache implementation (plain JSON file) only reads the cache at the startup, modifies it in-memory, then writes it at the end. This is not optimal, and as a collateral effect "only the last one wins". With an SQLite backend, I hope to mitigate this.
Curiously, the current implementation has the thread locking working fine and there have been no data corruptions, but SQLite would also help with that too.
See if it's possible to check for the required Python modules (pyaml and urllib3) and warn the users about it without a nasty crash :)
When trying to analyze the supportconfig report dir (vercheck-nocache.py -d scc...) appear bulk of messages:
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
/vercheck-nocache.py:638: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(v + '-' + r)
If vercheck is run against a supportconfig file for a SLE11 system, zero-length report files are written. It should not generate these files as SCC does not support SLE11 queries.
Hi,
according to https://docs.python.org/3/library/json.html#json.loads needs explicit encoding on older Python 3:
Changed in version 3.6: s can now be of type bytes or bytearray. The input encoding should be UTF-8, UTF-16 or UTF-32.
We applied this to get it working internally:
--- vercheck.py.old 2021-08-30 13:55:05.124482403 +0200
+++ vercheck.py 2021-08-30 13:15:28.445376243 +0200
@@ -478,11 +478,11 @@
if r.status == 200:
if tries > 0:
print('thread %d got a good reply after %d tries' % (self.instance_nr, tries))
- return_data = json.loads(r.data)
+ return_data = json.loads(r.data.decode('utf-8'))
valid_response = True
elif r.status in self.error_states:
if r.data:
- json_data = json.loads(r.data)
+ json_data = json.loads(r.data.decode('utf-8'))
print('cannot be processed due to error: [' + json_data['error'] + ']')
print('thread %d got a fatal error (%d). Results will be incomplete!\nPlease contact the service administrators or try again later.' % (self.instance_nr, r.status))
break
I noticed that single package searches are not updating the local JSON package cache, so for example:
# vercheck.py -p 2468 -n kernel-default
loaded 6532 items from cache (/home/erico/.cache/scc-tools/scc_data.json)
searching for package "kernel-default" in product id "2468" (SUSE Linux Enterprise Desktop 15 SP5 x86_64)
cached data for kernel-default is too old ( -134 days), discarding cache entry
searching for kernel-default for product ID 2468 in SCC
latest version for kernel-default on product ID 2468(SUSE Linux Enterprise Desktop 15 SP5 x86_64) is 5.14.21-150500.55.52.1, found on Basesystem Module (sle-module-basesystem/15.5/x86_64)
will always complain about the old cached data. The full directory analysis seems to update the cache correctly.
By default vercheck.py writes CSV files to the current directory. Adding a flag to specify the output directory for these files would be helpful for creating automatic checks.
I found following difference in the resulted logs
Analyzing supportconfig directory: scc_ABC-SGS-SP808_xxxxxx
product name = SUSE Linux Enterprise Server 15 SP1 x86_64 (id 1763, x86_64)
found 1171 total packages to check
gathering results...
notfound: gnome-desktop,2.28.2-0.6.18
different: gnome-desktop-lang,2.28.2-0.6.18,3.26.2-2.13
as package "gnome-desktop-lang" will be build as sub package of "gnome-desktop" i am pretty sure there is somewhere a error (probably parsing or program logic) hidden in vercheck.py
the affected packages according to rpm.txt
gnome-desktop SUSE Linux Enterprise 11 2.28.2-0.6.18
gnome-desktop-lang SUSE Linux Enterprise 11 2.28.2-0.6.18
probably they are leftovers from a upgrade from SLES11 to SLES15
e20ca18 May have broken some of the comparing, wasn't seeing these before hand.
> vercheck.py -d.
Analyzing supportconfig directory: .
product name = SUSE Linux Enterprise Server 15 SP2 x86_64 (id 1939, x86_64)
found 838 total packages to check
gathering results...
error comparing: 2.35-7.11.1 and 2.35.1-7.18.1 ([2, 35, '-', 7, 11, 1] and [2, 35, 1, '-', 7, 18, 1])
error comparing: 2.35-7.11.1 and 2.35.1-7.18.1 ([2, 35, '-', 7, 11, 1] and [2, 35, 1, '-', 7, 18, 1])
error comparing: 2.35-7.11.1 and 2.35.1-7.18.1 ([2, 35, '-', 7, 11, 1] and [2, 35, 1, '-', 7, 18, 1])
> grep -E '2.35.1-7.18.1|2.35-7.11.1' rpm.txt
binutils SUSE Linux Enterprise 15 2.35.1-7.18.1
libctf-nobfd0 SUSE Linux Enterprise 15 2.35.1-7.18.1
libctf0 SUSE Linux Enterprise 15 2.35.1-7.18.1
Add another way to specify product IDs, like "sles-15-sp1" or "sles_sap-12-sp4" (using CPE codes for standardization).
After yet another severe rate-limiting event from the API server, we found a situation where dozens of threads were waiting on the results from the server for hours, and the vercheck process hit a solid 100% CPU until it was killed.
Apparently, Python threading doesn't yield idle time back to the CPU properly. Will have to research that, or maybe go for the multiprocessing library as an alternative.
Users are getting this warning on newer Python versions:
$ ./vercheck.py -l
./vercheck.py:13: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils.version import LooseVersion
This is due to PEP 632, which will deprecate the Distutils package in the near future.
We need to re-implement the same features with setuptools.
HTTP error 429 is "Too many requests". It needs to be added to the retry_states list so it can be properly managed.
loaded 16277 items from cache (/home/user/.cache/scc-tools/scc_data.json)
loaded 5 items from cache (/home/user/.cache/scc-tools/public_cloud_amazon.json)
A colleague asked for adding a --version parameter. This is a bit tricky when dealing with GIT, unless I start using proper release tags and using git archive to do so. Even so, it seems things are a wee bit more complicated to have something like the old "$Id$" string in other version control systems.
Getting "got error 504 from the server" errors and noticed that the script just keeps going with new requests rather than retrying failed ones.
Adding handling for this could be helpful to prevent partial results
(11:22 jchandler@concord ~) > vercheck.py
File "/storage/tools/vercheck.py", line 438
ata['data']:
^
SyntaxError: invalid syntax
I noticed that whenever processing supportconfig data from a SUSE Manager server, the susemanager-* are not found in the repositories. It turns out that I need to use the proper product ID when doing my queries, instead of the base system product ID.
I can determine the product version from the basic-environment.txt with a regex, and from there use the proper ID.
Add a flag to allow the tool to check for versions on the current system.
As reported by Ralph Roth:
ralph at p5520 in ~/Documents/Eigene_TechDox/HealthCheck/scc-tools on master [?]
$ ll
total 128K
drwxr-xr-x 3 ralph users 193 25. Nov 16:41 ./
drwxr-xr-x 3 ralph users 4,0K 25. Nov 15:57 ../
drwxr-xr-x 8 ralph users 163 25. Nov 15:56 .git/
-rw-r--r-- 1 ralph users 1,1K 25. Nov 15:56 LICENSE
-rw-r--r-- 1 ralph users 6,6K 25. Nov 15:56 README.md
-rw-r--r-- 1 ralph users 18K 25. Nov 16:41 vercheck-different-.csv
-rw-r--r-- 1 ralph users 2,2K 25. Nov 16:41 vercheck-notfound-.csv
-rw-r--r-- 1 ralph users 448 25. Nov 16:41 vercheck-unsupported-.csv
-rw-r--r-- 1 ralph users 45K 25. Nov 16:41 vercheck-uptodate-.csv
-rwxr-xr-x 1 ralph users 34K 25. Nov 15:56 vercheck.py*
The script was called with:
$ ./vercheck.py -d /home/ralph/Desktop/.../unpacked/scc_czdcpv-uxnfs204_211101_1146/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.