clearlinux / swupd-client

Software update client

Swupd Client

swupd is an OS-level software update program that applies updates to a Clear Linux OS. More information on how to use swupd can be found in the official documentation, swupd command man page and other documents on docs directory.

Reporting problems

Use github issue report tool (https://github.com/clearlinux/swupd-client/issues) to report bugs, issues and feature requests.

To help us to reproduce a bug reported, please attach the swupd command line, the command output, swupd version ('swupd -v' output) and clear linux version ('swupd info' output).

For other questions, contact us in Clear Linux mailing list ([email protected]) or IRC channel ( #clearlinux on Freenode).

How to contribute

Swupd client is open for contributions, so we are more than willing to review your patches! For more information take a look on the how to contribute guide

Reporting Security Concerns

If you have discovered potential security vulnerability in Swupd, please visit https://01.org/security for best practices on reporting security issues and to report your concern.

swupd-client's Issues

why no ipv6 support?

Why is curl "hardcoded" to resolve all urls to ipv4?

curl.c
#warning "setup a means to validate IPv6 works end to end"
curl_ret = curl_easy_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4);

compiler warnings in xattrs.c

src/xattrs.c: In function ‘xattr_get_value’:
src/xattrs.c:56:40: warning: unused parameter ‘path’ [-Wunused-parameter]
 static int xattr_get_value(const char *path, const char *name, char **blob,
                                        ^~~~
src/xattrs.c:56:58: warning: unused parameter ‘name’ [-Wunused-parameter]
 static int xattr_get_value(const char *path, const char *name, char **blob,
                                                          ^~~~

compiler warning in manifest.c

src/manifest.c: In function ‘load_mom’:
src/manifest.c:665:4: warning: ignoring return value of ‘system’, declared with attribute warn_unused_result [-Wunused-result]
    (void)system(log_cmd);
    ^~~~~~~~~~~~~~~~~~~~~

search leaks memory

==25992== 
==25992== HEAP SUMMARY:
==25992==     in use at exit: 50,610 bytes in 728 blocks
==25992==   total heap usage: 4,862,164 allocs, 4,861,436 frees, 404,103,383 bytes allocated
==25992== 
==25992== 1,181 bytes in 22 blocks are definitely lost in loss record 59 of 66
==25992==    at 0x4C2AB8B: realloc (vg_replace_malloc.c:785)
==25992==    by 0x62AAB1C: __vasprintf_chk (in /usr/lib64/libc-2.22.so)
==25992==    by 0x4E3D43E: vasprintf (stdio2.h:210)
==25992==    by 0x4E3D43E: string_or_die (helpers.c:651)
==25992==    by 0x404CCB: query_total_download_size (search.c:364)
==25992==    by 0x404CCB: download_manifests (search.c:419)
==25992==    by 0x4052AA: search_main (search.c:499)
==25992==    by 0x61B757F: (below main) (in /usr/lib64/libc-2.22.so)
==25992== 
==25992== 4,690 bytes in 110 blocks are definitely lost in loss record 63 of 66
==25992==    at 0x4C28BF6: malloc (vg_replace_malloc.c:299)
==25992==    by 0x62AAB47: __vasprintf_chk (in /usr/lib64/libc-2.22.so)
==25992==    by 0x4E3D43E: vasprintf (stdio2.h:210)
==25992==    by 0x4E3D43E: string_or_die (helpers.c:651)
==25992==    by 0x404CCB: query_total_download_size (search.c:364)
==25992==    by 0x404CCB: download_manifests (search.c:419)
==25992==    by 0x4052AA: search_main (search.c:499)
==25992==    by 0x61B757F: (below main) (in /usr/lib64/libc-2.22.so)
==25992== 
==25992== 10,470 bytes in 130 blocks are definitely lost in loss record 64 of 66
==25992==    at 0x4C2AB8B: realloc (vg_replace_malloc.c:785)
==25992==    by 0x62AAB1C: __vasprintf_chk (in /usr/lib64/libc-2.22.so)
==25992==    by 0x4E3D43E: vasprintf (stdio2.h:210)
==25992==    by 0x4E3D43E: string_or_die (helpers.c:651)
==25992==    by 0x404CFE: query_total_download_size (search.c:369)
==25992==    by 0x404CFE: download_manifests (search.c:419)
==25992==    by 0x4052AA: search_main (search.c:499)
==25992==    by 0x61B757F: (below main) (in /usr/lib64/libc-2.22.so)
==25992== 
==25992== 27,648 (56 direct, 27,592 indirect) bytes in 1 blocks are definitely lost in loss record 66 of 66
==25992==    at 0x4C2A988: calloc (vg_replace_malloc.c:711)
==25992==    by 0x4E3F061: alloc_manifest (manifest.c:154)
==25992==    by 0x4E3F061: manifest_from_file (manifest.c:244)
==25992==    by 0x4E400DD: load_mom (manifest.c:597)
==25992==    by 0x404C38: download_manifests (search.c:410)
==25992==    by 0x4052AA: search_main (search.c:499)
==25992==    by 0x61B757F: (below main) (in /usr/lib64/libc-2.22.so)
==25992== 
==25992== LEAK SUMMARY:
==25992==    definitely lost: 16,397 bytes in 263 blocks
==25992==    indirectly lost: 27,592 bytes in 400 blocks
==25992==      possibly lost: 0 bytes in 0 blocks
==25992==    still reachable: 6,621 bytes in 65 blocks
==25992==         suppressed: 0 bytes in 0 blocks

file descriptor leak

swupd commands sometime show a confusing message like:

"possible file descriptor leak fd_number="4", fd_details="socket:[######]"

Input validation bug for --format

If I supply an invalid value as input to --format, regardless of subcommand, I expect swupd to always exit with an error at the validation step. However, it appears that certain values slip through the input validation, which should only permit positive integer values, or the special string "staging".

# swupd update --format 7F
swupd-client software update 3.6.5
   Copyright (C) 2012-2016 Intel Corporation

Attempting to download version string to memory
Error: Network issue, unable to proceed with update
Update took 0 seconds

Handle type changes

For the trivial case, empty dir -> symlink the code should work to just remove the dir and create a new link. However, considering the case of a directory that's populated, we need to do better:

If dir is empty: rmdir & create link
If dir is not empty: all files should be marked deleted, delete all marked files first & try to rmdir
If leftover content remains: mv dir -> dir.timestamp, create the new link/file & continue.
#168

autoreconf warning

When running ./autogen.sh or autoreconf -fi, the following warning is emitted:

Makefile.am:220: warning: ':='-style assignments are not portable
Makefile.am:220: warning: patsubst %.bats,%.log,$(filter %.bats,$(dist_check_SCRIPTS: non-POSIX variable name
Makefile.am:220: (probably a GNU make extension)
Makefile.am:286: warning: '%'-style pattern rules are a GNU make extension

interrupted download not resumed

swupd client 3.3.0
Python 2.7.9 SimpleHTTP as server

If I interrupt "swupd bundle-add full" ("full" being the bundle name) while it is downloading pack-full-from-0-to-41309480.tar, it leaves /var/lib/swupd/pack-full-from-0-to-41309480.tar behind.

Retrying the operation then fails:

# swupd bundle-add --url http://192.168.7.1:8000 full
swupd-client bundle adder 3.3.0
   Copyright (C) 2012-2016 Intel Corporation

Added bundle full for installation
Downloading required packs...
Attempting to download version string to memory
Downloading full pack for version 41309480
Curl error: 33 - see curl.h for details
pack downloads failed, cannot proceed with the installation, exiting.
Possible filedescriptor leak: fd_number="3",fd_details="/dev/urandom"

strace shows:

write(1, "Downloading full pack for versio"..., 43Downloading full pack for version 41309480
) = 43
lstat64("/var/lib/swupd/pack-full-from-0-to-41309480.tar", {st_mode=S_IFREG|0600, st_size=3718986, ...}) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 5
fcntl64(5, F_GETFL)                     = 0x2 (flags O_RDWR)
fcntl64(5, F_SETFL, O_RDWR|O_NONBLOCK)  = 0
connect(5, {sa_family=AF_INET, sin_port=htons(8000), sin_addr=inet_addr("192.168.7.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
...
getpeername(5, {sa_family=AF_INET, sin_port=htons(8000), sin_addr=inet_addr("192.168.7.1")}, [16]) = 0
getsockname(5, {sa_family=AF_INET, sin_port=htons(36868), sin_addr=inet_addr("192.168.7.2")}, [16]) = 0
...
send(5, "GET /41309480/pack-full-from-0.t"..., 107, MSG_NOSIGNAL) = 107
...
recv(5, "HTTP/1.0 200 OK\r\nServer: SimpleH"..., 16384, 0) = 16384
...
close(5)                                = 0
write(1, "Curl error: 33 - see curl.h for "..., 40Curl error: 33 - see curl.h for details
) = 40
lstat64("/var/lib/swupd/pack-full-from-0-to-41309480.tar", {st_mode=S_IFREG|0600, st_size=3718986, ...}) = 0
write(1, "pack downloads failed, cannot pr"..., 70pack downloads failed, cannot proceed with the installation, exiting.
) = 70
close(4)                                = 0

I am using Python 2.7.9 SimpleHTTP as server (https://github.com/ostroproject/ostro-os/blob/master/scripts/swupd-http-server).

adding a bundle, removing the bundle, and then adding the same bundle behaves strangely

There appears to be some staging directory strangeness when adding a bundle:

swupd bundle-add rust-basic

then removing that bundle

swupd bundle-remove rust-basic

then adding that bundle back again

swupd bundle-add rust-basic
swupd-client bundle adder 3.12.1
   Copyright (C) 2012-2017 Intel Corporation

Attempting to download version string to memory
Downloading packs...
	...100%
Installing bundle(s) files...
	...0%Path /usr/bin/cargo is missing on the file system
Path /usr/bin/rust-gdb is missing on the file system
Path /usr/bin/rust-lldb is missing on the file system
Path /usr/bin/rustc is missing on the file system
Path /usr/bin/rustdoc is missing on the file system
	...35%Path /usr/lib64/libarena-5fe3094a53c2fc91.so is missing on the file system
Path /usr/lib64/libfmt_macros-091a58a94fe20486.so is missing on the file system
...
<snip>
...
Path /usr/share/man/man1/cargo.1 is missing on the file system
	...42%Path /usr/share/man/man1/rustc.1 is missing on the file system
Path /usr/share/man/man1/rustdoc.1 is missing on the file system
	...43%Path /usr/share/zsh/site-functions/_cargo is missing on the file system
	...87%
Calling post-update helper scripts.
Bundle(s) installation done.

In the case above the bundle at least seems to be fully installed to the system.

Wiping the state dir in between the bundle adds fixes the issue of course.

update with uid/gid change

Is swupd supposed to handle OS updates where the IDs defined in /etc/passwd and/or /etc/group change, or is that a situation which needs to be avoided?

Right now, when that happens, "swupd updated" fails with "File content hash mismatch for /var/lib/swupd/staged/05....2bd (bad server data?)" where the file is one of those whose user or group ID changed its numeric value.

The reason is that tar uses the symbolic names during unpacking on the client: it assigns the current numeric ID as defined in /etc, whereas the server used the new numeric ID. Because the numeric ID is the input for hashing, a hash mismatch occurs, which prevents updating.

For bsdtar, adding "--numeric-owner" to the TAR_COMMAND fixes that aspect of the problem. A similar fix should be possible for GNU tar.

But that only addresses a subset of the problem. Processes running with the old ID also need to be restarted, which is harder.

verify_fix_path not called for manifests

This function is meant to be a generic helper to insure integrity and survive-ability. It is not applied to manifests that don't hash check and actually should be.

versionstamp never created

This file might be created outside the scope of swud-client but in a Yocto/meta-swupd environment I don't see any where that creates the /usr/share/clear/versionstamp file if it doesn't exist. Shouldn't verifytime create this if the file is missing or invalid (it seems to update it if the file exists and has a timestamp).

compiler warning in check_update.c

src/check_update.c: In function ‘check_update’:
src/check_update.c:142:6: warning: the address of ‘check_network’ will always evaluate as ‘true’ [-Waddress]
  if (!check_network) {
      ^

Add negative testing for hash verification

The lack of negative testing for hash verification allowed an issue in bundleadd to go completely unnoticed since the addition of the feature. The bundleadd command never actually checked the hashes of the individual files it was laying on the system. This was never caught because there were no tests to insure mismatched files were rejected by swupd-client.

Content-Length not checked

A malicious/misbehaving server can respond without Content-Length header data and send infinite data to requests, which will block the client.
Tested on swupd v2.87 with a custom server.

Below the HTTPRequestHandler for a python HTTP Server for reproduction:

class InfiniteData(BaseHTTPServer.BaseHTTPRequestHandler):
    """Handler that returns infinite data"""
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        while True:
            self.wfile.write(chr(255))

One solution would be to actual check for Content-Length header information.

remove manifest timestamp

We've never used this timestamp for anything. Manifests are meant to be immutable once created. So the timestamps should only flow forward and no issues.

In the unfortunate situation of a DevOps group needing to recreate old Manifests, they will have a new timestamp, and a changed hash relative what an old MoM (ie: assume MoM's got recreated, but some clients had downloaded and cached an old MoM, then downloaded a new bundle manifest on bundle add), which leads to an error/warn in the client at runtime.

The error/warn can be addressed via Issue #94, but that code path would also have never hit if the manifest did not have the timestamp in it.

Let's simply remove the timestamp. This can be done without a format bump.

Improve progress for redirects

Redirected output prints a . on the same line for every 10 files it installs on the system (or processes or downloads etc.). User-facing programs that display this output to the user, such as Ister, only display the output when a newline is printed. Because of this the user still will not see swupd progress until the task is complete, at which point it will print out a single extremely long line of dots:

................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

This is not very useful to the user as they still could think the process is hanging. Once it finally completes they get a very meaningless block of dots (since the line is wrapped).

It would be better to print redirected progress as a percentage of the total, each percentage on a new line:

5%
10%
15%
...
100%

bundle-remove should say which bundles are blocking the removal of a bundle

Which other installed bundles are blocking me to remove a bundle

bundle-remove should say which bundles are blocking the removal of a bundle

$ sudo swupd bundle-remove xfce4-desktop
swupd-client bundle remover 3.10.0
   Copyright (C) 2012-2016 Intel Corporation

Removing bundle: xfce4-desktop
Error: bundle requested to be removed is required by other installed bundles
Error: Bundle remove failed
1 bundle(s) of 1 failed to remove

Improve progress reporting for pack download/extraction

Pack download/extraction progress reporting is sub-par and currently just makes the output more verbose without actually giving the user better information. The following format should be achieved

Downloading 3 packs...
[1/3] Downloading pack os-core for version 17200...
[1/3] Extracting pack...
[2/3] Downloading pack test-bundle for version 17110...
[2/3] Extracting pack...
[3/3] Downloading pack test-bundle-2 for version 17120...
[3/3] Extracting pack...

The problem is that we don't know if those packs exist until we try to download them.

Some (failing) work can be found on https://github.com/matthewrsj/swupd-client/tree/improve-progress

Add alias to bundle-add and bundle-remove

As a linux user the "common mindset is to use install" when it comes to install a new component in a system:

Example:

# dnf install <foo>
# apt-get install <foo>
# apt install <foo>
# rpm --install <foo>
# dpkg --install <foo>

So as a way to lower the adotpion curve for any new user would be very handy to have aliases for bundle-add and bundle-remove so the following can work:

• Add a new bundle
  swupd install sysadmin-hostmgmt

• Remove a bundle
  swupd remove sysadmin-hostmgmt

Consider using libnica

The libnica project from https://github.com/ikeydoherty/libnica provides a nice, minimal C library that swupd-client could leverage.

As an example, in #74, it was suggested that we could use the nc_mkdir_p instead of the equivalent of system("mkdir -p").

bundle add/remove memory leaks

Bundle add and remove have memory management issues. Eg:

sudo valgrind --leak-check=full swupd verify --fix -m 8970 -p ./foo
sudo valgrind --leak-check=full swupd bundle-add -p ./foo sysadmin

==23866== 
==23866== HEAP SUMMARY:
==23866==     in use at exit: 7,107 bytes in 77 blocks
==23866==   total heap usage: 68,850 allocs, 68,773 frees, 4,995,890 bytes allocated
==23866== 
==23866== 24 bytes in 1 blocks are definitely lost in loss record 30 of 69
==23866==    at 0x4C28BF6: malloc (vg_replace_malloc.c:299)
==23866==    by 0x4E3EA03: list_alloc_item (list.c:58)
==23866==    by 0x4E3EA03: list_prepend_data (list.c:153)
==23866==    by 0x4E3EC4A: list_clone (list.c:296)
==23866==    by 0x4E3E5CC: files_from_bundles (helpers.c:884)
==23866==    by 0x4E43E44: install_bundles (bundle.c:429)
==23866==    by 0x4E44064: install_bundles_frontend (bundle.c:542)
==23866==    by 0x61B757F: (below main) (in /usr/lib64/libc-2.22.so)
==23866== 
==23866== 71 (24 direct, 47 indirect) bytes in 1 blocks are definitely lost in loss record 45 of 69
==23866==    at 0x4C28BF6: malloc (vg_replace_malloc.c:299)
==23866==    by 0x4E3EA03: list_alloc_item (list.c:58)
==23866==    by 0x4E3EA03: list_prepend_data (list.c:153)
==23866==    by 0x4E3F2B2: manifest_from_file (manifest.c:237)
==23866==    by 0x4E4017D: load_manifest (manifest.c:640)
==23866==    by 0x4E40290: recurse_manifest (manifest.c:836)
==23866==    by 0x4E43E33: install_bundles (bundle.c:422)
==23866==    by 0x4E44064: install_bundles_frontend (bundle.c:542)
==23866==    by 0x61B757F: (below main) (in /usr/lib64/libc-2.22.so)
==23866== 
==23866== 391 (24 direct, 367 indirect) bytes in 1 blocks are definitely lost in loss record 67 of 69
==23866==    at 0x4C28BF6: malloc (vg_replace_malloc.c:299)
==23866==    by 0x4E3EA03: list_alloc_item (list.c:58)
==23866==    by 0x4E3EA03: list_prepend_data (list.c:153)
==23866==    by 0x4E4024A: recurse_manifest (manifest.c:842)
==23866==    by 0x4E43E33: install_bundles (bundle.c:422)
==23866==    by 0x4E44064: install_bundles_frontend (bundle.c:542)
==23866==    by 0x61B757F: (below main) (in /usr/lib64/libc-2.22.so)
==23866== 
==23866== LEAK SUMMARY:
==23866==    definitely lost: 72 bytes in 3 blocks
==23866==    indirectly lost: 414 bytes in 9 blocks
==23866==      possibly lost: 0 bytes in 0 blocks
==23866==    still reachable: 6,621 bytes in 65 blocks
==23866==         suppressed: 0 bytes in 0 blocks

sudo valgrind --leak-check=full swupd bundle-remove -p ./foo sysadmin

==23512== 
==23512== HEAP SUMMARY:
==23512==     in use at exit: 6,645 bytes in 66 blocks
==23512==   total heap usage: 37,162 allocs, 37,096 frees, 2,514,778 bytes allocated
==23512== 
==23512== 24 bytes in 1 blocks are definitely lost in loss record 26 of 59
==23512==    at 0x4C28BF6: malloc (vg_replace_malloc.c:299)
==23512==    by 0x4E3EA03: list_alloc_item (list.c:58)
==23512==    by 0x4E3EA03: list_prepend_data (list.c:153)
==23512==    by 0x4E4024A: recurse_manifest (manifest.c:842)
==23512==    by 0x4E43B25: load_bundle_manifest (bundle.c:113)
==23512==    by 0x4E43B25: remove_bundle (bundle.c:296)
==23512==    by 0x4046FD: bundle_remove_main (clr_bundle_rm.c:158)
==23512==    by 0x61B757F: (below main) (in /usr/lib64/libc-2.22.so)
==23512== 
==23512== LEAK SUMMARY:
==23512==    definitely lost: 24 bytes in 1 blocks
==23512==    indirectly lost: 0 bytes in 0 blocks
==23512==      possibly lost: 0 bytes in 0 blocks
==23512==    still reachable: 6,621 bytes in 65 blocks
==23512==         suppressed: 0 bytes in 0 blocks

Paths should be configurable and shouldn't assume Clear Linux conventions

To help ease adoption on non-Clear Linux distributions paths which are currently hard-coded (and defaulting to Clear Linux conventions) should be easily overridden.

The following hard-coded paths are hard-coded to match Clear Linux conventions in swupd-client 3.5.3:

• BUNDLES_DIR set to "/usr/share/clear/bundles" in configure.ac
• UPDATE_CA_CERTS_PATH set to "/usr/share/clear/update-ca" in configure.ac
• SIGNATURE_CA_CERT set to "test-do-not-ship-R0-0.pem" in configure.ac
• CURLOPT_PINNEDPUBLICKEY set to "/usr/share/clear/update-ca/425b0f6b.key" in curl.c

Several other paths are hardcoded, but with slightly less unusual defaults, including:

• STATE_DIR set to "/var/lib/swupd" in configure.ac
• LOG_DIR set to "/var/log/swupd" in configure.ac
• LOCK_DIR set to "/run/lock" in configure.ac
• MOTD_FILE set to "/usr/lib/motd.d/001-new-release" in configure.ac

test suite fails when ClearLinuxRoot.pem is not installed

When /usr/share/clear/update-ca/ClearLinuxRoot.pem is not installed on the host system, the test suite fails. This is due to an extra line being printed unexpectedly: "Failed to fopen /usr/share/clear/update-ca/ClearLinuxRoot.pem".

segfault during out-of-disk error

Old swupd 2.87 so this might not be applicable anymore and I can check that after updating to a newer version, but in the meantime let me dump the information that I have so far here.

When swupd client runs out of disk space during a "swupd bundle-add" on Ostro OS, it segfaults. The reason is using a NULL pointer in file->filename:

gdb) where
#0  __GI___strnlen (str=0x0, maxlen=4095)
    at /usr/src/debug/glibc/2.23-r0/git/string/strnlen.c:120
#1  0xb7e8cd41 in __GI_strncpy (s1=0xbfffe76c "", s2=0x0, n=4095)
    at /usr/src/debug/glibc/2.23-r0/git/string/strncpy.c:29
#2  0xb7fb59e9 in strncpy (__len=4095, __src=<optimized out>, 
    __dest=0xbfffe76c "") at /usr/include/bits/string3.h:126
#3  format_log_message (log_type=log_type@entry=1, 
    priority=priority@entry=log_error, file=file@entry=0x80022fe8, 
    msg=0xb7fc3bf4 "short write", filename=0xb7fc3bb1 "src/curl.c", 
    linenr=151, fmt=0xb7fc3aa8 "\\*outfile=\"%s\",strerror=\"%s\"*\\", 
    ap=0xbffff81c "\330I\003\200\v\332\363\267")
    at /usr/src/debug/swupd-client/2.87-r0/swupd-client-2.87/src/log.c:416
#4  0xb7fb5ef8 in __log_message (priority=log_error, file=0x80022fe8, 
    class_msg=class_file_io, msg=0xb7fc3bf4 "short write", 
    filename=0xb7fc3bb1 "src/curl.c", linenr=151, 
    fmt=0xb7fc3aa8 "\\*outfile=\"%s\",strerror=\"%s\"*\\")
    at /usr/src/debug/swupd-client/2.87-r0/swupd-client-2.87/src/log.c:559
#5  0xb7fb136a in swupd_download_file (ptr=0x80023bc4, size=1, nmemb=16384, 
    userdata=0x80022fe8)
    at /usr/src/debug/swupd-client/2.87-r0/swupd-client-2.87/src/curl.c:150
#6  0xb7ba0604 in Curl_client_chop_write (conn=0x80035068, type=1, 
    ptr=0x80023bc4 "\261\234*E\001\347@\351;t\v\277W\303!=\244\250T\360\022%+\332\t\213\367\375{B\301\r\t:\270 \253\026\301jL\230\274\373\274\316\310\t\ax\313\2---Type <return> to continue, or q <return> to quit---
71\337\217\375\344k\344\373^G\352w+W\312@\207\353j\223\275+1r\177\371\200\300\352\332$v\345\\}{\005\263\376\231r\244\325\300z\202Q\244jGN \271\300\223\203\244\342\206\301TM\270\376\066\362V\221Y+\332\246\325\022\323\343\"\323\063\374H38\037M\272\062\061\320\235\367E\310\001\177\026\304\233\341L\245\261\064\311\312\037\332\303\216O\334s&\227\345\213\030\021\031\067\354_J\232R\322N\235\025\245K\322\326\376m\220v\343R\222TvO\205\261\373\257!\002"..., len=16384)
    at /usr/src/debug/curl/7.47.1-r0/curl-7.47.1/lib/sendf.c:434
#7  0xb7ba07c5 in Curl_client_write (conn=0x80035068, type=1, 
    ptr=0x80023bc4 "\261\234*E\001\347@\351;t\v\277W\303!=\244\250T\360\022%+\332\t\213\367\375{B\301\r\t:\270 \253\026\301jL\230\274\373\274\316\310\t\ax\313\271\337\217\375\344k\344\373^G\352w+W\312@\207\353j\223\275+1r\177\371\200\300\352\332$v\345\\}{\005\263\376\231r\244\325\300z\202Q\244jGN \271\300\223\203\244\342\206\301TM\270\376\066\362V\221Y+\332\246\325\022\323\343\"\323\063\374H38\037M\272\062\061\320\235\367E\310\001\177\026\304\233\341L\245\261\064\311\312\037\332\303\216O\334s&\227\345\213\030\021\031\067\354_J\232R\322N\235\025\245K\322\326\376m\220v\343R\222TvO\205\261\373\257!\002"..., len=<optimized out>)
    at /usr/src/debug/curl/7.47.1-r0/curl-7.47.1/lib/sendf.c:511
#8  0xb7bb21f0 in readwrite_data (done=0xbffff983, 
    didwhat=<synthetic pointer>, k=0x8002369c, conn=0x80035068, 
    data=0x80023650)
    at /usr/src/debug/curl/7.47.1-r0/curl-7.47.1/lib/transfer.c:753
#9  Curl_readwrite (conn=0x80035068, data=0x80023650, done=0xbffff983)
    at /usr/src/debug/curl/7.47.1-r0/curl-7.47.1/lib/transfer.c:1074
---Type <return> to continue, or q <return> to quit---
#10 0xb7bbaf98 in multi_runsingle (multi=multi@entry=0x80023318, now=..., 
    data=0x80023650)
    at /usr/src/debug/curl/7.47.1-r0/curl-7.47.1/lib/multi.c:1544
#11 0xb7bbbd9e in curl_multi_perform (multi_handle=0x80023318, 
    running_handles=0xbffffa40)
    at /usr/src/debug/curl/7.47.1-r0/curl-7.47.1/lib/multi.c:1821
#12 0xb7bb3c66 in easy_transfer (multi=0x80023318)
    at /usr/src/debug/curl/7.47.1-r0/curl-7.47.1/lib/easy.c:724
#13 easy_perform (events=false, data=0x80023650)
    at /usr/src/debug/curl/7.47.1-r0/curl-7.47.1/lib/easy.c:812
#14 curl_easy_perform (easy=0x80023650)
    at /usr/src/debug/curl/7.47.1-r0/curl-7.47.1/lib/easy.c:831
#15 0xb7fb1917 in swupd_curl_get_file (
    url=0x80034a40 "http://192.168.7.1:8000/41213300/pack-full-dev-from-0.tar", filename=0x800349d8 "/var/lib/swupd/pack-full-dev-from-0-to-41213300.tar", 
    file=0x0, in_memory_version_string=0x0, pack=true)
    at /usr/src/debug/swupd-client/2.87-r0/swupd-client-2.87/src/curl.c:260
#16 0xb7fb8d6a in download_pack (module=0x8000bae8 "full-dev", 
    newversion=41213300, oldversion=0)
    at /usr/src/debug/swupd-client/2.87-r0/swupd-client-2.87/src/packs.c:63
#17 download_subscribed_packs (oldversion=0, newversion=41213300, 
    required=true)
    at /usr/src/debug/swupd-client/2.87-r0/swupd-client-2.87/src/packs.c:133
---Type <return> to continue, or q <return> to quit---
#18 0xb7fbde90 in install_bundles (bundles=<optimized out>)
    at /usr/src/debug/swupd-client/2.87-r0/swupd-client-2.87/src/bundle.c:391
#19 0x800040d2 in bundle_add_main (argc=4, argv=0xbffffd98)
    at /usr/src/debug/swupd-client/2.87-r0/swupd-client-2.87/src/clr_bundle_add.c:153
#20 0x80001865 in main (argc=5, argv=0xbffffd94)
    at /usr/src/debug/swupd-client/2.87-r0/swupd-client-2.87/src/swupd.c:138
(gdb) p file->filename
$5 = 0x0

Enable functional test suite to run in a user namespace

To reduce the reliance on 'sudo' for the testsuite, it would be nice to see the functional test suite enabled to run with systemd-nspawn or unshare -U to run in a user namespace.

swupd download slow when using IMA

swupd client 3.3.0 constantly opens and closes the file it is writing to during downloads (pack-from-0 in my test). That behavior incurs a huge (*) performance penalty when running on a device where IMA is active with on-device hashing, because each close() implies hashing the file content to update the security.ima xattr.

(*) a few seconds when using just curl vs. half an hour and still not done with swupd client.

swupd should open each file once and keep it open until the download is complete.

verify --install should be made a top-level command

the --install flag to the verify top-level command should be made a top-level command, and decoupled from the verify codepath for a couple reasons:

• --install is designed to do something completely different (install an OS to a blank partition/directory)
• --install actually does something completely different in that it doesn't verify the existing files in the target directory, it just assumes it doesn't need to do anything for them. Perhaps it should be verifying those files when it exists, but it currently does not.

When moved to a top-level command it should probably be named something different to prevent a new user from running

swupd install <bundlename>

expecting to install a new bundle (like apt-get install pkgname) but accidentally installing a whole OS to the current directory (obviously checks would be in place, however)

Fix direct renames

Currently the complicated cases of renames works (rename and content change) but the simple case (direct rename, no content change) does not. swupd-client will put the new file in place but leave the old file (same content, different name) in place as well.

Move verify_file to consistent location

The verify_file function is called in several unexpected and obscure parts of the code, making it hard to determine which codepaths are actually verifying file hashes and which are not. The verify_file function should be moved to a more visible location that touches all code paths.

The best location that I can tell is in do_staging which is called by all paths that actually put files on the system.

compiler warnings in search.c

src/search.c: In function 'search_main':
src/search.c:249:6: warning: argument 2 null where non-null expected [-Wnonnull]
  pos = strstr(filename, path);
  ~~~~^~~~~~~~~~~~~~~~~~~~~~~~
In file included from src/search.c:30:0:
/usr/include/string.h:329:14: note: in a call to function 'strstr' declared here
 extern char *strstr (const char *__haystack, const char *__needle)
              ^~~~~~
src/search.c:255:23: warning: argument 1 null where non-null expected [-Wnonnull]
  if (strcasestr(pos + strlen(path), search_term)) {
                       ^~~~~~~~~~~~
In file included from src/search.c:30:0:
/usr/include/string.h:384:15: note: in a call to function 'strlen' declared here
 extern size_t strlen (const char *__s)
               ^~~~~~
src/search.c:255:6: warning: argument 2 null where non-null expected [-Wnonnull]
  if (strcasestr(pos + strlen(path), search_term)) {
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from src/search.c:30:0:
/usr/include/string.h:359:14: note: in a call to function 'strcasestr' declared here
 extern char *strcasestr (const char *__haystack, const char *__needle)
              ^~~~~~~~~~

review --url x --versionurl

Please ensure --versionurl is available for all commands that needs version_server_urls. Currently we have few inconsistencies:

• check-update uses --url while it should be called --versionurl.
• bundle-add, bundle-rm and search all miss --versionurl, only calculating it from --url.

The idea is that we can use different version URL to fix versions, like "qa-approved" and "milestones". They would be different numbers (tags) to the same update stream.

Add basic validation of value passed to --with-formatid

The --with-formatid flag to configure sets the builtin default format number for swupd-client. The value should either be "staging", or a positive integer value; validation for these restrictions should be added to configure.ac.

bundle-remove does not handle dependencies

The bundle-remove subcommand should be dependency aware and not remove bundles when they are required by another bundle still installed on the system.

verify: bogus message "/proc was not in a pack"

I'm running client version v3.12.0 on Clear Linux build 17610.

On occasion, I see the "/proc was not in a pack" message appear when I run swupd verify --fix. However, after downloading the fullfile tar and extracting it, swupd performs no work to "fix" whatever issue it detected earlier on, since it reports:

Adding any missing files

Fixing modified files
Inspected 149309 files
  0 files were missing
  0 files found which should be deleted

I then reran swupd verify --fix, and the message did not appear. This appears to be the correct behavior.

update inactive partition

We'd like to do A/B partition updates with swupd. "swupd verify --install/--fix" almost work, but not quite.

The expectation is:

• Existing files are verified (attributes + content).
• This full check should be on by default, with an option to disable it when the existing file content is trusted and the hash for it is known (i.e. there's a manifest for it) - not crucial.
• Delta packs should be used when possible.
• Switching between unrelated update streams has to work (for exampe, when comparing versions the code must be aware that version 100 on disk might not be the same as the target version 100 when the update stream is different).
• This is needs to be a "first class citizen" in the features supported by swupd:
  • top-level command would be better than hiding it behind "verify" (not crucial)
  • automated testing for the feature under various circumstances
• The swupd state directory either has to be on the target partition, or there must be an option to put the staging directory there.
• The return code of the command must indicate success or failure.

"swupd verify --install" does not actually replace modified files. To reproduce with 3.12.0:

# rm -rf /tmp/swupd-state/ /tmp/swupd-root/ && mkdir /tmp/swupd-root
# swupd verify --install --picky -c https://download.clearlinux.org/update/ -v https://download.clearlinux.org/update/ -F 19 -m 17700 --time -S /tmp/swupd-state -p /tmp/swupd-root
...
# swupd verify --install --picky -c https://download.clearlinux.org/update/ -v https://download.clearlinux.org/update/ -F 19 -m 17730 --time -S /tmp/swupd-state -p /tmp/swupd-root
...
Adding any missing files

Inspected 2093 files
  0 files were missing
Calling post-update helper scripts.
...
# swupd verify --fix --picky -c https://download.clearlinux.org/update/ -v https://download.clearlinux.org/update/ -F 19 -m 17730 --time -S /tmp/swupd-state -p /tmp/swupd-root
...
Adding any missing files

Fixing modified files
Hash mismatch for file: /tmp/swupd-root/usr/lib/os-release
	fixed
Hash mismatch for file: /tmp/swupd-root/usr/lib/systemd/systemd-bootchart
	fixed
Hash mismatch for file: /tmp/swupd-root/usr/share/clear/version
	fixed
Hash mismatch for file: /tmp/swupd-root/usr/share/clear/versionstamp
	fixed
--picky removing extra files under /tmp/swupd-root/usr
Inspected 2067 files
  0 files were missing
  4 files did not match
    4 of 4 files were fixed
    0 of 4 files were not fixed
  0 files found which should be deleted

Only the "verify --fix" really changed the files.

Running the "verify --install -m 17700" followed by "verify --fix -m 17730" under strace and inspecting /tmp/swupd-state shows that it downloaded individual files instead of the pack-os-core-from-17700.tar.

I don't think it is necessary to support a "copy" mode of operation for moving files from staging to the actual tree. Just ensure that one can put the staging directory into the partition even when the state directory is elsewhere.

Do pre-bundle-add and pre-os-install disk space checks

Instead of trying to do a bundle-add or OS installation and failing due to out-of-disk errors, perform a disk-space check before attempting the operation.

This can be coupled with the bundle-list --deps functionality and the fixed contentsize in the manifests to get a required disk space metric.

compiler warning in clr_bundle_rm.c

src/clr_bundle_rm.c:224:1: warning: ‘static’ is not at beginning of declaration [-Wold-style-declaration]
 void static free_saved_opts(void)
 ^~~~

Paths need to be configurable

Swupd should allow configuring of
LOCK_DIR
BUNDLES_DIR
MOTD_FILE

using --with-* options.

Implement better functional test creation workflow

Currently, in order to write a new functional test, the BKM is to copy an existing functional test similar to the new one you want to write, and modify it accordingly. Pain points include several manual steps: (a) calculating swupd hashes of files, (b) updating the manifests according to the test's needs, and (c) calculating the swupd hash of each bundle manifest to include in the Manifest.MoM as a final step.

A workflow should be implemented to facilitate automating test creation, in particular the three items listed above.

Retry code rework

The retry code in swupd is not an ideal implementation, making it difficult to use throughout the code base. Some parts retry differently based on needing to support retries for network errors, or retry based on IO errors. Refactoring this code into a consistent, callable implementation would benefit many codepaths and potentially reduce the complexity and obscure bugs.

show details when bundle cannot be removed

# swupd bundle-remove libX11client
swupd-client bundle remover 3.8.1
   Copyright (C) 2012-2016 Intel Corporation

Error: bundle requested to be removed is required by other installed bundles
Error: Bundle remove failed

It would be more useful if the message was something like:

Error: bundle requested to be removed is required by other installed bundle(s):
   kvm-host
   tcl-basic
   xfce4-desktop
Error: Bundle remove failed

provide a comand to show files into the bundle, installed and available

It would be nice to nice to have "--listfile" options to show files inside a bundle that is installed and ones that are available but not installed.

(Jonh)

swupd is unable to downgrade from version

In order to downgrade CLR from version: 17230 to version 16860 I executed:

• swupd verify -f -m 16860

• As result swupd reported this error:

...
Hash mismatch for file: /usr/share/vulkan/icd.d/intel_icd.x86_64.json
        fixed
Inspected 232925 files
  8231 files were missing
    8231 of 8231 missing files were replaced
    0 of 8231 missing files were not replaced
  7915 files did not match
    7915 of 7915 files were fixed
    0 of 7915 files were not fixed
  0 files found which should be deleted
Calling post-update helper scripts.
sh: relocation error: /usr/lib64/haswell/libc.so.6: symbol __tunable_get_val, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference
sh: relocation error: /usr/lib64/haswell/libc.so.6: symbol __tunable_get_val, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference
sh: relocation error: /usr/lib64/haswell/libc.so.6: symbol __tunable_get_val, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference
Fix successful

• After that Clear Linux was broken:

# cat /usr/lib/os-release
cat: relocation error: /usr/lib64/haswell/libc.so.6: symbol __tunable_get_val, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

Perform better checks for disk space when installing

Now that contentsize is more accurate from swupd-server, it should be possible to do a sanity check on bundles to be installed and fail out early instead of reporting a ton of not fixed errors when space runs out.

provide a comand to show update URL swupd is consuming

Currently, there is no other way for users to know the URL swupd consumes for update but by looking at /usr/share/defaults/swupd/contenturl.

If a user creates a mix with his/her own content, the update URL for swupd-client would differ from default clearlinux URL.

would be really helpful for those who run a mix to have a command to check the content URL swupd is consuming. e.g:

# swupd info contenturl

it could show both contenturl and versionurl.

Add -m flag for swupd update

Add flag to allow user to update to a particular version as long as that version is > the current version. Context surrounding this feature can be found in #249

Delay in server response may lead to truncated data in client

Delays in the server response writes can make the client receive only truncated data, which can cause the client think there is no new updates available.
Tested on swupd v2.87 with a custom server.

Below the HTTPRequestHandler for a python HTTP Server for reproduction:

class SlowResponse(BaseHTTPServer.BaseHTTPRequestHandler):
    """Handler that returns data with a set delay between writes"""
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        response = "99990"
        delay = 0.00001 # seconds
        for c in response:
            self.wfile.write(c)
            sleep(delay)

Below are the responses observed for curl command and swupd check-update:

$ curl $IP:$PORT/version/format3/latest
99990
$ swupd check-update --url=$IP:$PORT
swupd-client software update checker 2.87
   Copyright (C) 2012-2015 Intel Corporation
   bsdiff portions Copyright Colin Percival, see COPYING file for details

There are no updates available

Failed to load 12390 xfce4-desktop manifest

Dear all,

Please let me report an issue related to swupd manifest of xfce4-desktop.

PHENOMENON

Can not install 'os-utils-gui' bundle due to 'Failed to load 12390 xfce4-desktop manifest'.

HOW TO RE-PRODUCE

• Newly install VirtualBox VDI from live image.
• Run:
  $ swupd update
  $ swupd bundle-add kernel-lts
  $ clr-boot-manager set-timeout 10
  $ clr-boot-manager update
  $ reboot #boot into 4.4 lts
  $ clr-boot-manager set-timeout 0
  $ clr-boot-manager update
  $ swupd bundle-add os-utils-gui os-core storage-utils

Result
swupd-client bundle adder 3.7.4
Copyright (C) 2012-2016 Intel Corporation
Attempting to download version string to memory
Failed to load 12390 xfce4-desktop manifest
Failed to load 12390 xfce4-desktop manifest
Failed to load 12390 xfce4-desktop manifest
Failed to load 12390 xfce4-desktop manifest
Unable to download manifest xfce4-desktop version 12390, exiting now

WORKAROUND

• Run:
  $ swupd search xfce4

Result
Downloading Clear Linux manifests
19.06 MB total...
Warning: Downloading missing manifest for bundle xfce4-desktop version 12390.
Cannot load official manifest MoM for version 12390
Completed manifests download.
... # xcfe4 search result as usual

$ swupd bundle-add os-utils-gui

Result
swupd-client bundle adder 3.7.4
Copyright (C) 2012-2016 Intel Corporation
Downloading packs...
Attempting to download version string to memory
Downloading os-utils-gui pack for version 12380
Extracting pack.
Downloading xfce4-desktop pack for version 12390
Extracting pack.
Downloading cryptography pack for version 12380
Extracting pack.
Installing bundle(s) files...
Calling post-update helper scripts.
Bundle(s) installation done.

Thank you.
Best regards,

Luck