isManifestUnknownError fails against Harbor registries, breaking sigstore signature upload about image

[mtrmac]

Thanks for your report.

As you see, the code is trying to copy signatures. This is supposed to work.

AFAICS the issue here is interoperability with the specific destination registry implementation; a signed push with a sigstore signature to that registry would fail just the same.

---

First, please make sure you are using the recently-released Skopeo 1.14.0; the behavior in this area has changed, quite possibly fixing this.

If that’s not the case, we’ll need to fix that in c/image; for starters, please provide a full log from skopeo --debug copy …, and there might be more debugging information necessary.

[Plat0nicS0lid]

Hello,
tried the skopeo version within arch linux docker image.
Please see the skopeo -v output:
[root@6bbacdc9b6e4 ~]# skopeo -v
skopeo version 1.14.0 commit: 6abf96bb82666fbb3d4ad9faf1812e5ae2d31a74
[root@6bbacdc9b6e4 ~]#

To give you some more information about the environment. Both source and destination are harbor registries - please see goharbor.io if you need more information about the product.
Harbor is able to handle cosign signed images per default. As stated in my first entry, using the internal image replication leads to the result that signed images within the source registry will be copied together with the signature to the target registry. But for our use case, using the graphical tool is not an option.

The "test:0.1" image is a very simple image only containing an alpine image with a simple echo hello line as cmd. That image has been signed with cosign and a local self created key. The image has been uploaded to registry 1 into the project "images".
That is the starting situation.

Please see the following sysouts. First one copying the image into a local dir into the running container that provides skopeo version 1.14.0:

---snip---

[root@6bbacdc9b6e4 ~]# skopeo copy docker://192.168.a.b/images/test:0.1 dir:test_0-1
Getting image source signatures
Checking if image destination supports signatures
Copying blob 5b088f1e648c done |
Copying blob 96526aa774ef done |
Copying config 33b8df73a9 done |
Writing manifest to image destination
Storing signatures
[root@6bbacdc9b6e4 ~]# cd test_0-1/
[root@6bbacdc9b6e4 test_0-1]# ls -la
total 49052
drwxr-xr-x 2 root root 4096 Nov 29 21:29 .
drwxr-x--- 1 root root 4096 Nov 29 21:29 ..
-rw-r--r-- 1 root root 1127 Nov 29 21:29 33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
-rw-r--r-- 1 root root 46789009 Nov 29 21:29 5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
-rw-r--r-- 1 root root 3401967 Nov 29 21:29 96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
-rw-r--r-- 1 root root 740 Nov 29 21:29 manifest.json
-rw-r--r-- 1 root root 1629 Nov 29 21:29 signature-1
-rw-r--r-- 1 root root 33 Nov 29 21:29 version
[root@6bbacdc9b6e4 test_0-1]#

[root@6bbacdc9b6e4 ~]# skopeo --debug copy dir:test_0-1 docker://192.168.y.z/images/test:0.1
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf"
DEBU[0000] Found credentials for 192.168.y.z/images/test in credential helper containers-auth.json in file /run/containers/0/auth.json
DEBU[0000] Lookaside configuration: using "docker" namespace 192.168.y.z
DEBU[0000] No signature storage configuration found for 192.168.y.z/images/test:0.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.y.z
DEBU[0000] Sigstore attachments: using "docker" namespace 192.168.y.z
DEBU[0000] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0000] IsRunningImageAllowed for image dir:/root/test_0-1
DEBU[0000] Using default policy section
DEBU[0000] Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
Checking if image destination supports signatures
DEBU[0000] GET https://192.168.y.z/v2/
DEBU[0000] Ping https://192.168.y.z/v2/ err Get "https://192.168.y.z/v2/": dial tcp 192.168.y.z:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.y.z/v2/", Err:(*net.OpError)(0xc0000403c0)})
DEBU[0000] GET http://192.168.y.z/v2/
DEBU[0000] Ping http://192.168.y.z/v2/ status 401
DEBU[0000] We can't modify the manifest, hoping for the best...
DEBU[0000] Checking if we can reuse blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0000] Checking /v2/images/test/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0000] GET http://nuc04/service/token?account=admin&scope=repository%3Aimages%2Ftest%3Apull%2Cpush&service=harbor-registry
DEBU[0000] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0000] ... already exists
DEBU[0000] Skipping blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa (already present):
DEBU[0000] Checking if we can reuse blob sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0000] Checking /v2/images/test/blobs/sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
DEBU[0000] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
Copying blob 96526aa774ef skipped: already exists
DEBU[0000] ... already exists
Copying blob 96526aa774ef skipped: already exists
Copying blob 5b088f1e648c skipped: already exists
DEBU[0000] No compression detected
DEBU[0000] Compression change for blob sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2 ("application/vnd.docker.container.image.v1+json") not supported
DEBU[0000] Using original blob without modification
DEBU[0000] Checking /v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0000] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0000] ... already exists
Copying config 33b8df73a9 done |
Writing manifest to image destination
DEBU[0000] PUT http://192.168.y.z/v2/images/test/manifests/0.1
Storing signatures
DEBU[0000] Looking for sigstore attachments in 192.168.y.z/images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0000] GET http://192.168.y.z/v2/images/test/manifests/sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0000] Content-Type from manifest GET is "application/json; charset=utf-8"
DEBU[0000] Fetching sigstore attachment manifest failed: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found
FATA[0000] writing signatures: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found
[root@6bbacdc9b6e4 ~]#

---snap---

If I understand the last line correctly, instead of "signature-1" a file named sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig will be expected - is this correct?

Second one showing sysout copying the image directly between harbor 1 and harbor 2:

---snip---

[root@6bbacdc9b6e4 ~]# skopeo --debug copy docker://192.168.a.b/images/test:0.1 docker://192.168.y.z/images/test:0.1
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf"
DEBU[0000] Found credentials for 192.168.y.z/images/test in credential helper containers-auth.json in file /run/containers/0/auth.json
DEBU[0000] Lookaside configuration: using "docker" namespace 192.168.y.z
DEBU[0000] No signature storage configuration found for 192.168.y.z/images/test:0.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.y.z
DEBU[0000] Sigstore attachments: using "docker" namespace 192.168.y.z
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "192.168.a.b/images/test:0.1"
DEBU[0000] Found credentials for 192.168.a.b/images/test in credential helper containers-auth.json in file /run/containers/0/auth.json
DEBU[0000] Lookaside configuration: using "docker" namespace 192.168.a.b
DEBU[0000] No signature storage configuration found for 192.168.a.b/images/test:0.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.a.b
DEBU[0000] Sigstore attachments: using "docker" namespace 192.168.a.b
DEBU[0000] GET https://192.168.a.b/v2/
DEBU[0001] Ping https://192.168.a.b/v2/ err Get "https://192.168.a.b/v2/": dial tcp 192.168.a.b:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.a.b/v2/", Err:(*net.OpError)(0xc000578780)})
DEBU[0001] GET http://192.168.a.b/v2/
DEBU[0001] Ping http://192.168.a.b/v2/ status 401
DEBU[0001] GET http://nuc02/service/token?account=admin&scope=repository%3Aimages%2Ftest%3Apull&service=harbor-registry
DEBU[0001] GET http://192.168.a.b/v2/images/test/manifests/0.1
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0001] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0001] IsRunningImageAllowed for image docker:192.168.a.b/images/test:0.1
DEBU[0001] Using default policy section
DEBU[0001] Requirement 0: allowed
DEBU[0001] Overall: allowed
Getting image source signatures
DEBU[0001] Reading /var/lib/containers/sigstore/images/test@sha256=3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db/signature-1
DEBU[0001] Looking for sigstore attachments in 192.168.a.b/images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0001] GET http://192.168.a.b/v2/images/test/manifests/sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0001] Found a sigstore attachment manifest with 1 layers
DEBU[0001] Fetching sigstore attachment 1/1: sha256:73d5db8bd77407d1cc1863c88fb584f3b658807e934c41d9b9f1edd8166abae0
DEBU[0001] Downloading /v2/images/test/blobs/sha256:73d5db8bd77407d1cc1863c88fb584f3b658807e934c41d9b9f1edd8166abae0
DEBU[0001] GET http://192.168.a.b/v2/images/test/blobs/sha256:73d5db8bd77407d1cc1863c88fb584f3b658807e934c41d9b9f1edd8166abae0
Checking if image destination supports signatures
DEBU[0001] GET https://192.168.y.z/v2/
DEBU[0002] Ping https://192.168.y.z/v2/ err Get "https://192.168.y.z/v2/": dial tcp 192.168.y.z:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.y.z/v2/", Err:(*net.OpError)(0xc000578050)})
DEBU[0002] GET http://192.168.y.z/v2/
DEBU[0002] Ping http://192.168.y.z/v2/ status 401
DEBU[0002] We can't modify the manifest, hoping for the best...
DEBU[0002] Checking if we can reuse blob sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0002] Checking /v2/images/test/blobs/sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
DEBU[0002] GET http://nuc04/service/token?account=admin&scope=repository%3Aimages%2Ftest%3Apull%2Cpush&service=harbor-registry
DEBU[0002] Checking if we can reuse blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0002] Checking /v2/images/test/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0002] GET http://nuc04/service/token?account=admin&scope=repository%3Aimages%2Ftest%3Apull%2Cpush&service=harbor-registry
DEBU[0002] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0002] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
DEBU[0002] ... already exists
DEBU[0002] ... already exists
DEBU[0002] Skipping blob sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef (already present):
Copying blob 5b088f1e648c skipped: already exists
Copying blob 5b088f1e648c skipped: already exists
Copying blob 96526aa774ef skipped: already exists
DEBU[0002] Downloading /v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0002] GET http://192.168.a.b/v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0002] No compression detected
DEBU[0002] Compression change for blob sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2 ("application/vnd.docker.container.image.v1+json") not supported
DEBU[0002] Using original blob without modification
DEBU[0002] Checking /v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0002] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0002] ... already exists
Copying config 33b8df73a9 done |
Writing manifest to image destination
DEBU[0002] PUT http://192.168.y.z/v2/images/test/manifests/0.1
Storing signatures
DEBU[0002] Looking for sigstore attachments in 192.168.y.z/images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0002] GET http://192.168.y.z/v2/images/test/manifests/sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0002] Content-Type from manifest GET is "application/json; charset=utf-8"
DEBU[0002] Fetching sigstore attachment manifest failed: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found
FATA[0002] writing signatures: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found
[root@6bbacdc9b6e4 ~]#

---snap---

Second attempt showing the same result.

Added the following lines to /etc/containers/registries.d/default.yaml:
docker:
192.168.a.b:
use-sigstore-attachments: true
192.168.y.z:
use-sigstore-attachments: true

Hope that helps. Please let me know if any further information is needed.

[mtrmac]

Thanks!

Storing signatures
DEBU[0000] Looking for sigstore attachments in 192.168.y.z/images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0000] GET http://192.168.y.z/v2/images/test/manifests/sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig 
DEBU[0000] Content-Type from manifest GET is "application/json; charset=utf-8"
DEBU[0000] Fetching sigstore attachment manifest failed: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found 

OK, the registry (ref. https://github.com/goharbor/harbor/blob/7cef4217b03db1cb61d9fabded22540fafbff4b2/src/controller/artifact/controller.go#L298) apparently uses a non-standard "code" value NOT_FOUND: https://github.com/goharbor/harbor/blob/7cef4217b03db1cb61d9fabded22540fafbff4b2/src/lib/errors/const.go#L19 (and that turns into unknown) but that value should turn into a 404, looking at https://github.com/goharbor/harbor/blob/7cef4217b03db1cb61d9fabded22540fafbff4b2/src/lib/http/error.go#L39.

If I understand the last line correctly, instead of "signature-1" a file named sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig will be expected - is this correct?

No, that name refers to a tag on the destination registry (and a component of the URLs quoted above). That works as expected.

---

Either way, we’ll want to capture the full HTTP response of the above-quoted HTTP request (with the relevant authentication headers). Is that easy to do for you in the current environment? If not, we’ll need a source code patch to capture that in Skopeo.

[mtrmac]

… looking more closely (and compare goharbor/harbor#19179 , not the same request but a sample of the error message):

• The registry returns a 404, as requested
• The registry chooses to return JSON, using the standard format per https://github.com/opencontainers/distribution-spec/blob/main/spec.md#error-codes . But it violates the spec in not using the right "code" value.

our isManifestUknown can, currently, only check for status 404 if the registry does not use the standard JSON format; if the standard JSON format is used, the HTTP status value is not currently available in isManifestUnknown.

[Plat0nicS0lid]

Hello again,
many thanks for your help and looking for the cause. Apologies for the delay and not answered earlier. May I ask for some guidance how to deliver the full http response? If this is something a sopeo option handles, could you please tell me how to manage it? Or is this something that can be recorded with a running tcpdump, or similar? If yes, which options have I to use, to record the necessary information for you?

If I understand you correctly, the situation is that a not expected response will be returned to prevent the signature to be delivered to harbor. Is this something that can be shipped around inside skopeo or must this be changed on harbor side? As the image copy works between two harbor registries directly, also if the source image has a signature, the functionality is available in general.
Just to see the copy result between two registries working if copying with skopeo, could you help me out with a registry product which would work? Basically, there will be no option to switch to another product than harbor, but it is my personal interest to see that process working in general.