Consider adopting package-url for external package references about spdx-spec

[pombredanne]

@wking getting out of this business for SPDX is exactly what I suggest with the purl and we have a good support and feedback from several package manager authors (dpkg, npm, go, maven and more)

[kestewart]

During tech team call today this was discussed.
There was general agreement to move this in as an optional field for 2.2, once the specification has been stabilized and adopted by the other stakeholders.
Plan will be to snapshot the PURL specification version, and add it as an optional field in the text of the SPDX specification. A table with the already defined package managers will be added.
There is some overlap with External References, but we will reconcile that in the 3.0 timeframe.

[zvr]

@pombredanne purl is a pretty well-known service for permanent URLs (purl.org), run by the Internet Archive.
Consider changing the name...

[wking]

On Thu, Dec 07, 2017 at 03:36:40PM -0800, Alexios Zavras (zvr) wrote: Consider changing the name...

It looks like Package URLs don't contain host information [1]. Perhaps they should be Package URIs (puri [2])? [1]: https://en.wikipedia.org/wiki/Puri_%28food%29 [2]: https://github.com/package-url/purl-spec/blame/d250235b02d0fba8a3f9a8977e979db43ceb33da/README.rst#L93-L95

[wking]

After reading over the Package URL spec a few times, my initial impression is that the real problem is that package-manager and version-control authors haven't gone through the trouble to register URI schemes (although there is a provisional registration for git). Both the current SPDX spec and the Package URL spec try to fill this gap by providing their own supplemental registry. But I think the simplest approach (for SPDX) would be to get out of this business, and leave it to the IANA and its existing registration procedure. Luckily, the external-repository appendix seems very weakly bound to the spec (only referenced from here?), so dropping it doesn't seem like it would be too difficult.

[pombredanne]

@wking you wote:

It looks like Package URLs don't contain host information [1]. Perhaps they should be Package URIs (puri [2])?

You can read the comments from URLs authorities in package-url/purl-spec#9 .... they have no objections there.

And FWIW, this is really a locator alright even though there is no "Authority" component.

[wking]

getting out of this business for SPDX is exactly what I suggest with the purl ...

So we agree on just dropping appendix VI and its one reference from the spec? Folks who want to use URIs to reference external packages can discover schemes (possibly eventually including purl) via the IANA registry, so we don't need to name-drop here.

[pombredanne]

I think that external references are a good thing of themselves, but detailing the way they should be specified as it has been done could be improved for sure.

[kestewart]

Key is going to be to make sure the references to packages are parseable and well specified - whats there has been understood to not be complete but a place to build from.

There are two main use cases for making external references - for security artifacts (CPEs, possibly SWIDs) and to package managers. There were some arguing strongly they should all just be external references, but my preference would be to call security out independently, and then align in with the purl/puri/.. for external references, as the package-url doesn't handle the CPEs. At any rate a good topic to revisit.

[wking]

… and then align in with the purl/puri/.. for external references…

But why would it be SPDX's job to register schemes? I've floated a potential approach in #54 that just makes these references URIs (preserving the category tagging). Then it's up to schema designers like package-url/purl-spec to get their schemes registered with the IANA (or to get their schemes into the SPDX ecosystem without IANA registration). I don't see a need for SPDX to be a second-tier URI-scheme registrar, with the IANA already filling that position for us.

[pombredanne]

re:

But why would it be SPDX's job to register schemes?

Did anyone say we would/should register schemes?

And FWIW, for purl we are honing on a single pkg: scheme, not multiple ones.

[wking]

Did anyone say we would/should register schemes?

Maybe I was reading tooo much into "align in with the purl/puri". So is #54 the direction we're all thinking about?

[kestewart]

We should discuss in tech call. It will require deprecating some features and splitting out security, but this is a better direction and will align with other open source projects handling these topics (Grafeas).

[pombredanne]

FYI, I presented purl at FOSDEM: https://fosdem.org/2018/schedule/event/purl/

[wking]

On Tue, Feb 20, 2018 at 06:44:45PM +0000, Kate Stewart wrote: There was general agreement to move this in as an optional field for 2.2, once the specification has been stabilized and adopted by the other stakeholders. Plan will be to snapshot the specification information, and add it as an optional field in the text of the SPDX specification. A table with the already defined package managers will be added.

Was there any discussion of the approach taken by #54 to resolve this issue? I don't see a need to define URI schemes for these external references, whether via the current External References or via package-url. Once you have ExternalRef taking a URI, you can punt to the IANA registry and/or unregistered URI-scheme specs, and there's no need for the SPDX to bless particular schemes.

[goneall]

From SPDX tech call 6/5/2018: Request to review PR #54 to see if this satisfies this issue.

[yevster]

I don't think we've reached consensus on abolishing "type", have we? Because that's what #54 seems to do.

[wking]

I don't think we've reached consensus on abolishing "type", have we?

I'm intentionally removing it there, because I don't think it adds information beyond what is already contained in the URI. For example this change:

-    ExternalRef: SECURITY cpe23Type cpe:2.3:a:pivotal_software:spring_framework:4.1.0:*:*:*:*:*:*:*
+    ExternalRef: SECURITY cpe:2.3:a:pivotal_software:spring_framework:4.1.0:*:*:*:*:*:*:*

has no need for a cpe23Type, because that information is already encoded in the URI's cpu:2.3: prefix. This would be a breaking change vs. the current approach, which is why #54 is tagged 3.0.

[yevster]

Type exists for a reason: to identify how to interpret the locator. Yes, you could conflate them into a single field, but if you wanted to, you could conflate the whole spec into a single field. The type and the locator have distinct meanings and hence they are distinct fields.

[wking]

Type exists for a reason: to identify how to interpret the locator.

I think <category> is distinct, which is why I kept it in #54. But I'm not clear on what <type> allows beyond what's already covered in the URI. In the CPE case, the consumer either knows how to resolve cpe scheme (defined here) or you don't. If you don't understand the cpe scheme, why would you care about distinguishing cpe22Type from cpe23Type?

For maven-central, npm, etc., those package managers can either define and register their own scheme, or folks can use package URLs. In all of those cases, consumes can switch on the scheme (e.g. pkg for package URLs) to decide if they understand them or not.

By requiring type qualifiers on top of the URI scheme, you basically break the well-supported URI approach and replace it with an SPDX specific typed-URI system. I think that's a lot of work (both for spec maintainers and tool maintainers) without an upside. Or is there an upside I'm missing?

[goneall]

Discussed on tech call 6-12-2018: For 2.2, we need to be compatible. Abolishing type would create an incompatibility. We should consider this for 3.0. For 2.2, we would like to support package-url so we will need a different mechanism. Will pick up the discussion once @wking joins the tech call. We would also like to have @pombredanne . Propose discussing on 26 June 2018.

[kestewart]

Can we use type of "PURL" for the ones coming from that domain , and then reconcile the overlap between whole type issue in 3.0?

[kestewart]

Single string as identier is ok by Philippe as well as parse-able.
PURL status: adoption progressing nicely on OWASP side. Contributed JAVA implementation. Has been adopted by RENOVATE (node based application and stack), RUST side has adopted.
Software Transperancy discussing PURL as identifiers. Discussion in ClearlyDefined but some of data structures and models - identification and how fetch/provision it (slightly different variant). ORT is also looking to add it. ScanCode/AboutCode tools use it.

[kestewart]

If we use External Identifiers, must have 1 and only 1 of Category, Type, idstring

Note: we need to update External Identifier TYPE to be case insesitive. Right now ambigous.

Options discussed inclusion with Type:
When = PACKAGE-MANAGER: PURL - no
When = PACKAGE-ID: PACKAGE-URL - ok
When = ID: PURL -ok
When = PURL: - no

Preference is to use ID PURL - - as its shorter, based on discussion in call.

Options would look like:
ExternalRef: SECURITY cpe23Type cpe:2.3:a:pivotal_software:spring_framework:4.1.0:::::::*
ExternalRef: PACKAGE-MANAGER MAVEN...
ExternalRef: ID PURL pkg:npm/%40angular/[email protected]
ExternalRef: OTHER LocationRef-acmeforge acmecorp/acmenator/4.1.3-alpha

Example PURLs:
pkg:maven/org.apache.xmlgraphics/[email protected]?packaging=sources pkg:maven/org.apache.xmlgraphics/[email protected]?repository_url=repo.spring.io/release pkg:npm/%40angular/[email protected]
pkg:npm/[email protected]

@pombredanne, @kestewart, @tsteenbe, @yevster, @goneall were in meeting for discussion and agree with above conclusion.

[kestewart]

Merged #152