Rawspeed considered as copylib about rawspeed

[klauspost]

Good - I am not quite sure what you are asking, since I don't know your terminology.

1. If upstream is me, then there is no formal releases, since there are many updates for new cameras. I encourage that people who use Rawspeed do manual updates from GIT, so they pin camera support for their own specific releases, and can test it consistently.

2. There is no separate libraries, source code is intended to be supplied in source/compiled into releases of programs using it.

I guess that makes it your definition of a "copylib".

[Germano0]

Your reply sounds very good to me, I just replied in the comment https://fedorahosted.org/fpc/ticket/550#comment:2
Thank you very much!

[klauspost]

No problem. Feel free to reopen if you have further questions.

[rathann]

Hello, I'm a member of the Fedora Packaging Committee. Could you please answer our standard set of questions: https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries#Standard_questions ? Please also read the beginning of that page for some arguments against bundling.

Would you consider making a shared library with regular formal releases? This would make packager's life a lot easier, even if there are API/ABI breaks (just please take care of bumping the version properly). Also, nobody is asking you to support older releases (though it would be nice), that is the job of distribution package maintainers.

According to comments in http://redmine.darktable.org/issues/10582 , darktable bundles an active fork of rawspeed. Are you aware of their fork? What is your relation to the darktable project?

[klauspost]

Regarding the "standard questions", they seem to be directed at darktable/rawstudio/other users. It doesn't seem like something I should answer.

Would you consider making a shared library with regular formal releases? This would make packager's life a lot easier, even if there are API/ABI breaks (just please take care of bumping the version properly).

The library does not have a C-interface, and since C++ libraries aren't ABI compatible that would have to be made. I am open to a PR, but I have no real motivation myself.

This would make packager's life a lot easier

How would you test if a library update breaks something in the user application? Camera support is updated on a regular basis, and to support it both this library and the application using the library would be updated in sync. I don't see any significant gain in breaking that link.

darktable bundles an active fork of rawspeed. Are you aware of their fork?

Yes, and they push it upstream on a regular basis. We have a very good collaboration going. AFAIK the only forked parts are disabling cameras that RawSpeed supports but Darktable doesn't.

[rathann]

Thank you for the answer. It's encouraging. Could you explain why camera support needs to be updated both in rawspeed and any application using it? Forgive me for missing something obvious (I have limited knowledge of the subject), but doesn't that mean that every application using rawspeed is reimplementing the same code and such code should be moved into the library itself?

[pedrocr]

Supporting a camera requires more than just being able to read its raw format. rawspeed is great at that part and then other things need to be put in place (a color matrix, some wb presets, noise profile, etc). Here's how we handle it for darktable:

http://www.darktable.org/resources/camera-support/

That doesn't mean rawspeed couldn't be made into a standalone lib (like exiv2 or lensfun) with proper versioning. The camera definitions file is even input programatically so we could still ship our own to be used by the system lib. The issue is not insurmountable but I don't really see the value right now. There are two main users of rawspeed (rawstudio and darktable) and they both need different versions at the moment. Even if someone did the work of building frequent rawspeed releases (a big if) you'd still need librawspeed1 and librawspeed2 packages to handle the differences so you'd get no actual gain.

[rathann]

Thank you very much for the detailed answer. This makes the situation much clearer and - if presented in the request for bundling exception ticket - would've earned a vote for a temporary exception from me and probably other FPC members as well. I'm happy that you're open to the idea of sharing the library between the two main consumers eventually. Given that all three projects involved here are in active development, I'm not too worried about your handling of security issues in rawspeed itself, and a temporary (let's say 2 Fedora releases ~= 1 year) bundling exception could get a majority vote in favour. The issue would be revisited in 1 year and we would see where the three projects stand.

However, I am worried about darktable bundling libRaw and having to fix the same security issues, i.e.:
http://www.darktable.org/2015/06/released-darktable-1-6-7/
and
LibRaw/LibRaw@4606c28 .

[LebedevRI]

@rathann
This is no longer true for our current development version (what will be 2.0 release):

• darktable-org/darktable#839
• darktable-org/darktable@ae65ddf
• darktable-org/darktable@3f94757
• darktable-org/darktable@bf9d048

[rathann]

That is excellent news. I applaud your unbundling efforts. Now, if the same could be done with rawspeed in coordination with both rawspeed and rawstudio upstreams, we wouldn't need any bundling exceptions and any security issues in rawspeed would need only one update: rawspeed itself. I'm keeping my fingers crossed that you get to that point sooner rather than later.

[pmjdebruijn]

There are release engineering issues with linking to an external version of rawspeed. We pull in fixes sometimes days before a release, and we release almost once a month these days. Having to coordinate that would either slow us significantly down, or come with a ridiculous overhead. New cameras are released monthly as well, and the release pace only seems to be speeding up. Therefore our ability to quickly respond to the market will be more important than ever.

Also since darktable is non-destructive (all edits are kept as parameterized operations), each view of an image is actually those parameterized operations being applied to the raw image that is being decoded live. So any rawspeed version mismatch (both newer and older) could potentially cause problems (which might be construed as data-loss for end-users).

Another concern is the fact that "camera support" is an application property, not an operating system property. Each RAW converter project makes their own claims about which version of their software supports which camera in what way. Irrelevant of where and how an application is shipped there should never be discrepancy in that.

Thus, I don't think rawspeed should ever be externalised from our codebase.

That said, I do think rawspeed is pretty much the only valid (permanent) exception, and as others have already noted, we've been working for some time now to get steadily rid of all other copylib cases.

[rathann]

As I said, I support a a temporary bundling exception for both rawstudio and darktable to bundle rawspeed. However, this cannot be a permanent one. The library is too big in my opinion (and other members of Fedora Packaging Committee share this view). I've just reopened the exception request ticket and we'll reconsider it at the next meeting.

Regarding release engineering issues, if you're worried about release speed of rawspeed upstream, then you should offer to help there instead of keeping your own fork (and the same goes for rawstudio). It's better to focus efforts in one place instead of dividing them and duplicating work. Fedora maintainers should have no issue with updating all three packages at the same time or keeping a specific snapshot of each.

I am surprised that it's not possible to use a rawspeed version that supports more cameras than current darktable (or indeed rawstudio). A library should be backwards compatible. What you're saying tells me that there are design (API?) issues to be worked out, hence my support for a temporary exception. Both @klauspost 's and @pedrocr 's comments tell me that there's some will to work this out among the developers, which is encouraging.

[pedrocr]

@rathann most issues preventing this are surpassable but there is no one available to do the work and I doubt there will be in the near future. All the interactions we've had with Fedora have been quite pushy. Feel free to not package darktable in Fedora if the way we build the software doesn't match your requirements.

[rathann]

@pedrocr I'm very sorry if you feel you're being pushed by us. That was not the intention at all, but some of the previous communications on this issue could convey such impression. I believe there's a slight language barrier at work here. You guys are upstream. We want to work with you, not order you around. That is one of our four foundations ("Friends"). We can only ask and explain our point of view. In cases where upstream doesn't want to follow our rules for any reason, it's up to the package maintainer to do the work necessary to achieve compliance. Of course, (in our opinion) it's best for everyone if the work gets merged upstream, because then everybody benefits from it.

[klauspost]

@rathann - If that is your intention, you should definitely work on your communication. Start by looking at phrases like "[...]cases where upstream doesn't want to follow our rules for any reason, it's up to the package maintainer to do the work necessary to achieve compliance".

You may have some corporate compliance guidelines you need to adhere to, but making it "our" problem by referencing some internal guidelines doesn't make it any clearer for us why or what you are trying to accomplish, and frankly it is quite demotivating given your way of communicating.

I don't want to hurt developers using my library, and given that I don't work on Linux as my main platform I would not be able to supply things like a build system (and all that follows) that I would feel comfortable with.

[pedrocr]

@rathann, your words were: "if you're worried about release speed of rawspeed upstream, then you should offer to help there instead of keeping your own fork". As klaus mentioned our working relationship with rawspeed is excellent. With the little resources we have we've been able to increase the rawspeed camera support to a point where libraw is no longer needed to support all relevant raw formats. We also don't keep much of a fork, everything we do gets submitted upstream, there is currently nothing in darktable that isn't also either already merged or submitted as a pull request. This is in fact a much better situation than what happens with lensfun and exiv2 where we depend on their infrequent releases and can't really control what the users have installed so end up having to deal with all the "my lens isn't supported in darktable" bug reports that we can't actually fix.

But hey, this is open-source so if someone would like to work on making rawspeed a linkable lib maybe klaus would consider a PR for that. Just don't expect any of the current developers to work on it, we're too busy making stuff actually work.

[rathann]

@klauspost Again, sorry if I came across the wrong way. Please forgive me, I will try to improve.

Our packaging guidelines are not "corporate", they are a collection of packaging best practices established as a result of Fedora community experience. Other distributions have similar guidelines. I'm not sure why you perceive what I wrote as demotivating (could you explain?). The link to the guidelines was posted in the opening post, but let me repost: https://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries . It explains the why and what pretty exhaustively, I think. Of course, I am biased because I helped write some of them and I apply them instinctively when packaging various projects, but basically, the prohibition on bundling is there for several reasons:

• security - we can fix the vulnerabilities in one place, the shared library, and every package depending on it is not vulnerable anymore,
• forking - if everyone is bundling their own fork, it gets harder to fix bugs, including security vulnerabilities and it creates more work for every maintainer involved as the forks diverge,
• general bug fixes and features - every package depending on a shared library gets the benefits of bugfixes and new features instantly (and we have the option to package a parallel-installable older version if a depending package can't be updated/fixed to work with the latest library version)

Since, as @pedrocr wrote, you are busy making stuff actually work, it falls on our package maintainer to work on this if they can. If they're unable to at the moment, we can grant a temporary bundling exception (and given the healthy state of all three projects, I don't see any reason not to). We may make this a permanent one if no progress is made, but we'd really prefer not to.

Again, nobody is trying to force you (the upstream developers) to comply to our guidelines. Compliance is our package maintainer's responsibility. We just think it's in everyone's best interest if rawspeed is a shared library in the long term.

[boucman]

@rathann please understand that the different libs mentionned here should not be considered "a bunch of code" but more like "hardware descriptors that happen to be described in C" from a packaging point of view.

there are new lenses/cameras every month and they need synchronized updates to both rawspeed and darktable/rawstudio. That means that a given version of darktable will only work with a given version of rawspeed.

This is not something that can be solved with lib versioning. Lib versioning is not meant to solve that sort of problem.

If we were to "unbundle" fedora would need four packages : darktable, rawspeed-darktable, rawstudio, rawspeed-rawstudi. There is no way around that.

You would gain nothing as far as security is concerned (since you would either use darktable as the source of rawspeed-darktable or have to track the very exact version of rawspeed that DT need and backport security fixes, which DT would have backported faster than you)

You will gain nothing as far as forking is concerned since (as has been pointed multiple times) Users usually don't send new camera data to rawspeed but to the final software they use (DT or rawstudio) and we are the ones doing the testing and upstreaming. Our relations are really good and we upstream everrything that needs upstreaming. Unbundling might endanger that healthy ecosystem

and as far as bug fixes and features, both DT and rawstudio follow rawspeed very closely so there would be very little gain. Theoretically faster cross-polination between rawstudio and DT, but in practice that's not even true since both DT and rawstudio follow the git version of rawspeed, so whenever one of us pushes something upstream, the other one gets it right away.

So please change your perception about what is rawspeed. Unbundling rawspeed for Fedora will result in broken DT/rawstudio with less hardware support than the corresponding upstream release and would force us to provide undesired advices to bug reports from fedora users (please do not use the packaged version, recompile the version provided by DT instead) That would be bad for everybody.

[Germano0]

https://fedorahosted.org/fpc/ticket/550#comment:6

[pedrocr]

@Germano0 good, it's resolved, so maybe we can just go back to building software...

[sergiomb2]

Hello, First thanks for develop rawspeed , I'd like to return to this subject in a new and fresh approach, I haven't read all that was wrote in this ticket .
Try to see Fedora packaging guidelines as an attempt to organize things and if we follow we can have a better and organized Linux.
"AFAIK the only forked parts are disabling cameras that RawSpeed supports but Darktable doesn't." where it is disabled ?
"The library does not have a C-interface, and since C++ libraries aren't ABI compatible that would have to be made."
But have an API , @abrander from rawstudio , reply to me that "maybe" I can update rawspeed in rawstudio [1](if API doesn't change) "no promises about API compatibility" .
Make a tag when the API changes, could help us a lot , of course if the API is not always changing or under a big development.
Practical questions , where is defined API ? , since 8ea2a3a API changed or can I update rawspeed in rawstudio ?

[1] rawstudio/rawstudio#6

Thanks,