Build failure with network 2.5 about http

[hsenag]

I think you're right about old versions, though the base dependency has been around for longer and so over time would rule out the versions like 4000.2.4 in practical use cases.

I inherited the package without any bounds and haven't got round to filling them all in yet - I think I added network because of the breaking change you ran into above

I'd be happy to deprecate the old versions but the "edit package information" link on hackage isn't working for me so I'm not sure how to.

[tibbe]

Well, the cause is that HTTP used to not have an upper bound on network, asserting that it would work with all future versions of network, which is something that clearly can't be known.

This breakages shows the danger of only adding upper bounds when you find that a new version of one of your dependencies causes failures: cabal will happily use an older version with incorrect bounds if that leads to a working build plan. What's even worse, even if HTTP bumps its dependency on network now, cabal is still allowed to pick the old version of HTTP, and it will do so in some cases (e.g. it will prefer installed versions.)

[snoyberg]

Not disagreeing with you @tibbe, but to me this is a great example of theory vs practice. In theory, PVP will keep code building forever. In practice, as soon as someone makes a mistake once, the entire house of cards falls down. Just sayin' ;)

[tibbe]

I disagree, this shows what happens if people don't follow the PVP (which is what actually happened here). We get suffering and we end up in a state that's hard to fix (because we cannot update old versions).

[hsenag]

Practically, if I can just deprecate the older versions now, the problem will go away :-) If hackage doesn't yet support this directly, can I can ask the admins to set a lower bound?

[There are also missing upper bounds on other libraries like parsec that I ought to fix and hope nothing breaks before the recent versions become irrelevant]

[snoyberg]

Just for the record: I'm -1 on adding more upper bounds to other packages, like parsec. I think they're more likely to cause breakage than anything else, such as this circumstance.

[tibbe]

Just for the record: I'm -1 on adding more upper bounds to other packages, like parsec. I think they're more likely to cause breakage than anything else, such as this circumstance.

I disagree. If we don't add them now we're guaranteed to one day in the future have a breakage just like this one, which we cannot properly fix because even if we add the upper bound then cabal will pick the older versions.

If we add an upper bound today that might still happen, but all versions after the next one will be usable and won't have to be blacklisted, just as all old HTTP versions now have to be blacklisted.

[snoyberg]

If blacklisting is reliable, I don't have a problem with this. If blacklisting isn't reliable, and can't be made reliable, then we'll have a repeat of this issue forever. Imagine:

• parsec 2 allows all versions of bytestring
• parsec 2.1 adds an upper bound on bytestring
• parsec 5 adds support for bytestring 2.0 which introduces major breaking changes
• bytestring 2.1 is released
• User tries to install parsec, and gets parsec 2, which misses all the fixes in parsec 5

[tibbe]

Lets move this discussion off the bug tracker, so we avoid spamming poor @hsenag.

[hvr]

Just for the record, this issue (i.e. suffering from past "laxness") came up in #47 recently. Otoh, haskell/hackage-server#52 which might help a bit here to fix past mistakes is still work in progress.

[sethfowler]

I think it's worth noting that people are hitting this when trying to update cabal-install. Because of that, this has broken Travis CI Haskell tests, at least for me, and I saw at least one case of this issue confusing a new Haskell user in #haskell.

[hsenag]

This is specifically the old broken HTTP version that they're hitting? Should I ask the hackage admins for manual blacklisting, and if so of what?

I'm just preparing a version that should have tight upper bounds for everything and keep with fairly loose lower bounds (builds with GHC 6.10 and up), so we could in theory blacklist everything before that one, but I don't know how that would play with the Platform having got the old versions.

[sethfowler]

Yup, it's the old broken version. Here's an example failure from Travis.

[snoyberg]

Simply releasing a new version with a relaxed network upper bound will solve the immediate problem. I really think we need to not let the perfect get in the way of the good: this is currently causing breakage in many places, and a simply one-line change will fix that breakage.

[tibbe]

I've added a preferred versions constraint on the package, which will make cabal use >= 4000.2.5 as long as that's possible to construct a valid install plan.

Releasing a new version of HTTP only solves the problem for people that allow that new version. This solution should solve it for everyone.

[snoyberg]

I'm confused: who wouldn't allow a new version of HTTP in this case? Unless I'm mistaken, the new version number would be 4000.2.12.1 (or 4000.2.13 if you really wanted), so no one should have an upper bound disallowing its usage.

I'm +1 on the version constraint btw, I don't see a downside to it, but a new version on Hackage would be even better, since it would also avoid bifurcation.

[hsenag]

That's great, thanks. If anyone is still getting broken installs please let me know and I'll rush out the bumped version as snoyberg suggests, otherwise I'd rather clean this up properly.

[tibbe]

@snoyberg Anyone who has an upper bound on HTTP that's lower than 4000.2.5. More likely they have an upper bound of < 4000.2 so those users are screwed either way.

[hsenag]

FWIW I can't build the tests with network 2.5.0.0 because httpd-shed has a rather similar problem with an old non-PVP compliant version and a new unbumped version - though I think a real code change may be needed too.

@snoyberg - did you manage to build or run the tests with the bump?

[snoyberg]

httpd-shed just needs a relaxed upper bound in the cabal file, nothing else. So it's in the exact same situation as HTTP.

I'm currently getting a compile error when running the HTTP test suite, but that appears to be due to a different bug in the cabal file...

[snoyberg]

First pull request sent to httpd-shed: andygill/httpd-shed#1. Now working on the problems in HTTP.

[snoyberg]

OK, pull request #56 sent.

[snoyberg]

This issue is continuing to cause me issues when running my test suites. Is there any ETA on resolving it, given that the fix is a one-line change?

[tibbe]

@snoyberg the old HTTP versions should be blacklisted on Hackage since a while ago, are you seeing old HTTP versions be picked up by cabal install still?

[snoyberg]

Yes, it just happened to me. I'm using cabal-install 1.16.0.2, not sure if respecting of blacklisting was added later or not.

[snoyberg]

More information: I was trying to run the Warp test suite, and for some reason cabal selected network-2.5.0.0 and HTTP-4000.2.4.

[tibbe]

Can you post the output the dependency solver part of the cabal install -v3 output, please? Note that the blacklisting only works if there's a way to come up with a build plan that doesn't include the blacklisted versions. If you're forcing network-2.5 (do you need the new feature in that version?) and older HTTP version can still be picked.

@hsenag will you make a release including the version bump please?

[snoyberg]

I believe this behavior occurs if I have some of the dependencies already installed against network 2.5. I've posted the full output at:

https://gist.github.com/snoyberg/11111877

This is the output for the command cabal install -v3 --enable-tests --force-reinstalls inside the warp directory, with an existing warp built against network-2.5.0.0. The most relevant parts of the output seem to be:

[_31] rejecting: HTTP-4000.2.12/installed-7df... (conflict: network==2.5.0.0/installed-80f..., HTTP => network==2.4.2.3/installed-907...)
[_31] rejecting: HTTP-4000.2.12, 4000.2.11, 4000.2.10, 4000.2.9, 4000.2.8, 4000.2.7, 4000.2.6 (conflict: network==2.5.0.0/installed-80f..., HTTP => network<2.5)
[_31] trying: HTTP-4000.2.5
[_32] trying: HTTP-4000.2.5:-warn-as-error
[_33] trying: HTTP-4000.2.5:-old-base
[_34] rejecting: HTTP-4000.2.5:-network23 (conflict: network==2.5.0.0/installed-80f..., HTTP-4000.2.5:network23 => network>=2.4 && <2.5)
[_34] rejecting: HTTP-4000.2.5:+network23 (conflict: network==2.5.0.0/installed-80f..., HTTP-4000.2.5:network23 => network<2.4)
[_32] fail (backjumping, conflict set: HTTP, network, warp, HTTP-4000.2.5:network23, warp-2.1.4.1:test)
[_31] trying: HTTP-4000.2.4
[_32] trying: HTTP-4000.2.4:-warn-as-error
[_33] trying: HTTP-4000.2.4:-old-base
[_34] trying: HTTP-4000.2.4:-mtl1
[_35] trying: HTTP-4000.2.4:!test

I can move this to a cabal issue if you prefer, but as far as this issue goes, I think it's clear that we can't rely on blacklisting solving dependency problems.

[snoyberg]

If I unregister network-2.5.0.0 and keep 2.4.x, then everything works. However, if I start with a completely empty user package database with no version of network installed at all, it still selects network-2.5.0.0 and HTTP-4000.2.4. So it seems like the blacklisting is not having any impact at all on cabal-install. I've tested this with both cabal-install 1.16 and 1.18.

[tibbe]

@snoyberg that's interesting. Could you cabal update to make sure you have the latest blacklist? If this happens with a completely empty user package DB we should file a bug against cabal (something like: blacklist being ignored) with the solver output from -v3.

[snoyberg]

OK, here's where blacklisting seems to break down. Starting with a clean user package database, GHC 7.6.3, and cabal-install 1.16, I get the following:

• cabal install HTTP --dry-run indicates that it will install network-2.4.2.3 and HTTP-4000.2.12. Good.
• cabal install HTTP warp --dry-run installs those versions too. Again good.
• cabal install warp --dry-run uses network-2.5.0.0 and does not include HTTP (HTTP is only used by the test suite). Once again, good.
• cabal install warp --dry-run --enable-tests still uses network-2.5.0.0 and HTTP-4000.2.4. This is the problem.

tl;dr: blacklisting does not seem to apply to the dependencies of the package you're installing.

[snoyberg]

Sure, I'll put this information into the cabal issue tracker.

[tibbe]

Thanks!

On Sun, Apr 20, 2014 at 1:48 PM, Michael Snoyman
[email protected]:

Sure, I'll put this information into the cabal issue tracker.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/55#issuecomment-40893292
.

[snoyberg]

Issue open: haskell/cabal#1792. I'll restrict further comments on this thread to the HTTP dependency issue itself.

And yes, I ran cabal update to make certain I had the newest blacklist.

[hvr]

@hsenag fyi, adapting version bounds has been enabled as of today. So you can try now to carefully edit wrong package meta-data to set more accurate upper bounds for HTTP's past mistakes :-)

PS: I just noticed @dcoutts already fixed up the meta-data for HTTP-4000.2.4