Can Paket resolve dependencies across multiple sources? about paket

[forki]

source "http://nuget.org/api/v2"

nuget "Castle.Windsor-log4net" "~> 3.2"

source "http://nuget.org/api/v3"
nuget "Rx-Main" "~> 2.0"

will result in something like

NUGET
  remote: http://nuget.org/api/v2
  specs:
    Castle.Windsor (2.1)
    Castle.Windsor-log4net (3.3)
      Castle.Windsor (>= 2.0)
      log4net (>= 1.0)
    log (1.2)
    log4net (1.1)
      log (>= 1.0)
  remote: http://nuget.org/api/v3
  specs:
    Rx-Core (2.1)
    Rx-Main (2.0)
      Rx-Core (>= 2.1)

So if we start from on source we keep it.

[mavnn]

And should it be able to?

[forki]

do you mean indirect dependencies? then no it cannot.

[ilkerde]

I think it can be fairly dangerous (and tricky) to support cross-source dependencies. What if the same package appears in both? How to scope this (if even more than 2 sources exist)? How to deal with transitive dependencies from the dependency of the 2nd source?

Even if all the above questions were answered, one would need to think about the benefits of such a feature. I have no advantage at hand for this. I would recommend to stick with single-sourced scope.

[agross]

@ilkerde Did you consider that there might be a lot of TeamCity and/or MyGet repositories that have dependencies on packages from nuget.org? Or is it something else you talk about?

[ilkerde]

@agross I thought we were talking about indirect dependencies (dependencies of dependencies) that are subject to be resolved in a different repository than the dependency?

The case you draw is that a package (application/library) has dependencies (direct dependencies to be precise) from other (or even different) sources. This is all fine by me.

However, it is indeed quite hard for my limited brain to find a consistent and simple way to support the case that a package (i.e. library A) might have dependencies on other sources, which themselves have dependencies (indirect dependencies of A) on yet other sources (maybe even the originating one).

I actually can imagine a solution by supporting multiple sources to be resolved in order (the famous "aggregated source" from nuget comse into mind). However, I fear that there are practical corner cases which are not so easy to deal with.

What do you think?

[agross]

I thought we were talking about indirect dependencies (dependencies of dependencies) that are subject to be resolved in a different repository than the dependency?

From paket's point of view right now, everything that is not specified in packages.fsx is an indirect dependency. So considering dep A from TeamCity (defined in packages.fsx) requires B from nuget.org, we better resolve B. (I built such packages in the past.)

I think that indeed we will need to have the aggregate source you talk about. The order in which the user specifies them should then be the order theses sources are searched for B (for consistency, first one with a matching package version wins). This would also lift the requirement to mix source and nuget to have the nuget use the correct source.

For reference, the package.fsx from my current project:

source "http://nuget.org/api/v2"

nuget "Castle.Windsor-log4net" "!>= 0"
nuget "Metrics.NET-net40" "~> 0.0"
nuget "Nancy.Hosting.Self" "~> 0.23.2"
nuget "Nancy.Bootstrappers.Windsor" "!~> 0.23"
nuget "Nancy.Serialization.JsonNet" "~> 0.23"
nuget "RestSharp" "~> 104.0"
nuget "Rx-Main" "!~> 2.2"

nuget "FakeItEasy" "~> 1.15"
nuget "NUnit.Runners" "~> 2.6"
nuget "SpecFlow.NUnit" "~> 1.1"

source "http://teamcity/guestAuth/app/nuget/v1/FeedService.svc"

nuget "Microsoft.Rtc.Collaboration-2010" "~> 4.0"
nuget "Microsoft.Rtc.Collaboration-2013" "~> 5.0"

[mavnn]

@agross is completely correct about the scenario which brought this to mind - internally private nuget feeds that reference dependencies from nuget.org (or from each other). In @agross example above, the packages in the teamcity feed might well have external dependencies.

I'm not sure about "The order in which the user specifies them should then be the order theses sources are searched for B (for consistency, first one with a matching package version wins)" - surely the most appropriate version available from any repository should win? If the internal feed has a patched version of (say) log4net with a higher version number I'd like that to be picked, whilst if NUnit have released a newer version on nuget.org than the one we're using internally that should be picked as well.

[agross]

@mavnn re: Ordering. Right, the best match should be picked from any source.

[forki]

Is this what you want:

source "http://teamcity/guestAuth/app/nuget/v1/FeedService.svc"
source "http://nuget.org/api/v2"  // fallback

nuget "Microsoft.Rtc.Collaboration-2010" "~> 4.0"
nuget "Microsoft.Rtc.Collaboration-2013" "~> 5.0"

which would specify only the direct dependencies. These packages might have dependencies to other packages. We would check both sources - in order?

[forki]

Technically we would need to change https://github.com/fsprojects/Paket/blob/master/src/Paket/BasicTypes.fs#L42 into a string list

[mavnn]

Unfortunately I think to get sane results you have to look for every package and dependency in every source, and then pick the best match. Annoyingly slow, but I can't see any other way of knowing if you're actually following the resolution strategy you've chosen.

[forki]

Mhm, fair point.
On Sep 5, 2014 12:59 PM, "Michael Newton" [email protected] wrote:

Unfortunately I think to get sane results you have to look for every
package and dependency in every source, and then pick the best match.
Annoyingly slow, but I can't see any other way of knowing if you're
actually following the resolution strategy you've chosen.

—
Reply to this email directly or view it on GitHub
#48 (comment).

[agross]

👍

[forki]

needs a testing

[agross]

Analyzing Dependencies
  FakeItEasy >= 1.15 < 2.0
  Presence-2013 1.6.5
Couldn't get package details for package Presence-2013 on [Nuget "http://teamcity/guestAuth/app/nuget/v1/FeedService.svc";
 Nuget "http://nuget.org/api/v2"]

Using Fiddler I see paket is requesting information from TeamCity (HTTP status 200), but then also from nuget.org with 404.

[agross]

I tested again with another package from TeamCity that has no dependencies itself.

Analyzing Dependencies
  FakeItEasy >= 1.15 < 2.0
  ManagedUC.Legacy-2010 3.0.7
Lockfile written to D:\Users\agross\Downloads\Dependencies.lock
FakeItEasy 1.24.0 already downloaded
Downloading ManagedUC.Legacy-2010 3.0.7 to C:\Users\agross\AppData\Local\NuGet\Cache\ManagedUC.Legacy-2010.3.0.7.nupkg
FakeItEasy 1.24.0 unzipped to D:\Users\agross\Downloads\packages\FakeItEasy
ManagedUC.Legacy-2010 3.0.7 unzipped to D:\Users\agross\Downloads\packages\ManagedUC.Legacy-2010
The input sequence was empty.
Parameter name: source

[agross]

I fiddled with the dependencies of the package on TeamCity:

source "http://nuget.org/api/v2"

nuget "FakeItEasy" "~> 1.15"

source "http://teamcity/guestAuth/app/nuget/v1/FeedService.svc"

nuget "Presence" "1.6.7"

Analyzing Dependencies
  FakeItEasy >= 1.15 < 2.0
  Presence 1.6.7
No package found which matches Presence "1.6.7" on [Nuget "http://teamcity/guestAuth/app/nuget/v1/FeedService.svc";
 Nuget "http://nuget.org/api/v2"].
Version available:

>nuget install Presence-2013 -version 1.6.7
Attempting to resolve dependency 'Castle.Windsor (= 3.2.1)'.
Attempting to resolve dependency 'Castle.Core (= 3.2.0)'.
...
Installing 'Newtonsoft.Json 6.0.5'.
Successfully installed 'Newtonsoft.Json 6.0.5'.
Installing 'Presence-2013 1.6.7'.
Successfully installed 'Presence-2013 1.6.7'.

[forki]

source "http://nuget.org/api/v2"
nuget "FakeItEasy" "~> 1.15"

source "http://teamcity/guestAuth/app/nuget/v1/FeedService.svc"
nuget "Presence" "1.6.7"

talks about "Presence" but in the manual nuget call you said "Presence-2013"

[forki]

source "http://nuget.org/api/v2"
nuget "FakeItEasy" "~> 1.15"

source "http://teamcity/guestAuth/app/nuget/v1/FeedService.svc"
nuget "msu.Addins" "03.00.44"

works fine on my machine.

[agross]

Does msu.Addins have dependencies on packages from nuget.org?

Alex

Alexander Groß
Tiny phone, tiny mail

On Tue, Sep 9, 2014 at 5:19 PM, Steffen Forkmann [email protected]
wrote:

source "http://nuget.org/api/v2"
nuget "FakeItEasy" "~> 1.15"
source "http://teamcity/guestAuth/app/nuget/v1/FeedService.svc"
nuget "msu.Addins" "03.00.44"

works fine on my machine.

Reply to this email directly or view it on GitHub:
#48 (comment)

[forki]

nope.
not sure how I can test this

[mavnn]

I'll try and have a go at it tomorrow.
On 9 Sep 2014 18:16, "Steffen Forkmann" [email protected] wrote:

nope.
not sure how I can test this

—
Reply to this email directly or view it on GitHub
#48 (comment).

[agross]

@forki You were absolutely right about my typo in the package name. But even when corrected to Presence-2013 it still fails w/ the same message.

[agross]

@forki My package has a dependency on Castle stuff from NuGet. You should be able to test it easily w/ #69 merged.

[agross]

Seems like this is due to a parser error when Paket retrieves information from TeamCity.

   at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
   at Paket.SemVer.parse(String version) in D:\Users\agross\Downloads\Paket\src\Paket\SemVer.fs:line 121
   at Paket.Nuget.parseVersionRange(String text) in D:\Users\agross\Downloads\Paket\src\Paket\Nuget.fs:line 81
   at [email protected](Tuple`2 tupledArg) in D:\Users\agross\Downloads\Paket\src\Paket\Nuget.fs:line 138
   at [email protected](String _arg1) in D:\Users\agross\Downloads\Paket\src\Paket\Nuget.fs:line 137
   at [email protected](a a)

We probably need to check for WebException in Nuget.fs, line 260 and let it crash otherwise (and not trying remaining sources).

[agross]

OK, I think I got it.

TeamCity returns the following XML for my package's dependencies (formatted manually):

<d:Dependencies>
  Castle.Windsor:[3.2.1]|
  Castle.Windsor-log4net:[3.2.0]|
  Rx-Main:[2.2.5]|
  Newtonsoft.Json:[6.0.5.0]|
  Microsoft.Rtc.Collaboration-2013:null
</d:Dependencies>

The last entry is converted to a string with a value of "null" (String.Equals(parsed, "null") == true) which passes the null check at the beginning of Nuget.parseVersionRange and fails later when it's parsed with Minimum(SemVer.parse text)(Nuget.fs, line 81).

[mavnn]

Ah yes, null versions. A lovely feature of nuget :(
On 9 Sep 2014 21:50, "Alexander Groß" [email protected] wrote:

OK, I think I got it.

TeamCity returns the following XML for my package's dependencies
(formatted manually):

<d:Dependencies>
Castle.Windsor:[3.2.1]|
Castle.Windsor-log4net:[3.2.0]|
Rx-Main:[2.2.5]|
Newtonsoft.Json:[6.0.5.0]|
Microsoft.Rtc.Collaboration-2013:null
/d:Dependencies

The last entry is converted to a string with a value of "null" (String.Equals(parsed,
"null") == true) which passes the null check at the beginning of
Nuget.parseVersionRange and fails later when it's parsed with Minimum(SemVer.parse
text)(Nuget.fs, line 81).

—
Reply to this email directly or view it on GitHub
#48 (comment).

[forki]

I think this is fixed. Please open a new specific issue if there are still problems.

[agross]

Still having issues with #48 (comment).