emgarten / sleet

A static nuget feed generator for Azure Storage, AWS S3, and more.

Build Status

AppVeyor	Azure Pipelines	Github

What is Sleet?

Sleet is a static NuGet package feed generator.

• Serverless. Create static feeds directly on Azure Storage, Amazon S3 or another S3 compatible storage. No compute required.
• Cross platform. Sleet is built in .NET, it can run on .NET Framework, Mono, or dotnet CLI
• Fast. Static feeds are created using the NuGet v3 feed format.
• Simple. Sleet is a simple command line tool that can add, remove, and update packages.
• Flexible. Configuration and credentials can be set using files, env vars, command line args, or AWS specific patterns to support a variety of workflows and CI builds.

Why use static feeds?

• Package binaries are typically kept outside of git repos, static feeds provide a long term storage solution that can be paired with checked in code.
• NuGet feeds are typically read for restore far more than they are updated.
• Cloud storage accounts are a cheap and secure way to share nupkgs for public feeds.
• You keep full control of your packages.

Getting Sleet

Manually getting sleet.exe (Windows and Mono)

1. Download the latest SleetExe nupkg from NuGet.org
2. Extract tools/Sleet.exe to a local folder and run it.

Install dotnet global tool

1. dotnet tool install -g sleet
2. sleet should now be on your PATH

Read the guides

Documentation can be found in this repo under /doc

Quick start guides

These provide a walk through on the basics of configuring sleet, creating, and using a feed.

• Setting up an Azure feed
• Setting up an AWS S3 feed
• Setting up a local feed with IIS hosting
• Integration with CI Server
• Setting up a private feed on AWS using S3 + CloudFront + Lambdas

Check out the full getting started guide here.

CI builds

CI builds are located on the following NuGet feed:

https://nuget.blob.core.windows.net/packages/index.json

The list of packages on this feed is here.

Contributing

We welcome contributions. If you are interested in contributing to Sleet report an issue or open a pull request to propose a change.

Sleet is..

Cold static packages from the cloud. ☁️ + 📦 = ❄️

History

Sleet was created to achieve the original goals of the NuGet v3 feed format: Provide maximum availability and performance for NuGet restore by using only static files.

The v3 feed format was designed to do all compute when pushing a new package since updates are infrequent compared to the number of times a package is read for restore. Static files also remove the need to run a specific server to host the feed, allowing a simple file service to handle it.

Related projects

• Sleet.Azure provides MSBuild props/targets for running Sleet.
• Sleet.Search provides a search service for Sleet feeds.

License

MIT License

sleet's Issues

Azure feed with BaseURI fails

Steps

1. Create a new azure feed
2. Set baseURI in sleet.json
3. Init

E:\tmp> .\Sleet.exe init --verbose
[System.InvalidOperationException] Unable to make 'https://example.com/test/sleet.settings.json' relative to 'https://sleettestfeed.blob.core.windows.net/basetest/'

This works for local feeds, but not for azure feeds.

//cc @natemcmaster

Root path

When I try to configure path to "http://nuget.example.com/" I get the following error: Object reference not set to an instance of an object

To work around this issue a have to add a folder to path as "http://nuget.example.com/feed".

Powershell Find-Module Support ?

Hello, does the sleet endpoint work with Find-Module for powershell 6?
I'm having trouble getting it to work, but not sure if it's even supported?

Thanks!

Allow IIS local package host to be hosted somewhere other than c:\ root

As I understand from the instructions and trying it out briefly, when using sleet for working with a static file host, you have to work from the root of c::

Sleet will create files for the feed in a new directory corresponding to the URI set in path, so if you changed path to https://example.com/feed, the files will be created in a directory named feed on you C:\ drive.

It would be good if this could be placed somewhere else, especially given the restrictions that those in enterprise have.

I will try to take a look at this in the week. It'll be my first time working with this code.

Cannot reference from regular build projects anymore

Given the following build script project:

<Project Sdk="Microsoft.Build.NoTargets/1.0.53">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Sleet" Version="2.3.12" PrivateAssets="all" />
  </ItemGroup>

</Project>

Version 2.3.12 successfully restores, while 2.3.17 (and every version following that) fails with:

error NU1202: Package Sleet 2.3.17 is not compatible with netcoreapp2.1 (.NETCoreApp,Version=v2.1). Package Sleet 2.3.17 supports: netcoreapp2.0 (.NETCoreApp,Version=v2.0) / any
error NU1212: Invalid project-package combination for Sleet 2.3.17. DotnetToolReference project style can only contain references of the DotnetTool type

Setting PackAsTool=true on that project, as suggested somewhere on the internets also doesn't fix it.

This makes the package effectively unusable as a reference for projects that invoke it directly (i.e. Sleet.Azure.

Note, also, that using DotNetCliToolReference instead of PackageReference fails in the exact same way too.

Reproduced with: Microsoft (R) Build Engine version 16.0.461+g6ff56ef63c and Microsoft (R) Build Engine version 16.0.461+g6ff56ef63c

Thanks!

Setting up a local V3 feed

I am using sleet to create a personal V3 feed which contains most of the packages from the official Nuget feed for an on-premise closed environment.

I managed to use sleet push to generate all of the necessary files (index.json, packageindex, catalog, autocomplete, etc...)

I am now trying to publish my feed via an ASP.NET IIS website which exposes the V3 index.json.

However, I am faced with an error when trying to browse my repo using the Visual Studio package manager:

[Custom Nuget] Only 'http' and 'https' schemes are allowed.
Parameter name: requestUri

My Custom Nuget configuration in VS2015:

Name: Custom Nuget
Source: http://nuget-test/index.json

Where nuget-test is a server running an empty ASP.NET web template with the Nuget.Sever package installed

All of the packages are stored locally on the server nuget-test.
The ASP.NET website was made using the guide on nuget.org

What am I doing wrong?

Also, I don't know how common of a use case this is but I think it's worth adding a guide on how to setup a local nuget feed using a personal sever and not one that is hosted on Azure

Clean catalog

I am currently using this to upload the latest builds of an in development project to Azure Blob storage, and we are only keeping the latest builds in the nuget feed. As silly as it may sound we are always uploading the same "version" of our package ie: package.1.0.0.1234.nupkg. It is this way because we are creating chocolatey packages for our pre-built versions of our software. However, the page.0.json file keeps growing along with the json files in the catalog\data folder. I realize this keeps transaction history, but (it may just be my imagination, there's probably no correlation) it seems like our uploads are slowing down the longer this gets, also I just don't like having it hanging out there when we don't need it.

Would it be possible to have a flag on the push command to update in-place without adding transaction logs or possibly a new command to clear the logs out?

Any thought about the ability to read configuration items via environment variables?

Just wondering if you've thought of supporting the ability to get the configuration items from env vars or another source instead of just sleet.json? Our azure data storage has a key in the url, and we'd like to not check that into our repo in the sleet.json.

Alternately, are you open to considering a pull request with the same?

Thanks

Usage question...

I'm getting accesskey and secretkey externally. How do I use sleet?

sleet download -o myfolder -p accessKeyId=mykeyid -p secretAccessKey=mysecretkey

doesn't seem to work for me...

Custom index.html

Maybe an index.html could be generated at the root of the container, and then using the new static website feature of Azure this would be served on the main url. That could include a static page to list the packages and the versions using some SPA application. This file could also be hosted somewhere else, and be compatible with any v3 feed I assume.

Error with sleet NuGet feed

Hello,

I'm trying to use an S3 bucket as a NuGet feed using Sleet. My end goal is to store the packages on S3 and use the S3 feed in our Octopus Deploy instance. I successfully instantiated Sleet by pushing a package, and I can load the index.json in the browser by loading https://xxxxxxxxxxxxxxxxxxxxx.amazonaws.com. But when I provide the Sleet feed URL in Octopus, I'm getting the error:

The V2 feed at 'https://xxxxxxxxxxxxxxxxxxxxx.amazonaws.com/Search()?$filter=IsAbsoluteLatestVersion&searchTerm='ninetails'&targetFramework=''&includePrerelease=true&$skip=0&$top=10&semVerLevel=2.0.0' returned an unexpected status code '404 Not Found'.

Could it be because Sleet is generating NuGet V3 feed while Octopus Deploy is calling a V2 API?

Any tips are greatly appreciated. Thanks!

Checking for nuspec should be case insensitive

In line

Sleet/src/SleetLib/Commands/PushCommand.cs

Line 222 in e11393a

nuspecName = identity.Id + ".nuspec";

"nuspec" name is composed by the package identity id + ".nuspec".

Normally this maps to the nuspec file name but if the casing is different when calling zip.GetEntry in line 223 then it will consider as if there was no nuspec since GetEntry is case sensitive.

A sample of how this breaks is package SQLite version 3.12.2-alpha.

If you need the actual package let me know and I'll share it

"Waiting to obtain feed lock" Error

Is anybody else getting this error with AWS?

When I go to my S3 bucket and look for .feedlock, I don't see one there to delete (per the error message instructions).

My build machine that is throwing this error is always installing the latest version of sleet for each build.

No other machine in my environment puts,deletes or otherwise locks the sleet repo.

Error message I am receiving:
"Waiting to obtain feed lock. Client holding the lock did not provide a message, it may be using an older version of Sleet. Delete /.feedlock to forcibly unlock the feed."

[Question] Host a authenticated NuGet v3 feed?

Is it possible to host an authenticated (basic auth), NuGet v3 feed, created with the Sleet toolchain, that Visual Studio PM, NuGet CLI, and DotNet CLI can consume packages?

Use system proxy with S3

It seems that I am hitting this limitation in dot net core when I try any sleet commands with S3

We have a proxy which windows knows about and which I can make use of when using .net framework, by simply adding:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.net>
    <defaultProxy useDefaultCredentials="true"></defaultProxy>
  </system.net>
</configuration>

to the app.config file. This doesn't seem to be supported by .net core though I expect there is a workaround. Any ideas if it should just work with proxies by default?

Request support for PackageIcon

It seems sleet doesn't support the <PackageIcon> attribute in .csproj.

I've been trying to push a package containing project with following settings in .csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netstandard2.1</TargetFramework>
    <Nullable>enable</Nullable>
    <Authors>xxxxxx</Authors>
    <Description>xxxxx.</Description>
    <Copyright>xxxxxx</Copyright>
    <PackageProjectUrl>xxxxx</PackageProjectUrl>
    <PackageIcon>icon.png</PackageIcon>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
  </PropertyGroup>
  <ItemGroup>
    <None Include="icon.png" Pack="true" PackagePath="" />
  </ItemGroup>
</Project>

The package can be successfully published to my customized AWS S3 nuget server, but the icon just doesn't show up.

Any ideas?

dotnet command line issues http request that are not handled by sleet

So I setup my S3 bucket, used sleet to upload my packages and verified that the bucket is populated after the push operation.

I also added a CloudFront distribution and auth handling (which I have temporarily disabled). With it enabled, I can successfully use Basic Auth and hit the index.json and get a response from my CloudFront Address

Now when I run something like:

 dotnet list package | awk '{print $2}' | grep <PackagePrefix> | xargs -n1 dotnet add package -v $(date +'%Y.%-m.%-d')-* -s https://<uniqueid>.cloudfront.net

So I can en masse update my internal packages (calver'd so the date thing works)

I see output like then when it goes to find the package from the provided source:

GET https://<uniqueid>.cloudfront.net/FindPackagesById()?id='My.Package'&semVerLevel=2.0.0

If I try to fetch that same address, I get a 404 so the dotnet add package command fails. or I let the command run and get this for every package:

error: Failed to retrieve information about 'My.Package' from remote source 'https://<uniqueid>.cloudfront.net/FindPackagesById()?id='My.Package'&semVerLevel=2.0.0'.
error: Response status code does not indicate success: 404 (Not Found).

I should also note that attempting to run this from Visual Studio Mac also fails in the same way

Am I missing something in my setup?

My sleet.json is as follows:

{
  "username": "",
  "useremail": "",
  "sources": [
    {
      "name": "feed",
      "type": "s3",
      "baseURI": "https://<uniqueid>.cloudfront.net",
      "bucketName": "<bucket name>",
      "region": "us-west-2",
      "profileName": "<aws profile>"
    }
  ]
}

Question About 403 Errors

We are building in CodeBuild on AWS.

Generally everything is working. But we are getting a crapton of 403 errors.

We have a NuGet.Config which has nuget.org first (I'm pretty sure the order does not matter)
and our cloudfront url as the next PackageSources.

We are getting 403 errors when it looks for packages not in the sleet repo.

These packages should be resolved on nuget.org.

These errors are noisy and make it difficult to debug real problems.

Is there something better we can do to get rid of all these 403s?

[Container] 2020/01/08 06:19:44 Running command dotnet restore --ignore-failed-sources
Retrying 'FindPackagesByIdAsync' for source 'http://d1rf46e0s0qx8w.cloudfront.net/flatcontainer/newtonsoft.json/index.json'.
Response status code does not indicate success: 403 (Forbidden).
Retrying 'FindPackagesByIdAsync' for source 'http://d1rf46e0s0qx8w.cloudfront.net/flatcontainer/yadayada.autoparts.entity/index.json'.
Response status code does not indicate success: 403 (Forbidden).
Retrying 'FindPackagesByIdAsync' for source 'http://d1rf46e0s0qx8w.cloudfront.net/flatcontainer/system.componentmodel.annotations/index.json'.
Response status code does not indicate success: 403 (Forbidden).
Retrying 'FindPackagesByIdAsync' for source 'http://d1rf46e0s0qx8w.cloudfront.net/flatcontainer/newtonsoft.json/index.json'.
Response status code does not indicate success: 403 (Forbidden).
Retrying 'FindPackagesByIdAsync' for source 'http://d1rf46e0s0qx8w.cloudfront.net/flatcontainer/yadayada.autoparts.entity/index.json'.
Response status code does not indicate success: 403 (Forbidden).
Retrying 'FindPackagesByIdAsync' for source 'http://d1rf46e0s0qx8w.cloudfront.net/flatcontainer/system.componentmodel.annotations/index.json'.
Response status code does not indicate success: 403 (Forbidden).

Question: Can packages be pushed remotely?

I am currently using sleet on a CI server to push builds to nuget and the feed is local to the server so this is fine.

However, now that I have added remote build agents, how can I push a nuget package to the central server after the build? We have SMB access available and could set up https but all of the docs examples use sleet.exe directly.

Is there anything built-in or do I need to write a web wrapper? Thanks

Provide $(Sleet) path via the package with Sleet.targets

So that the tool can be trivially be invoked from build scripts without having to guess at the install path ;)

I wish NuGet generated this automatically on restore for everything that has something in tools.

This should be similar to gluck/il-repack#192

Support SNUPKG format

Support current symbol package format.

• Ref: https://docs.microsoft.com/en-us/nuget/create-packages/symbol-packages-snupkg

Wiki Wording Issue on Item "Delete"

see: https://github.com/emgarten/Sleet/wiki/Command-reference#options-3

Delete
Push adds packages to your feed. It can used to add individual packages or complete directories of >packages.

Item=Delete, Text = "Push"...

regards
Werner

S3 speeds?

Hi,

Awesome project. I'm completely baffled this works :O

I'm uploading 32.000 packages to S3, and it's quite slow. I didn't expect it to take minutes, but I did expect way higher speeds than 3-4 Mbps (using S3 CLI and others usually gives 30-40 Mbps).

I can see Sleet uses some parallel upload thing, but is it possible that this doesn't work or is inefficient? .. Or do I simply have a billion tiny files?

Usability/exception wording Issue for Azure

Hi
working with your project (nice!!) i found some detail that can be improved.

Steps to reproduce:

1. do a "createconfig" for Azure
2. do a "init" for that azure config WITHOUT changing the json file from step 1.

expected result:

• doing the init should not be possible because the azure account is not configured (json file not changed)
• expecting a meaningfull exception message

found result/exception message:
[System.FormatException] No valid combination of account information found.

The exception is coming from the Azure client and not helpful (not the exception type, not the message, not one word about azure context)

Suggest to provide more context in that error message.

There are multiple ways to improve that....
please give a look at the following:
https://github.com/WernerMairl/Sleet/commit/d9db17236e63e3c27d59424bb1d07deaabfe306c

i can create a PR...

regards
Werner

Push command fails to read versioning

When using 'sleet push' on a directory filled with packages pulled from the nuget.org V3 feed, I encounter the following error:

Package 'D:\Packages\Aaron.Core\3.2.4\Aaron.Core.3.2.4.nupkg' does not contain a valid version: '3.2.4.0'. See https://semver.org/ for details

I am using sleet version 1.0.0

Follow Recommended AWS Credential Management Practices

It's great that Sleet supports S3, but the current implementation of FileSystemFactory requires an AWS access key and secret key to be entered into the sleet.json configuration file. Since sleet is using the AWS SDK, this is unnecessary -- the SDK will search a series of locations on the instance/pc to find credentials -- and it is counter to the recommended practice. See the second bullet point here:

https://docs.aws.amazon.com/sdk-for-net/v2/developer-guide/net-dg-config-creds.html

The two needed values are the bucket and region (the latter can't be determined without an S3 client, and the client can't be created without a region -- a nice catch-22). Other than that, an optional AWS profile name would be the way a developer can instruct the SDK to find one of several configured profiles.

I'll follow-up this issue with a PR, but the minimal change is to simply not require the keys to be configured (remove those lines) and use a different constructor for the S3 client when the keys are not provided:

new AmazonS3Client(regionSystemName)

Adding an optional "profile name" value is a little more involved, obviously.

Thanks for Sleet, by the way, it's awesome.

How to integrate into a NuGet.Server Feed

How can this v3 feed generator integrated into a NuGet.Server (v2 Feed)?
Is this possible?

Using GitHub to host a NuGet feed/packages?

I want to setup CD for my repos, but I have a few problems.

1. I'd like to deploy every commit to a NuGet feed.
2. I don't want to spam nuget.org with each and every commit (even pull requests).
3. My packages contain native libraries which eat a lot of space. This rules out MyGet since there storage quotas are low.

This leads me to my next question.

What about creating a GitHub repository that contains the required files for a valid NuGet feed?

"But what about the large nupkg files?" Yeah, you wouldn't want to commit large binaries into a repository. So instead, I will configure my CI/CD to auto-upload my nupk files to a GitHub release which has no size limits. The static files the represent my NuGet feed will point to the nupkg files uploaded to the GitHub release.

I'd like to do something like this.

# Assume v0.1.0 is a GitHub release in the repo "pauldotknopf/myfeed".
sleet push pauldotknopf/myfeed:v0.1.0 --source myGitRepoFeed

The GitHub release would be inspected and would update the NuGet feed.

What do you think? Is this feasible? If so, would you be open to a PR?

Add instructions for symbol server

The documentation should include the directions to configure a symbol server.

Implement retention policy

Delete packages based on:

• older than last n pre-release versions
• older than last n release versions

Not by date. Just in order to keep a limited set of packages.

Use Sleet for chocolatey

I like sleet.exe and the simplicity to put package on azure blob.
But Is it also possible to use sleet for Chocolatey packages?

Help needed Sleet/Nuget.exe

Hi

I have created a (Sleet) azure feed (without sleet.exe but with sleetlib using dotnet core 2.0)
the feed should be available so anyone can inspect/reproduce that:
https://sitsfeed.blob.core.windows.net/sleet/index.json

using nuget.exe (in two different versions) i get always the same result:

command executed:
nuget.exe list -verbosity detailed -source https://sitsfeed.blob.core.windows.net/sleet/index.json

expected: a list of packages (there should be one package there)

Results:

NuGet Version: 4.3.0.4406
WARNING: This version of nuget.exe does not support listing packages from package source 'https://sitsfeed.blob.core.windows.net/sleet/index.json'.
No packages found.

NuGet Version: 4.4.0.4475
WARNING: This version of nuget.exe does not support listing packages from package source 'https://sitsfeed.blob.core.windows.net/sleet/index.json'.
No packages found.

inspecting the feed with a browser i can see:

index.json returns:

{"version":"3.0.0","sleet:version":"1.0.0","resources":[{"@id":"https://sitsfeed.blob.core.windows.net/sleet/sleet.settings.json","@type":"http://schema.emgarten.com/sleet#SettingsFile/1.0.0","comment":"Sleet feed settings."},{"@id":"https://sitsfeed.blob.core.windows.net/sleet/catalog/index.json","@type":"Catalog/3.0.0","comment":"Catalog service."},{"@id":"https://sitsfeed.blob.core.windows.net/sleet/autocomplete/query","@type":"SearchAutocompleteService/3.0.0-beta","comment":"Powershell autocomplete."},{"@id":"https://sitsfeed.blob.core.windows.net/sleet/search/query","@type":"SearchQueryService/3.0.0-beta","comment":"Static package list in search result form."},{"@id":"https://sitsfeed.blob.core.windows.net/sleet/sleet.packageindex.json","@type":"http://schema.emgarten.com/sleet#PackageIndex/1.0.0","comment":"Sleet package index."},{"@id":"https://sitsfeed.blob.core.windows.net/sleet/registration/","@type":"RegistrationsBaseUrl/3.4.0","comment":"Package registrations used for search and packages.config."},{"@id":"https://sitsfeed.blob.core.windows.net/sleet/","@type":"ReportAbuseUriTemplate/3.0.0","comment":"Report abuse template."},{"@id":"https://sitsfeed.blob.core.windows.net/sleet/flatcontainer/","@type":"PackageBaseAddress/3.0.0","comment":"Packages used by project.json"},{"@id":"https://sitsfeed.blob.core.windows.net/sleet/symbols/packages/index.json","@type":"http://schema.emgarten.com/sleet#SymbolsPackageIndex/1.0.0","comment":"Packages indexed in the symbols feed."},{"@id":"https://sitsfeed.blob.core.windows.net/sleet/symbols/","@type":"http://schema.emgarten.com/sleet#SymbolsServer/1.0.0","comment":"Symbols server containing dll and pdb files."}],"@context":{"@vocab":"http://schema.nuget.org/services#","sleet":"http://schema.emgarten.com/sleet#","comment":"http://www.w3.org/2000/01/rdf-schema#comment"}}

packages/index.json
returns the expected package (one)

{"packages":{"SQLiteExtensions":["1.2.0"]},"symbols":{}}

I'm not able to see the problem. What i'm doing wrong ?

regards
Werner

Support external search

Not sure if it repros in 16.1, but in my 16.2, search in a sleet feed from VS just returns all the packages in the index, no filtering performed at all :(

Improve handling of azure feed lock failures

Azure feed locks depend on blob leases which need to be renewed within 60 seconds. The current behavior is to fail silently and get a new lease if the lease cannot be renewed. This should be improved to add console logging for when it happens so that build logs can diagnose what happened. Sleet should also avoid pushing changes when the lock is in a bad state.

Add console logging for lease failures on azure blobs:

Sleet/src/SleetLib/FileSystem/AzureBlobLease.cs

Lines 49 to 54 in bfb6457

	catch (Exception ex)
	{
	// attempt to get the lease again
	await GetLease();
	Debug.Fail($"Renew failed: {ex}");
	}

Before committing and after declaring a commit successful the feed lock should be checked to verify that the lock is still held. If it isn't then push should fail so that the operation can be retried.

Sleet/src/SleetLib/FileSystem/FileSystemBase.cs

Line 63 in bfb6457

public async Task<bool> Commit(ILogger log, CancellationToken token)

GetCallerIdentityAsync throws an exception when calling it from a docker container

Hi,

In AWS CodeBuild, you have the ability to build your code in a Docker container.
To propagate the credentials to your container, you need to pass few extra parameters:

docker build --build-arg AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION --build-arg AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI .

To validate this config, I ran aws sts get-caller-identity and got this response:

Step 9/31 : RUN aws sts get-caller-identity 
 ---> Running in 080c749553b9 
{ 
    "Account": "123456789",  
    "UserId": "AROXXXXXXX:AWSCodeBuild-(...)",  
    "Arn": "arn:aws:sts::123456789:assumed-role/codebuild-core-srv-pricing-service-role/AWSCodeBuild-(...)" 
}

When running sleet in this environment, I get the following exception:

[System.ArgumentException] Failed to determine AWS identity - ensure you have an IAM role set, have set up default credentials or have specified a profile/key pair.

Unfortunately we do no have much details regarding the exception raised by the AWS client. Would be great to add a exception.ToString() so we can troubleshoot.

I'm not sure why we are getting this error but wan we try adding a parameter called --inherit-permissions bypassing this check?

Sort out the usage of relative paths for local feeds

From #50:

I tried to start from scratch and changed the path in the generated sleet.json file from an absolute path to a relative one:

{
  "username": "",
  "useremail": "",
  "sources": [
    {
      "name": "my-feed",
      "type": "local",
      "path": "my-feed",
      "baseURI": "https://example.com/feed/v3"
    }
  ]
}

Initializing the feed doesn't work:

$ sleet init --source my-feed
[System.ArgumentException] Cannot use a relative 'path' without a settings.json file.

settings.json should be sleet.json, because that's what Sleet is looking for.

Sleet behaves differently if the path to the configuration is specified as an argument:

$ sleet init -c sleet.json --source my-feed
[System.UriFormatException] Invalid URI: The format of the URI could not be determined.

However, it works if the path to the configuration file is an absolute path:

$ sleet init -c `pwd`/sleet.json --source my-feed
Initializing https://example.com/feed/v3/
# more log messages ...

Or a relative path:

$ sleet init -c ./sleet.json -s my-feed

I certainly didn't expect sleet.json and ./sleet.json to be handled differently.

Json files are generated with BOM

Some client's do not like it, for example AppServices SiteExtension manager (that seems to use nuget libraries).

System.Exception
--
StackTrace | at NuGet.Data.DataClient.<>c__DisplayClass4.<GetJObjectAsync>b__3()     at 
System.Threading.Tasks.Task`1.InnerInvoke()     at
 System.Threading.Tasks.Task.Execute()  
--- End of stack trace from previous location where exception was thrown ---     at
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at 
NuGet.Data.DataClient.<GetJObjectAsync>d__6.MoveNext()  --- End of stack trac… 
--- End of stack trace from previous location where exception was thrown ---     at
 System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()
 --- End of stack trace from previous location where exception was thrown ---     at
 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at
 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at
 System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext()

Unexpected character encountered while parsing value: �. Path '', line 0, position 0.
at Newtonsoft.Json.JsonTextReader.ParseValue() at Newtonsoft.Json.JsonTextReader.Read() at 
Newtonsoft.Json.Linq.JObject.Load(JsonReader reader, JsonLoadSettings settings) at 
Newtonsoft.Json.Linq.JObject.Parse(String json, JsonLoadSettings settings) at
 NuGet.Data.DataClient.<>c__DisplayClass4.<GetJObjectAsync>b__3()

Add a prune command

A new prune command would be very useful to save on storage specially for CI-populated feeds.

Some options (after slate prune ) that would be great:

• -i|--include: package id/s to consider for pruning. * would mean evaluate all.
  Can be specified multiple times.
• -n|--no-downloads: delete all packages that have zero downloads
• -m|--max-age: delete all packages that are older than the given days. When combined with
  --no-downloads, it would only delete old packages if they have no downloads.

Support dotnet tools

Add a second package that can be installed as a dotnet global tool.

Remove the dotnet CLI tool type, it looks like this is currently causing nuget.exe install sleet to fail.

Sleet unable to find 2.1.0-rc1 installation

sleet --help

Output:

It was not possible to find any compatible framework version
The specified framework 'Microsoft.NETCore.App', version '2.1.0-rc1-26428-03' was not found.
  - Check application dependencies and target a framework version installed at:
      /usr/local/share/dotnet
  - Installing .NET Core prerequisites might help resolve this problem:
      http://go.microsoft.com/fwlink/?LinkID=798306&clcid=0x409
  - The .NET Core framework and SDK can be installed from:
      https://aka.ms/dotnet-download
  - The following versions are installed:
      2.0.5 at [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
      2.1.0-preview1-26216-03 at [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
      2.1.0-preview2-26406-04 at [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
      2.1.0-rc1 at [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

As you can see, I do have 2.1.0-rc1 installed but it seems your application requires a very specific rc1 build. The currently released version of the the 2.1 SDK is 2.1.300-rc1-008673

Public dependencies resolving to private feed

I have a package that has some dependencies installed from the main public Nuget feed. When I publish my package to a S3 private feed using Sleet then attempt to reference that package, Nuget attempts to get those public dependencies from the private feed and fails as they don't exist. Is there a way to configure Sleet to use the public feed for public dependencies?

Symbol server case sensitivity issues

When publishing a symbol package to the Azure feed, Sleet preserves the casing of file names and creates the PDB hash directory in uppercase.
So the PDB path looks like this:
https://mysymbols.blob.core.windows.net/nuget/symbols/My.Library.pdb/7A9D4CD67D0946D48B6D815D13E5D0C8ffffffff/My.Library.pdb
But Visual Studio uses another casing schema when trying to download symbol files:
https://mysymbols.blob.core.windows.net/nuget/symbols/my.library.pdb/7a9d4cd67d0946d48b6d815d13e5d0c8FFFFFFFF/my.library.pdb
As a result, on Azure Blobs it gets 404 because Azure Blob Storage is case sensitive.
I'm not sure that it is an expected behavior, is there any workaround?

Symbol packages

Sleet could be symbol server too.

Update global CLI tool to netcoreapp2.1

The latest sleet package doesn't work with the 2.1.300-rc1 SDK as a global tool anymore. The CLI team is making netcoreapp2.1 a minimum requirement for CLI tool packages. https://github.com/dotnet/cli/issues/9073

Relative "path" setting

It would be useful if we had the option to use a relative path for the 'path' option, with it relative to the config file itself (like nuget.config allows). I tried it with both ".\\" and "" but both gave errors about invalid URIs

When running:

sleet push --config path_to_config.json -s myLocalFeed path_to_nupkg

{
  "username": "",
  "useremail": "",
  "sources": [
    {
      "name": "myLocalFeed",
      "type": "local",
      "path": ".\\",
      "baseURI": "https://mungojam.github.io/sleet-test/"
    }
  ]
}

[System.UriFormatException] Invalid URI: The format of the URI could not be determined.

{
  "username": "",
  "useremail": "",
  "sources": [
    {
      "name": "myLocalFeed",
      "type": "local",
      "path": "",
      "baseURI": "https://mungojam.github.io/sleet-test/"
    }
  ]
}

[System.UriFormatException] Invalid URI: The URI is empty.

Support AWS IAM Role credentials

Provide support for AWS IAM Role credentials rather than having to set long term credentials in the sleet.json or reference a user's .aws profile. This would allow Sleet to be used with automated build processes that run under an IAM Role and be more secure.

Bump RegistraionsBaseUrl to 3.6.0

Sleet supports Semver 2 versions. Should the RegistrationsBaseUrl be bumped from 3.4.0 to 3.6.0?

Sleet/src/SleetLib/Commands/InitCommand.cs

Line 89 in 51b0b8b

AddServiceIndexEntry(source.BaseURI, "registration/", "RegistrationsBaseUrl/3.4.0", "Package registrations used for search and packages.config.", serviceIndexJson);

Packegs folder disappeared from flatcontainer

Over the weekend, a previously working installation stopped working for a specific package in our feed. On examination, the packages folder was deleted from inside flatcontainer but the the index file sleet.packageindex.json with the versions was still correct. The folder was still present underneath the registrations folder.

Obviously this caused nuget to fail installation on a client machine. We honestly can't think how that folder would have been deleted manually. Neither of the two people who have access did it and we don't go into those folders anyway since packages are pushed directly from the CI process.

Is there any way that Sleet might have deleted it for a reason that wasn't correct? Since the index files weren't cleaned up as well, I assume this would be a more subtle bug otherwise all traces would have gone?

Thanks

Use finer grain lock?

Hey, my team have many repositories that frequently try to push concurrently to a common feed hosted in Azure. These repos always try to push different packages but even so the publishing get serialized because of the lock on the whole feed. Question is: do you think it is feasible to make the feed lock fine grained so that I can at least push different (as per version+name) packages simultaneously?

Catalog writer blows up when page 2 is being written

Test Name:	Sleet.Test.CatalogTests.CatalogTest_AddPackageAsync_SupportsWritingMultiplePages
Test FullName:	Sleet.Test.CatalogTests.CatalogTest_AddPackageAsync_SupportsWritingMultiplePages
Test Source:	C:\src\git\github.com\joelverhagen\Sleet\test\SleetLib.Tests\CatalogTests.cs : line 224
Test Outcome:	Failed
Test Duration:	0:00:01.832

Result StackTrace:	
at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
   at Sleet.CatalogUtility.UpdatePageIndex(JObject catalogIndexJson, JObject currentPageJson, Guid commitId) in C:\src\git\github.com\joelverhagen\Sleet\src\SleetLib\Utility\CatalogUtility.cs:line 307
   at Sleet.Catalog.<AddCatalogEntry>d__29.MoveNext() in C:\src\git\github.com\joelverhagen\Sleet\src\SleetLib\Services\Catalog.cs:line 342
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sleet.Catalog.<AddPackageAsync>d__13.MoveNext() in C:\src\git\github.com\joelverhagen\Sleet\src\SleetLib\Services\Catalog.cs:line 83
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sleet.Test.CatalogTests.<CatalogTest_AddPackageAsync_SupportsWritingMultiplePages>d__2.MoveNext() in C:\src\git\github.com\joelverhagen\Sleet\test\SleetLib.Tests\CatalogTests.cs:line 281
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Result Message:	System.InvalidOperationException : Sequence contains more than one element

Test added in PR: #13