loic-sharma / baget

A lightweight NuGet and symbol server

BaGet 🥖

A lightweight NuGet and symbol server.

Getting Started

1. Install the .NET SDK
2. Download and extract BaGet's latest release
3. Start the service with dotnet BaGet.dll
4. Browse http://localhost:5000/ in your browser

For more information, please refer to the documentation.

Features

• Cross-platform: runs on Windows, macOS, and Linux!
• Cloud native: supports Docker, Azure, AWS, Google Cloud, Alibaba Cloud
• Offline support: mirror a NuGet server to speed up builds and enable offline downloads

Stay tuned, more features are planned!

baget's Issues

[UI] Integrate frontend with ASP.NET

See https://natemcmaster.com/blog/2018/07/05/aspnetcore-hmr/ and https://github.com/natemcmaster/aspnetcore-webpack-hmr-demo

[Search] Support Azure Search

[UI] Display proper download count

The UI uses the NuGet Registration resource to display a single package. Since the documented registration does not provide download counts for a package, the UI cannot show download counts. So:

• Add per-version download counts to registration index resource
• Add overall download counts to registration index resource
• Display download counts on UI
• Publication time in search service

Saving the Package Dependency of a target Framework fails when id is null

Reproducion:

• Blank baget.db
• create a new c# class library (based on netstandard 2.0)
• latest sdk tools
• try to dotnet nuget push ..\ClassLibrary1\bin\Debug\ClassLibrary1.1.0.0.nupkg -s http://localhost:50561/v2/package

The error is: Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 19: 'NOT NULL constraint failed: PackageDependencies.Id'.

see:

BaGet/src/BaGet.Core/Services/IndexingService.cs

Line 192 in d5f957c

Id = null,

But i think null is correct ...(https://github.com/NuGet/NuGetGallery/blob/master/src/NuGetGallery.Core/Entities/PackageDependency.cs)

Implement catalog indexer

Allow BaGet to index packages off of a catalog. This would allow BaGet instances to mirror nuget.org in real-time.

See CatalogReaderExample or NuGet.CatalogReader for basic implementations. A simple solution would be to use the read-through caching by hitting an endpoint for each package on nuget's catalog.

Update gitignore file to ignore .vscode folder

Plus delete .vscode folder :)

Create a docker image

/cc @emgarten

Add API Key Authentication

The package publish API should support the X-NuGet-ApiKey HTTP header.

Add a UI

Add a simple UI that focuses on package search. Yarn does this very well.

[Azure Search] Importer

Create a tool that creates an Azure Search index from a BaGet database

See https://docs.microsoft.com/en-us/azure/search/search-import-data-dotnet

[Bug] Authors not split properly

Authors should be split on commas. For example, see package ProNuGet.

This should propagate to search index

Documentation

Build documentation for:

• Development
  • Current flow for UI development
• Set up
  • Sqlite & SQL Server & MySql
  • Azure
    • Setting up an App Service
    • Setting up Azure SQL
    • Setting up Azure Blob storage
    • Setting up Azure Search
• Authentication
  • API Keys
  • Private feeds
• Indexing nuget.org
  • How to crawl nuget.org's catalog to index all packages
  • Importing downloads from nuget.org
• Misc
  • Read-through caching
  • How to create a package whose readme is indexed by BaGet
    • This will be replaced by self-contained packages. See: #67
  • Differences between BaGet's APIs and NuGet's documented v3 protocol
    • Readme in flat container
    • Repository URL and repository type in registration
    • Dependents search API
    • Package types in registration

Sections (example I like):

• Overview
  • Example here. Quick intro, ends with "Next steps" section to quickstarts
• Quickstarts
• Tutorials
• Reference
• Resources

See https://squidfunk.github.io/mkdocs-material/

Unittesting against official Nuget v3 (core) Client

In the current (first draft) of testing infrastructure we use "Microsoft.AspNetCore.TestHost.TestServer" for testing Controllers.

Idea/Requirement
Inject HttpClient capabilites provided by the TestServer into NuGet.Protocol.HttpSource and use official NuGet.Client Assemblies (V3/Core) to (unit) test our Controllers (in process).

Helpful ?
Any concerns ?

[Search] Investigate Full-Text Search

Today, the database-backed search service cannot handle nuget.org's scale. Investigate:

• Over-all issue: dotnet/efcore#1590
• Full-text search for SQLite: dotnet/efcore#4823
• External full-text search (like Azure Search): dotnet/efcore#11489

Upgrade to EF Core 2.1

Use value conversions

Update ASP.NET Core to 2.1.0

Includes:

• Update dependencies
• Extension on Azure App Service
• Investigate HttpClient factory for package downloader
• Investigate EF 2.1
  • Full-text search
  • Better group by

Add google analytics

<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-72218961-2"></script>
<script>
  window.dataLayer = window.dataLayer || [];
  function gtag(){dataLayer.push(arguments);}
  gtag('js', new Date());

  gtag('config', 'UA-72218961-2');
</script>

Need to figure out how to pass configurable values to parcel for the GA id. If this isn't possible, parcel will need to output the front end as a razor view, and asp.net will need to replace the GA id.

Add GitHub templates

Do you want to request a feature or report a bug?

Feature

What did you expect to see?

Currently there's no format for a user to file an issue, bug, feature or pull request. This can be done by adding github templates.

This results in more quality and better formatted issues and is better for the overall quality of the project.

[Search] Support all query parameters

BaGet's search should support the prerelease and semVerLevel parameters.

See https://docs.microsoft.com/en-us/nuget/api/search-query-service-resource#request-parameters

Support more databases

Today, only SQLite is supported. This is great for small-scale deployments but has its limits. Eventually, support for more scalable solutions will be added. I am considering both SQL Server and Cosmos DB.

[Azure] Codify infrastructure

Use terraform to make BaGet's Azure infrastructure repeatable.

Improve database constraints

• Package.Id: Required, max length of 128
• Package.VersionString: required
• PackageDependencyGroup.Id: Required, max length of 128 , case insensitive on Sqlite

See https://github.com/NuGet/NuGetGallery/blob/7e0f3c7af27a0a12cdec012a39f78fafab28b0a4/src/NuGetGallery.Core/CoreConstants.cs#L10

Add caching

As read throughput increases, the database will get pummeled by requests. The registration service's responses are highly static and should be cached.

See https://docs.microsoft.com/en-us/aspnet/core/performance/caching/distributed?view=aspnetcore-2.1
See https://docs.microsoft.com/en-us/azure/redis-cache/

Index packages' icons and licenses

Download the resource pointed to by nupkg's iconUrl and licenseUrl and store it in the flat container

Add tests

Today, the project is as simple as possible to enable fast iteration. Tests should be added once things stabilize.

[Azure Search] Create data model

Create a data model that can support NuGet's v3 search API.

See https://docs.microsoft.com/en-us/azure/search/search-create-index-dotnet
See https://docs.microsoft.com/en-us/azure/search/search-howto-complex-data-types

Improve validation of AppSettings/IConfiguration

Currently we get a lot of ArgumentNull Exceptions on unexpected locations if we start the App (WebApp or Tests) without config Settings (appsettings.json).

Suggest:

• fail fast and early wiith better exception types and messages
• use defaults (Convention over Configuration) instead of exceptions where it is possible
• include (unit) tests.

[Azure Search] Investigate Analyzers

Searching for newton does not return Newtonsoft.Json. It seems that the Apache Lucene standard analyzer is not ideal for id searches.

See https://docs.microsoft.com/en-us/azure/search/search-analyzers
See https://docs.microsoft.com/en-us/rest/api/searchservice/custom-analyzers-in-azure-search#default-analyzer

Track commonly downloaded together packages

NuGet v4.7+ adds the header X-NuGet-Session-Id to restore operations. This lets us understand what packages were downloaded together for a single restore. The basic flow:

1. Track packages downloaded together in a single operation
2. Build a graph of the restore operation
3. Find the top-level dependencies
4. Recommend packages that are commonly restored together as top-level dependencies.

See https://docs.microsoft.com/en-us/nuget/api/overview#http-request-headers

[Azure Search] Map Search Service to NuGet V3 APIs

See https://docs.microsoft.com/en-us/nuget/api/search-query-service-resource
See https://docs.microsoft.com/en-us/azure/search/search-query-dotnet

Join .NET Foundation

See https://dotnetfoundation.org/get-involved
See https://github.com/dotnet/home/blob/master/guidance/new-projects.md
See https://github.com/dotnet-foundation/project-maturity-model/blob/master/maturity-ladder.md

Track Downloads

Needs:

• Migration to add Downloads column
• Make package downloads increment download count
• Create tool to populate downloads from nuget.org
  • Should download nuget.org's download.v1.json artifact
  • Bulk update using EFCore.BulkExtensions?

[Bug] Incorrect tags

Every other tag for package Strava.NET has an empty value

Add symbol server

See https://github.com/SymbolSource/SymbolSource

Extract packages' readme to flat container

The front-end will need to display the readme. This requires:

• Indexing a nupkg should store its readme to the flat container
• The registration should provide a URL to the readme

Implement package drop folder with automatic indexing

Allow BaGet to automatically index packages that are dropped into a designated folder.

See NuGet.Server's ServerPackageRepository.

Support private feeds

Support authentication for private feeds. Also, consider package management using a login system.

See https://docs.microsoft.com/en-us/nuget/reference/extensibility/nuget-exe-credential-providers

HTTP basic auth: https://gist.github.com/DutchCaveman/4287549fc36dace5ac780ce023ddb05a
Client certificate auth: NuGet/Home#5773 (comment)
Symbol server PAT auth: dotnet/diagnostics#1321 (comment)

/cc @emgarten

Azure Integration

Getting BaGet up and running on Azure should be easy. Requirements:

• App Service
  • Secrets. See this documentation
• Blob storage
• SQL Server (see #1)

Things to investigate and nice to haves:

• Application Insights. See this documentation
• Proper infrastructure deployment. ARM templates? Terraform? #29
• Azure Search? #19

Support Repository Signatures

nuget.org will soon add repository signatures to all packages it hosts.

See NuGet/Home#6378
See https://github.com/NuGet/NuGet.Client/blob/095d2d9563bac6b6bb3269e6996dd4e5af690f20/src/NuGet.Clients/NuGet.MSSigning.Extensions/RepoSignCommand.cs

[UI] Improve Markdown Rendering

Rendering currently uses React's dangerouslySetInnerHTML. Investigate using a native React component instead.

See https://github.com/loic-sharma/BaGet/blob/master/src/BaGet.UI/components/DisplayPackage.tsx#L146

Redirect for downloads

Today, package content downloads consume a thread for the duration of the download. This will not scale well. Instead, download endpoints should redirect to Azure Blob Storage/CDN.

Add dependency groups to registration

Registration index's catalogEntry should list package's dependencies, grouped by target framework.

See: https://docs.microsoft.com/en-us/nuget/api/registration-base-url-resource#registration-index

GitHub integration for source-only packages

See https://nikcodes.com/category/nuget/

Implement read-through caching

On package miss, go to nuget.org to download the package.

See NuGet/NuGetGallery#5430

FilePackageStorageService needs more Spec

Hi
I like the idea of a file based file based PackageStorageserver - specially for simpler scenarios and tests.

I see a few requirements for this service (and probably other file based services) that we should spec out before we improve the code.

• for debugging/tracing/support the FolderPath that is used by the service should be part of the log.
• System.IO.Path.GetFullPath can/should be applied to the Path value provided by IConfig (like to see where "./folder" is pointing to (Current Directory is always implicit, for debugging we should print out explicite)
• System.Environment.ExpandEnvironmentVariables may be also applied (attention there is a issue in CoreFx about xplat problems with "%xxx%" syntax)

if we do things like that, on which layer ?

• inside the service /constructor
• before the constructor ?

---

other detail that needs spec:
Currently the Fodler for this service is created inside class Startup.

We should spec out

• if the folder MUST be exist for the constructor (then constructor should throw exception if not there)
• if the folder is created on demand
  • inside the constructor
  • later on demand

---

my personal opinion/vote:

1. we inject Logger into FilePackageStorageService.Constructor (may be with a NullLogger as default)
2. inside the constructor we use System.IO.Path.GetFullPath to resolve the path and we print out the result to logger

2.1 System.Environment.ExpandEnvironmentVariables may be applied here now or later!
3. we CREATE the folder if it does not exists (ensures a early exception in case of missing write permission)
3.1 (in case of a existing folder we can write and delete a file with length 0 to ensure we have write permission, like System.IO.Path.GetTempFile)

[Azure Search] Improve Search

• Scoring Profiles
  • Make exact id matches always at top
  • Make id results stronger than description/summary
  • Make tag results strong
  • Prefer packages with high downloads
  • Prefer packages with high number of dependents (# packages that depend on this package)
  • Penalize packages that haven't received updates recently
• Analyzers
  • Typo-tolerant prefix search on id

See https://docs.microsoft.com/en-us/rest/api/searchservice/add-scoring-profiles-to-a-search-index
See https://docs.microsoft.com/en-us/azure/search/search-analyzers
See https://docs.microsoft.com/en-us/rest/api/searchservice/custom-analyzers-in-azure-search#default-analyzer

See https://medium.com/@Herger/chapter-2-azure-search-in-the-spotlight-72da0b4cf39c

See https://github.com/algolia/npm-search#schema

Create an indexing tool similar to NuGet's lightning

See: https://github.com/NuGet/NuGet.Services.Metadata/blob/master/src/Ng/Jobs/LightningJob.cs

Used to index a v3 source using its catalog.

[Azure Search] Index package uploads

Today, the only way to index documents into Azure Search is manually through the ad hoc tool. Newly uploaded packages should be indexed automatically.

Add response compression

See https://docs.microsoft.com/en-us/aspnet/core/performance/response-compression?view=aspnetcore-2.1&tabs=aspnetcore2x#response-compression

Support v2 APIs

Support OData endpoints. Use: https://www.nuget.org/packages/Microsoft.AspNetCore.OData