ksp-ckan / ckan

The Comprehensive Kerbal Archive Network

The Comprehensive Kerbal Archive Network (CKAN)

Click here to open a new CKAN issue

Click here to go to the CKAN wiki

Click here to view the CKAN metadata specification

What's the CKAN?

The CKAN is a metadata repository and associated tools to allow you to find, install, and manage mods for Kerbal Space Program. It provides strong assurances that mods are installed in the way prescribed by their metadata files, for the correct version of Kerbal Space Program, alongside their dependencies, and without any conflicting mods.

CKAN is great for players and for authors:

• players can find new content and install it with just a few clicks;
• modders don't have to worry about misinstall problems or outdated versions;

The CKAN has been inspired by the solid and proven metadata formats from both the Debian project and the CPAN, each of which manages tens of thousands of packages.

What's the status of the CKAN?

The CKAN is currently under active development. We very much welcome contributions, discussions, and especially pull-requests.

The CKAN spec

At the core of the CKAN is the metadata specification, which comes with a corresponding JSON Schema that you can also find in the Schema Store

This repository includes a validator that you can use to validate your files.

CKAN for players

CKAN can download, install and update mods in just a few clicks. See the User guide to get started with CKAN.

CKAN for modders

While anyone can contribute metadata for your mod, we believe that you know your mod best. So while contributors will endeavor to be as accurate as possible, we would appreciate any efforts made by mod authors to ensure our metadata's accuracy. If the metadata we have is incorrect please open an issue and let us know.

Contributing to CKAN

No technical expertise is required to contribute to CKAN

If you want to contribute, please read our CONTRIBUTING file.

---

Note: Are you looking for the Open Data portal software called CKAN? If so, their GitHub repository is found here.

ckan's Issues

Support sha1/md5 fingerprints of downloaded files

Right now we just blindly trust any file we download is what the user's after. It would be great if we supported a fingerprint (sha1/md5) field, allowing the ckan client to test that what it downloaded matches what we expect. This would catch incomplete downloads, incorrect versions, and upstream attackers using KSP mods to deliver code to a target machine.

These fields should be optional.

Implement `ckan update`

In order to have a fully fledged client, we need a ckan update which downloads all the CKAN metadata and stores it in a local registry. This is required for dependency handling, upgrades, insalls, and generally making the CKAN client a useful piece of software.

To store CKAN files in GameData or not?

My original thoughts were to store CKAN files alongside the mods in GameData. This has the advantage of manually installed mods getting their CKAN info somewhere we can find it (see #3), and makes it easy for clients to find all the installed mods. It also means if someone deletes a mod, the metadata pertaining to that mod is removed as well.

However searching a large GameData directory is expensive. Any decent client is likely to keep its own cache of what's installed, rather than walking GameData each time. That's okay, we can make sure there's always a --reindex option that folks can use.

I'm worried about what happens if we end up with two CKAN files relating to the same mod, or a CKAN file that's just plain wrong.

I think we should do the following

• CKAN clients must not install bundled CKAN files to GameData, even if it normally would be copied as part of the install directives. (We simply skip anything with a .ckan extension.)
• CKAN clients must copy the CKAN file they are using for the install (which may have come from the zip they're processing) to the location pointed to by their first install directive¹.
• For each bundled mod that is installed, CKAN clients must generate a CKAN file and place it in the location pointed to by the first install directive of each bundle¹.
• Generated CKAN files must be named identifer.ckan (eg: RealSolarSystem.ckan).

This means that any CKAN file we find is either the result of the user doing a manual install, and it having been bundled with the mod (in which case we'll just trust that it works), or has been processed or generated by a CKAN client (in which case we hope we can trust it!).

This also means on upgrade (manual or CKAN) we maximize the chance that we overwrite the meta-data with fresh info, and minimize the chance we end up with two copies of the same meta-data.

---

¹ I know I had an issue somewhere that discussed how we'd generate metadata for bundled mods, but it's slipped my mind now.

Gracefully Handle error when someone forgets 'show package name'

I forgot to add a package name and Exceptions barfed at me :)

leon@bofh-sider:~/.local/share/Steam/SteamApps/common/Kerbal Space Program$ ./ckan.exe show

Unhandled Exception: System.ArgumentNullException: Argument cannot be null.
Parameter name: key
  at System.Collections.Generic.Dictionary`2[System.String,CKAN.InstalledModule].get_Item (System.String key) [0x00000] in <filename unknown>:0 
  at CKAN.MainClass.Show (CKAN.ShowOptions options) [0x00000] in <filename unknown>:0 
  at CKAN.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.ArgumentNullException: Argument cannot be null.
Parameter name: key
  at System.Collections.Generic.Dictionary`2[System.String,CKAN.InstalledModule].get_Item (System.String key) [0x00000] in <filename unknown>:0 
  at CKAN.MainClass.Show (CKAN.ShowOptions options) [0x00000] in <filename unknown>:0 
  at CKAN.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0 
leon@bofh-sider:~/.local/share/Steam/SteamApps/common/Kerbal Space Program$ ./ckan.exe version
v0.04-0-g06307a1

Shortname needs a better name

The shortname field in the spec is one of the most important. It's:

• The name that ModuleManager would use when detecting the mod.
• The name which other CKAN files use to reference the mod.
• The name that the packaged .ckan file should be called.
• Absolutely required for a CKAN file to be considered valid.

In other words, it's the primary identifier for the mod in question. shortname makes it sound like it's what you use when you're too lazy to type, but it's much more important than that.

I suspect identifier is probably a good choice for the field name.

Index all mods available.

In order for us to do proper dependency resolutions (#32), we need to build an index of all mods which are available. This should probably live in the CKAN registry, and is likely to be rebuilt whenever we do a ckan update to fetch a new mod list.

How complex an install do we wish to handle?

Right now we have a very simple "take X, and place it in Y". That's great for the vast majority of mods, but some are a little more tricky. I'm going to use @BobPalmer's amazing Karbonite mod as an example. (Hi RoverDude! You rock!)

Karbonite zipfile structure

The current Karbonite zip looks like this, starting with GameData. I've
indicated extensive directory structure with an ellipsis (...):

GameData/CommunityResourcePack/...
GameData/Firespitter/Plugins/Firespitter.dll
GameData/ModuleManager.2.3.3.dll
GameData/TextureReplacer/CRP_ORS.cfg
GameData/UmbraSpaceIndustries
GameData/UmbraSpaceIndustries/Karbonite/...
GameData/UmbraSpaceIndustries/Karbonite.dll
GameData/UmbraSpaceIndustries/Karbonite.version
GameData/UmbraSpaceIndustries/Karbonite_PartPack/...
GameData/UmbraSpaceIndustries/OpenResourceSystem_1_2_0.dll
GameData/UmbraSpaceIndustries/ORSExtensions.dll
GameData/UmbraSpaceIndustries/USI.version
GameData/UmbraSpaceIndustries/USITools.dll
GameData/UmbraSpaceIndustries/USI_Converter.dll

While a simple CKAN install can just copy the entirety of GameData
across, that doesn't accurately reflect what's been bundled up
here. We have:

Karbonite Core:

• GameData/UmbraSpaceIndustries/Karbonite
• GameData/UmbraSpaceIndustries/Karbonite.dll
• GameData/UmbraSpaceIndustries/Karbonite.version
• GameData/UmbraSpaceIndustries/Karbonite_ParkPack

Open Resource System (bundled, required):

• GameData/UmbraSpaceIndustries/OpenResourceSystem_1_2_0.dll
• GameData/UmbraSpaceIndustries/ORSExtensions.dll

USI tools (bundled, required)

• GameData/UmbraSpaceIndustries/USI.version
• GameData/UmbraSpaceIndustries/USITools.dll
• GameData/UmbraSpaceIndustries/USI_Converter.dll

ModuleManager (bundled, required)

• GameData/ModuleManager.2.3.3.dll

Community Resource Pack (bundled, required)

• GameData/CommunityResourcePack

Firespitter (bundled, dll only, required)

• GameData/Firespitter/Plugins/Firespitter.dll

TextureReplacer tweak, for both CRP and ORS (oh god how do we classify this):

• GameData/TextureReplacer/CRP_ORS.cfg

Recommendations

I suggest that:

• We modify anywhere that allows a file to allow a list. We may even
  deprecate the key file with filelist. For example:

    "install" : [
        {
            "filelist" : [
                "GameData/UmbraSpaceIndustries/Karbonite",
                "GameData/UmbraSpaceIndustries/Karbonite.dll",
                "GameData/UmbraSpaceIndustries/Karbonite.version",
                "GameData/UmbraSpaceIndustries/Karbonite_ParkPack",
            ],
            "install_to" : "GameData"
        }
    ]

• We change the spec and implementation so that only paths up to and
  including GameData get stripped. (In the current implementation,
  all leading paths are stripped, which would break the above example.)

Open Questions

Do we need a more general "copy X to Y", such as:

"install" : [
    {
        "file"          : "YourMod/CopyTheseIfYouHaveScanSat/SCANsat/My_Scanners.cfg",
        "install_to"    : "GameData/SCANSat",
        "strip_leading" : true,
        "requires"      : "SCANsat"
    }
]

My opinion is we should not have this. It's complex, it's hard to understand,
and I'm very sure people will get it wrong. Most projects I've seen which do
this can solve their problems with :NEEDS in ModuleManager. It's okay if some mods can't be installed via the CKAN if that means we have additional surety that CKAN-compatible mods will install correctly.

Feedback and comments very welcome. Tagging @rafl, @NathanKell and @erendrake on this as folks who have good advice on CKAN in the past, along with @parisba because twitter.

SSL - Mono on Linux Doesn't appear to have Certs by Default

leon@beast:~/git/ksp/CKAN/meta$ mono ~/git/ksp/CKAN/CKAN/CKAN/bin/Debug/CKAN.exe install FerramAerospaceResearch.ckan 
FerramAerospaceResearch:

    * Downloading FerramAerospaceResearch-0.14.1.1.zip...

Unhandled Exception:
System.Net.WebException: Error getting response stream (Write: The authentication or decryption has failed.): SendFailure ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: Invalid certificate received from server. Error code: 0xffffffff800b010a
  at Mono.Security.Protocol.Tls.Handshake.Client.TlsServerCertificate.RemoteValidation (Mono.Security.Protocol.Tls.ClientContext context, AlertDescription description) [0x00000] in <filename unknown>:0 
  at Mono.Security.Protocol.Tls.Handshake.Client.TlsServerCertificate.validateCertificates (Mono.Security.X509.X509CertificateCollection certificates) [0x00000] in <filename unknown>:0 
  at Mono.Security.Protocol.Tls.Handshake.Client.TlsServerCertificate.ProcessAsTls1 () [0x00000] in <filename unknown>:0 
  at Mono.Security.Protocol.Tls.Handshake.HandshakeMessage.Process () [0x00000] in <filename unknown>:0 
  at (wrapper remoting-invoke-with-check) Mono.Security.Protocol.Tls.Handshake.HandshakeMessage:Process ()
  at Mono.Security.Protocol.Tls.ClientRecordProtocol.ProcessHandshakeMessage (Mono.Security.Protocol.Tls.TlsStream handMsg) [0x00000] in <filename unknown>:0 
  at Mono.Security.Protocol.Tls.RecordProtocol.InternalReceiveRecordCallback (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0 
  --- End of inner exception stack trace ---
  at Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0 
  --- End of inner exception stack trace ---
  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00000] in <filename unknown>:0 
  at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <filename unknown>:0 
  at System.Net.WebClient.DownloadFileCore (System.Uri address, System.String fileName, System.Object userToken) [0x00000] in <filename unknown>:0 
  at System.Net.WebClient.DownloadFile (System.Uri address, System.String fileName) [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.Net.WebException: Error getting response stream (Write: The authentication or decryption has failed.): SendFailure ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: Invalid certificate received from server. Error code: 0xffffffff800b010a
  at Mono.Security.Protocol.Tls.Handshake.Client.TlsServerCertificate.RemoteValidation (Mono.Security.Protocol.Tls.ClientContext context, AlertDescription description) [0x00000] in <filename unknown>:0 
  at Mono.Security.Protocol.Tls.Handshake.Client.TlsServerCertificate.validateCertificates (Mono.Security.X509.X509CertificateCollection certificates) [0x00000] in <filename unknown>:0 
  at Mono.Security.Protocol.Tls.Handshake.Client.TlsServerCertificate.ProcessAsTls1 () [0x00000] in <filename unknown>:0 
  at Mono.Security.Protocol.Tls.Handshake.HandshakeMessage.Process () [0x00000] in <filename unknown>:0 
  at (wrapper remoting-invoke-with-check) Mono.Security.Protocol.Tls.Handshake.HandshakeMessage:Process ()
  at Mono.Security.Protocol.Tls.ClientRecordProtocol.ProcessHandshakeMessage (Mono.Security.Protocol.Tls.TlsStream handMsg) [0x00000] in <filename unknown>:0 
  at Mono.Security.Protocol.Tls.RecordProtocol.InternalReceiveRecordCallback (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0 
  --- End of inner exception stack trace ---
  at Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0 
  --- End of inner exception stack trace ---
  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) [0x00000] in <filename unknown>:0 
  at System.Net.HttpWebRequest.GetResponse () [0x00000] in <filename unknown>:0 
  at System.Net.WebClient.GetWebResponse (System.Net.WebRequest request) [0x00000] in <filename unknown>:0 
  at System.Net.WebClient.DownloadFileCore (System.Uri address, System.String fileName, System.Object userToken) [0x00000] in <filename unknown>:0 
  at System.Net.WebClient.DownloadFile (System.Uri address, System.String fileName) [0x00000] in <filename unknown>:0 

This fixes it, probably something to be added to the install doco.

mozroots --import --ask-remove

Add `supports` field.

We currently have a recommends list, does it make sense to have a supports list? This would be useful for things like TweakScale or RealismOverhaul, which provide support for a wide range of mods, but which don't necessary recommend them.

Having a supports field allows quick and machine-friendly answers to questions like "does this work with TweakScale", which is possibly the number one question I've seen asked of any mod I follow.

The rich metadata side of me wants this very much, and wants to include it even if we don't use it in initial implementations. The getting things done side of me says that this is a distraction and I should go back to coding. :)

Detection of fancy-pants steam installs

People can have steam-managed installs of KSP in all sorts of locations. Apparently there are registry keys we can poke at, and files we can parse to figure out where they are.

We already handle detection of default location steam/KSP installs across all platforms.

Branched from #23 .

Allow upgrades

At some point, people are going to want to upgrade their mods. There are two ways this is commonly done:

• Delete everything, and unzip afresh.
• Unzip in place.

The first means one knows exactly what one should end up with, the second means we can more easily preserve state generated by a previous install (although #15 provides a potential way of doing this).

We need to decide what the default behaviour is (right now it's unzip-in-place), but give the option to delete everything and unzip afresh.

Don't require verb to be first entry on command line

Allow some options to be specified before verbs

E.G. ckan --debug list rather then requiring ckan list --debug

Coding Standards

We need 'em. Despite having a lot of experience in software development, I'm relatively new to C#, and would like standards/conventions that cause the least friction to other developers and potential developers.

My own background (Perl) has variables_with_underscores, but I've noticed C# examples predominantly use camelCase. Is there a commonly accepted standard on whether one should capitalise the first letter? In Perl the most commonly seen convention is that class variables and methods are capitalised, and instance methods and variables are not, although the enforcement of such varies from project to project.

It would also be lovely if there's a C# equivalent of Perl::Critic, which performs static analysis and can spot not only potential programming flaws, but also stylistic ones.

Allow for Non Default/Unusual KSP Install Location

I'm weird, but I'm probably not alone :) - I have installed KSP on my SSD, but most other steam games reside in my home folder which is spinning rust.

leon@beast:~/git/CKAN/CKAN/CKAN/bin/Debug$ mono CKAN.exe scan
Setting up CKAN for the first time...
Creating /home/leon/.steam/steam/SteamApps/common/Kerbal Space Program/CKAN
Scanning for installed mods...

Unhandled Exception:
System.IO.DirectoryNotFoundException: Directory '/home/leon/.steam/steam/SteamApps/common/Kerbal Space Program/GameData' not found.
  at System.IO.Directory.ValidateDirectoryListing (System.String path, System.String searchPattern, System.Boolean& stop) [0x00000] in <filename unknown>:0 
  at System.IO.Directory.GetFileSystemEntries (System.String path, System.String searchPattern, FileAttributes mask, FileAttributes attrs) [0x00000] in <filename unknown>:0 
  at System.IO.Directory.GetFiles (System.String path, System.String searchPattern) [0x00000] in <filename unknown>:0 
  at System.IO.Directory.GetFilesRecurse (System.String path, System.String searchPattern, System.Collections.Generic.List`1 all) [0x00000] in <filename unknown>:0 
  at System.IO.Directory.GetFiles (System.String path, System.String searchPattern, SearchOption searchOption) [0x00000] in <filename unknown>:0 
  at CKAN.KSP.scanGameData () [0x00000] in <filename unknown>:0 
  at CKAN.KSP.init () [0x00000] in <filename unknown>:0 
  at CKAN.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.IO.DirectoryNotFoundException: Directory '/home/leon/.steam/steam/SteamApps/common/Kerbal Space Program/GameData' not found.
  at System.IO.Directory.ValidateDirectoryListing (System.String path, System.String searchPattern, System.Boolean& stop) [0x00000] in <filename unknown>:0 
  at System.IO.Directory.GetFileSystemEntries (System.String path, System.String searchPattern, FileAttributes mask, FileAttributes attrs) [0x00000] in <filename unknown>:0 
  at System.IO.Directory.GetFiles (System.String path, System.String searchPattern) [0x00000] in <filename unknown>:0 
  at System.IO.Directory.GetFilesRecurse (System.String path, System.String searchPattern, System.Collections.Generic.List`1 all) [0x00000] in <filename unknown>:0 
  at System.IO.Directory.GetFiles (System.String path, System.String searchPattern, SearchOption searchOption) [0x00000] in <filename unknown>:0 
  at CKAN.KSP.scanGameData () [0x00000] in <filename unknown>:0 
  at CKAN.KSP.init () [0x00000] in <filename unknown>:0 
  at CKAN.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0 

A symlink makes things happy for now.

leon@beast:~/.cryptfast/git/CKAN/CKAN/CKAN/bin/Debug$ mono CKAN.exe scan
Setting up CKAN for the first time...
Creating /home/leon/.steam/steam/SteamApps/common/Kerbal Space Program/CKAN
Scanning for installed mods...
Creating /home/leon/.steam/steam/SteamApps/common/Kerbal Space Program/CKAN/downloads

RFC: Automated CKAN file generation.

Discussion

I've been asked many times how one might go about adding CKAN support into existing build processes. This is essential for everyone; updating things by hand is unnecessary and error-prone.

Embedding the CKAN files in the archives themselves is an option, and easy for us, but it also sucks because it means mod authors have to manually update the version number.

A much better system is to hook into existing release systems. The big ones¹ are:

• Github Releases ( #2 )
• KerbalStuff ( #69 )
• AVC

I propose that authors can include a CKAN fragment file in their releases, which is essentially all the CKAN meta-data, but without the data we can derive from the release system (which is at least the version number for Github, and potentially the KSP target for KS and AVC).

Since most CKAN data changes rarely, the vast majority of mods will just include their fragment file and forget about it.

Spec

I'd suggest the following fields:

• version_from
• ksp_version_from

Each of which is an enum that may contain the values Github, KerbalStuff, or AVC. In each case, we enforce that the appropriate details are listed in the resources section, so our bot knows where to look.

It would be an error to have both version and version_from, and ksp_version and ksp_version_from in the same file.

After compiling the new file from the fragment and discovered meta-info, we would verify that it validates, and has the identifier we expect it to, before automatically adding it to the CKAN.

Questions:

• Does this sound like a good idea?
• What do we call the fragment file? (Not .ckan, because it won't have a full set of meta-data. .ckaf maybe ?)
• Any other comments / suggestions?

---

¹ I don't believe curse gives us APIs, or makes our lives easy, but I'd be delighted to find out otherwise.

Cached Download Fails

Where is the cache stored? I should start perusing the code :)

leon@beast:~/git/ksp/CKAN/meta$ mono ~/git/ksp/CKAN/CKAN/CKAN/bin/Debug/CKAN.exe install FerramAerospaceResearch.ckan 
FerramAerospaceResearch:

    * Using FerramAerospaceResearch-0.14.1.1.zip (cached)

Unhandled Exception:
ICSharpCode.SharpZipLib.Zip.ZipException: Cannot find central directory
  at ICSharpCode.SharpZipLib.Zip.ZipFile.ReadEntries () [0x00000] in <filename unknown>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipFile..ctor (System.IO.FileStream file) [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: ICSharpCode.SharpZipLib.Zip.ZipException: Cannot find central directory
  at ICSharpCode.SharpZipLib.Zip.ZipFile.ReadEntries () [0x00000] in <filename unknown>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipFile..ctor (System.IO.FileStream file) [0x00000] in <filename unknown>:0

Non cached does work though :)

leon@beast:~/git/ksp/CKAN/meta$ mono ~/git/ksp/CKAN/CKAN/CKAN/bin/Debug/CKAN.exe install kOS.ckan 
kOS:

    * Downloading kOS-0.14.zip...
    * Installing GameData/kOS

Done!

Handle no-op upgrades

Scenario: The user has a mod installed. A new version of KSP is released. It's discovered the mod works fine on the new version, so updated metadata is released signalling its support.

An upgrade here doesn't have to do anything, but because we save the meta-data that existed at the time the module was installed, we may show it as being out-of-date when it's really not.

As part of ckan upgrade, it would be nice if we were able to handle these no-op upgrades and just update the stored metadata, rather than uninstalling the mod, and then reinstalling exactly the same mod. That may not be as easy as it sounds, because we'll still want to check relationships, install stanzas, and the like.

Since we already cache downloaded mods, a complete reinstall to update the metadata should be fine, even if it's sub-optimal, but this would be nice to have.

Support installation of tutorials.

Wherever they happen to live right now.

(With thanks to @Majiir for the suggestion)

KSP version fields

Right now, we have min_ksp and max_ksp, however many mods target just a single KSP version. We may wish to allow a ksp_version field.

We should also probably allow a version to just be 0.25, which would be compatible with any 0.25.x release.

I also know I had a great discussion about versions in general, but I cannot for the life of me remember who it was with, or what about. If I had to guess, it would be @rafl, but my ability to find said conversation seems to be lacking right now.

---

From the above, and also the discussion below.

• Update the version field as described above.
• Detect/ask the user if they're using a 64-bit build.
• Add an optional field to the CKAN spec specifying architecture. (Defaults to 'all').
• Update the standard names by which we cache files and store metadata. (Eg: Foo-1.23-x64 and Foo-1.23-x32 for architecture dependant mods; probably fine as-is for universal mods.)

Where do we store the CKAN repo?

Right now we've got sample files in the meta directory of this repo, but going forward that's probably not ideal. These files are likely to be updated often, either by mod authors or automated tools, and by having them here we clutter the commit log of changes to the code and design with changes to the data that we use.

I'm going to suggest we create a github organisation (KSP-CKAN, since CKAN is already taken), migrate this repo to there, and have another repo which is just for the meta files. This means:

• The CKAN client can dowload a single zip to get all the metainfo it needs. This is is fast, and easy, especially because we already have code that downloads and processes zipfiles.
• We can more easily implement code reviews before merge, especially of my own code, by sending pull requests to the repo. (HT to @eggrobin for the suggestion.)
• Teams are better than people. I actually want to be playing KSP, and having a org makes it easier for me to hand everything to another person when I get to do so. :)

Signal when only *part* of a mod has been bundled.

It's common to see the FireSpitter dll bundled with mods (which is permitted by its license), but bundling the entire FireSpitter mod is forbidden.

We need a way to indicate when part of a mod has been bundled, as users are likely to be very surprised if we upgrade them to a full install unexpectedly.

Replace FileSystem.IsDirectory with Directory.Exists

Thanks to @johndalton at https://twitter.com/johndalton/status/518588353805963264 .

This will reduce our line count and simplify a bunch of things. Hooray!

Let users see mod availability for other versions

I'd love something that lets me know which mods support a future (or past) version of KSP, without needing that installed. This means if I'm thinking of upgrading to X, I can see which of my mods are already updated for X, and which ones I have to hope and see.

I have no idea what the interface for this would look like.

Allow installation from already downloaded zip files

The perl implementation was already doing this with:

ckan -f SomeMod.zip -i SomeMod.ckan

We should add the same functionality to the C# code, so it doesn't need to re-download files we've already got.

Package Version

Is there a plan to incorporate a package version, i.e. a version number which indicates the revision of the package itself, not the upstream mod?

For example, a mod may release version 1.7 which produces the first package 1.7-1, however the packager forgot to specify the license so they repackage as 1.7-2, but then they realized they forgot to specify a dependency so they repackage as 1.7-3, then when the mod releases version 1.8, the packager releases package 1.8-1.

Conflicts processing

This should just be a Simple Matter Of Code. If we're installing a mod that conflicts with an already installed mod, or a mod that we're going to install at the same time, then we bail out.

Relates to #32, insofar that they'll use much of the same infrastructure.

Update to allow a multi-level CKAN structure.

Right now we're putting all CKAN files in a single directory, and I'm
writing the client to only look in a single directory. However it
might make sense to allow sub directories, as this will allow for
greater structure in the future.

For this, we need to:

• Update the client to search sub-dirs.
• Update travis tests to search sub-dirs.
• Move any WIP files onto branches, until they're ready to be merged.

Flip licenses in spec to Debian spec

The debian spec does a better job of specifying all the licenses out there. We should use it, or proxy to it, for our own spec.

Allow user to specify KSP location.

Right now, if the client can't detect the user's KSP install, their only open is to copy the CKAN client into the same directory.

We'd like to allow the user to specify where their KSP version is installed. There are two parts of this:

• Provide a switch (--ksp= or --kspdir=) that allows the user to specify the full path to the install this invocation should use. This setting is not permanent.
• Provide a way for the user to set the default KSP install that will be used. This setting should be stored in the Windows/Mono registry, so it can be used on each invocation.

I think that autodetected KSP installs should be cached in the registry, because we don't want which KSP we use to change just because we've changed our scanning rules.

If the CKAN client is actually installed alongside a copy of KSP itself (which can be the case with portable installs), then I think we should not use the Windows/Mono registry. That will surprise owners of portable installs when we use a completely different KSP to what's on their media.

Branched from #23, and relevant to @Wizarth and @techman83.

NUnit testing

It feels weird to have code without tests.

I'm not sold on NUnit, but apparently it can be configured to work with travis-ci, and I love travis.

CKAN file naming conventions

My original plan was to have the file named META.json and placed in the root of a distribution's zip. This provides a simple, well-known place for tools to look. However it also means that users installing mods manually end up with a whole bunch of warnings about files wanting to overwrite each other, and that sucks.

Instead, I think it's going to be best to have files named shortname.ckan, where shortname is from the CKAN definition itself (and matches the name that ModuleManager would see the mod as). Hence one would have kOS.ckan and RealSolarSystem.ckan as filenames. This has two advantages:

• Users no longer have to worry about overwriting files on unzip
• Simply having these in GameData means that we can easily tell later on which mods have been installed.

The second point here is important, because it means CKAN clients can pick up previously installed mods that were placed by hand. It also minimizes the number of dependencies a client needs to run; we already know that any client already has the libraries to parse JSON and walk directories. The CKAN metadata is also living there alongside the mods themselves, meaning folks who have multiple KSP installs (like yours truly) doesn't have them interfere with each other.

It may make sense for an installer to place these files in GameData/CKAN, simply to keep them tidy, but I don't think they should have to live there.

KSP itself ignores files other than .cfg, models, and textures, so it will never try reading a .ckan file, which is what we want.

As always, feedback and thoughts are very much appreciated.

List outdated/incompatible mods

Since we know what version of KSP the user is running, and what version a mod is taretting, we should update ckan list so that it will highlight mods which are out of date or incompatible (especially relevant if the user flips to win_x64).

We may wish to have ckan upgrade with no arguments apply all available upgrades (a la apt-get upgrade).

KerbalStuff support

KerbalStuff has a really nice API. Once we figure out how we're going to base release detection, we should make sure we support KS.

For an example of what we can get out: https://kerbalstuff.com/api/mod/21 (best viewed with a JSON pretty-printer).

No-dependency build script

Right now bin/build requires a couple of Perl modules, which is
cool for the Perl-heads working on this, but not so cool if you've
never used Perl before.

The fatten utility lets us bundle those dependencies into the
sript itself. This is a reminder to @pjf to add a pre-commit
hook or similar magic that reads the current build script
(which we'll probably rename as build.lean), and produces
a fattend version.

Allow CKAN files to be generated from github releases

The CKAN meta-file requires that a version and a download link be provided; this allows simple installers and mirror bots (which must check the license field) to be written easily. However it's a pain having to update the download URL and version every time one makes a release.

A potential solution is to provide a partial META.json file in a distribution, with some sort of magic releases flag enabled. This signals to external tools that the file contents should be combined with the release info from github (accessible via the API) to produce the final file. This means authors can continue to make releases as normal, and the CKAN index can do the hard work of filling in version numbers, download links, and changelog entries.

It should be noted that the resulting CKAN file can't be included in the released zip, since that zip has already been created and released. That's okay, because bundling CKAN files isn't required for operation, but it's something we encourage folks to do to allow for tools to more easily detect which mods are installed.

CKAN.dll

It would be great to give people a CKAN.dll they can link against. Especially for @AlexanderDzhoganov, who wants to write a GUI! <3 (related: #44)

Provide a way to specify licenses for each contained file.

Some mods will have multiple licenses for different assets (eg, code with one license, graphics another). The debian spec already has a way of managing this, we should do the same.

Luckily, the Debian spec makes it really easy, because it uses file patterns. Thus *.png *.jpg or images/* provide easy ways to tag a number of files at once.

Many thanks to @eggrobin for raising this. <3

Version comparison routine

The routine is finished, but getting it to work with C# is a pain. It's almost 2am, so I'm going to have to continue this on the plane.

Spec: Indicate when an upgrade will be save-breaking.

Some mods are save-breaking, and can't be upgraded automatically (RSS 8 is an example). Since ckan update; ckan upgrade is likely to be common, we don't want to surprise users by breaking their saves.

We need a way to indicate in the CKAN file when a mod is not safe to upgrade.

RSSTextures Fail to Download

I've checked the certs are present and checked the site with Firefox and all is good. Anyone else noting this issue?

All the others seem to download correctly.

leon@beast:~/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program$ ./ckan.exe install RSSTextures8192 --debug
94 [1] DEBUG CKAN.KSP (null) - Checking if KSP is in my exe dir: /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program
105 [1] INFO CKAN.KSP (null) - KSP found at /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program
105 [1] DEBUG CKAN.KSP (null) - Checking HKEY_CURRENT_USER\Software\CKAN\GameDir for KSP path
115 [1] DEBUG CKAN.KSP (null) - Found KSP dir in /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program via registry
116 [1] DEBUG CKAN.RegistryManager (null) - Using default CKAN registry at /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program/CKAN/registry.json
116 [1] DEBUG CKAN.RegistryManager (null) - RegistryManager not yet active, loading...
198 [1] DEBUG CKAN.RegistryManager (null) - Loaded CKAN registry at /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program/CKAN/registry.json
213 [1] DEBUG CKAN.RegistryManager (null) - Using default CKAN registry at /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program/CKAN/registry.json
RSSTextures8192:

    * Downloading RSSTextures8192-1.0.zip...
215 [1] DEBUG CKAN.Net (null) - Downloading https://nabaal.net/files/ksp/nathankell/RealSolarSystem/Textures/8192.zip to /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program/CKAN/downloads/RSSTextures8192-1.0.zip
1243 [1] DEBUG CKAN.Net (null) - Removing /home/leon/.cryptfast/Games/SteamLibrary/SteamApps/common/Kerbal Space Program/CKAN/downloads/RSSTextures8192-1.0.zip after web error failure

Oh no! Our download failed!

    Error getting response stream (Write: The authentication or decryption has failed.): SendFailure

If you're on Linux, try running:

    mozroots --import --ask-remove

on the command-line to update your certificate store, and try again.


Unhandled Exception:
CKAN.MissingCertificateException: Exception of type 'CKAN.MissingCertificateException' was thrown.
  at CKAN.Net.Download (System.String url, System.String filename) [0x00000] in <filename unknown>:0 
  at CKAN.Net.Download (System.Uri url, System.String filename) [0x00000] in <filename unknown>:0 
  at CKAN.ModuleInstaller.Download (CKAN.CkanModule module, System.String filename) [0x00000] in <filename unknown>:0 
  at CKAN.ModuleInstaller.CachedOrDownload (CKAN.CkanModule module, System.String filename) [0x00000] in <filename unknown>:0 
  at CKAN.ModuleInstaller.Install (CKAN.CkanModule module, System.String filename) [0x00000] in <filename unknown>:0 
  at CKAN.MainClass.Install (CKAN.InstallOptions options) [0x00000] in <filename unknown>:0 
  at CKAN.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: CKAN.MissingCertificateException: Exception of type 'CKAN.MissingCertificateException' was thrown.
  at CKAN.Net.Download (System.String url, System.String filename) [0x00000] in <filename unknown>:0 
  at CKAN.Net.Download (System.Uri url, System.String filename) [0x00000] in <filename unknown>:0 
  at CKAN.ModuleInstaller.Download (CKAN.CkanModule module, System.String filename) [0x00000] in <filename unknown>:0 
  at CKAN.ModuleInstaller.CachedOrDownload (CKAN.CkanModule module, System.String filename) [0x00000] in <filename unknown>:0 
  at CKAN.ModuleInstaller.Install (CKAN.CkanModule module, System.String filename) [0x00000] in <filename unknown>:0 
  at CKAN.MainClass.Install (CKAN.InstallOptions options) [0x00000] in <filename unknown>:0 
  at CKAN.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0 

Detect KSP version

This is pretty important so we don't install incompatible mods.

I'm not sure of the best way to detect the version, but options could include:

• md5/sha1 sum of the executable¹
• Looking in the README file, because it's right there
• Asking the user

¹ This is not guaranteed to be reliable, as some people (like me) apply binary patches to their executables, usually to fix bugs.

Handle 'show' gracefully when plugin not installed

Not installed

D:\>ckan show AJE

Unhandled Exception: System.Collections.Generic.KeyNotFoundException: The given
key was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at CKAN.MainClass.Show(ShowOptions options)
   at CKAN.MainClass.Main(String[] args)

Installed

D:\>ckan install AJE
AJE:

    * Using AJE-1.5.zip (cached)
    * Installing AJE
ModuleManager 2.2.0 already installed, skipping bundled version 2.3.5

Done!

D:\>ckan show AJE
Advanced Jet Engine (AJE) version 1.5

Support multiple KSP installs.

(Text mostly taken from @Wizarth in #23)

It's a common use case for modders to have multiple installations/copies of KSP.
I suggest a per-user index of installs. This is enough data to point one instance of
CKAN to different installs of KSP, each with it's own per-install data store.

We can safely assume any install folder will be writable by CKAN (or else how would we
install mods).

ckan list-installs
ckan add-install name path/to/install
ckan modify-install name changed/path/to/install
ckan remove-install name

This then allows users to specify which install they want to operate against. If one is not
specified, assume special case default.

ckan --ksp=0.24_IQ list

As per #28, if no installs defined, CKAN should attempt to automatically detect default. (E.G. Steam).
If not possible to detect, user should be prompted to define default install.

Install data should be stored in the Windows/Mono registry, since we can't access (or even create) the CKAN registry until we know which install we're dealing with.

Currently being worked upon by @Wizarth, unless he claims otherwise. ;)

Release Roadmap

Here's a map of what we need to do before we can start pushing CKAN to the world.

Meta

• Stabilise spec

Client

• Support basic download and install
• Cache download zip files
• Detect/Cache installed CKAN meta-data
• Support downloading CKAN index
• Build test framework ( #5)
• Make sure we're easy to build under Visual Studio and MonoDevelop

Cloud

• Find hosting for CKAN index (estimated size ~ 2 MB, probably github for now)

Module Dev

• Invite friendly mod authors to participate (comment below if this is you!)
• Prepare CKAN files for popular mods
• Trigger auto-generation of CKAN files on module release (#2)

Add support for installing to Ships

This is now in the metadata space. install_to can point to to Ships, which allows for installing into the VAB and SPH. We should check for this and act accordingly.

Unless the VAB and SPH allows subdirs, we should not create directories as part of this process.

Allow installing files to GameRoot.

A few mods require this, including one by
@AlexanderDzhoganov/ksp-advanced-flybywire, and the head-tracking one.

Preserved files

It seems to make a lot of sense to mark some files as preserved, which is going to be most useful for config, but may make sense for other files as well. Preserved files don't need to be included in a distribution, we simply don't delete or overwrite them if they exist when installing or upgrading mods.

Good examples of preserved files are the TACLS and RT2 config overwrites done by RealismOverhaul, but a lot of other mods provide the capability to preserve settings between upgrades.

The rules for preserved files should be pretty simple:

• When upgrading a mod, we don't remove preserved files.
• If a mod tries to upgrade a file marked as preserved, we query the user as to what should be done.

The second follows the Debian practice of not overwriting files in /etc that the user may have customised. Care should be taken to ensure the file contents are actually changing before prompting the user.

Preserved files should be used extremely sparingly.

Dependency processing

We need to be able to do dependency processing. Right now, we just check if your dependencies are installed, and fail if they're not. That's not ideal.

Real Solar System is a good example of something with "hard" dependencies here. It depends upon its texture files, but there are multiple texture files available, and they in turn depend upon having the RSS base installed.

I think the following algorithm should work:

• Compose a list of everything we already have installed.
• Add to that list everything that we're installing.
• Walk through what we're installing, and see if their dependencies are satisfied by the (installed+installing) list we've created.
• Anything which isn't satisfied gets added to the list of things to install, and we repeat.
• Make sure all of the above uses both the module names and the provides field, as mentioned in the spec.

This means installing RSS with a texture pack works fine. Installing just an RSS texture pack will pull in the dependency on RSS, which should work fine.

It's important that we do allow circular dependencies, because they already exist (X + Y must be installed together type things).

In order to add dependencies, we need features to allow the client to be aware of all mods it has available.

Allow multiple distributions per file?

Let's take RSS as an example. I would like to release a file that includes both RSS itself and some required dependencies. However, each of those dependencies are themselves distributions.

As not all users will use the preferred means of distribution, and we would like mods installed through means other than this to be recognized by this, it seems of decent importance that this (issue) be possible.

Build a GUI.

Command-lines are great for all the unix nerds playing KSP (and there's
a lot), but our typical user is going to want a GUI.

This is something which I personally lack experience in, so my hope
is to look at the rest of the Internet with big puppy-dog eyes.

We should probably change our Console.WriteLine calls to instead
call something else, (KSPUser.Notify(), for example), so it's
easy to send output to the right place based upon the mode being
used.