ponylang / corral Goto Github PK

Start with nightly. From there, incorporate as part of the release process.

Should be released once #80 is merged; that's a nasty bug for folks to work with. I want to get a release out ASAP.

This should be like Github, but maybe there are nuances to deal with.

Running corral init in a new directory does not create a bundle.json file on my machine. I believe this is because Bundle.modified is set to false upon construction of a new Bundle instance, and calling Bundle.save() checks modified before doing anything.

I think I can fix this with a one-liner. Stay tuned.

We would need to define a clear notation for indicating the kind of version constraint.

For now, only Linux using ponyup.

should include:

• how to manage dependencies with corral
• how to install
• how to build from source

this should be at least as good as the information that is currently in the stable README.

Pull in https://github.com/ponylang/appdirs as a submodule, and use it to finish #28

With stable it was possible for me to debug ponyc while invoking it from the stable env.

stable env lldb ponyc -- -p ./lib/osx/  --pic 

With corral this doesn't seem to be possible:

beast:pony.bzip2 rjbowli$ corral run -- lldb ponyc
run: lldb ponyc
  exit: 0
  out: (lldb) target create "ponyc"
Current executable set to 'ponyc' (x86_64).

  err: 
beast:pony.bzip2 rjbowli$ 

I don't know if there is a work-around I am missing? If not, I will probably dig into fixing this tomorrow as this is a showstopper for me.

Current parsing is optimistic and completely ignores unknown elements, and wrong types (like array instead of object). The parser should check that there are no unknown elements and all provided elements are of the correct type. Informative errors should be printed or returned.

We currently have these fields in the bundle-level info object:

• name
• description
• homepage
• license
• version
  Are these useful? We might also want:
• Ponyc compiler version (range)
• Information about non-source artifacts of the bundle. (eg, native code)
• ?

To replicate:
corral run /path/to/alternate/ponyc

cmd_run.pony:39 checks that the first argument to corral run is in $PATH. This might cause frustration if people test with different versions of ponyc than the one in $PATH.

One solution to this might be to replace that check with a more permissive one, like making sure the file specified by the argument exists (or is in $PATH) and is executable.

The current behavior is to start at CWD and search up until a bundle.json is found. We should have an option to supply the directory to load the bundle.json from, bypassing the search.

Ponyc 0.34.0 has removed the Unsupported error from ProcessMonitor. It was never used. However, corral referenced it to get a displayable error message in a match statement here: https://github.com/ponylang/corral/blob/master/corral/util/action.pony#L146

When 0.34.0 is released, we'll update corral to remove the match statement and call a new method on the ProcessError (probably called string) that will get a String representation of the error.

@kulibali is in the process of adding the string error message support to ProcessMonitor prior to the 0.34.0 release.

I believe this is mostly adding in order as we verify everything:

1- PR support for FreeBSD
2- Nightly builds
3- Release builds

Because GitHub actions don't have FreeBSD support, we'll need to use CirrusCI.

To allow local development of Pony bundles, and replacement of public bundles with local or private shared versions. It would be convenient if this aliasing could be a simple override mechanism to define aliases for bundles referenced in deps that are otherwise unmodified. The aliases should apply to transitive deps too. Something like:

{
  "deps": [{ "locator": "github.com/ponylang/corral-test-repo.git/bundle2" }],
  "aliases: [{ "locator": "github.com/ponylang/corral-test-repo.git/bundle2", "alias":  "../bundle2" }]
}

But it could also be an alias field in each dep, or both. And, with either approach, Corral could load them from a distinct file that could be kept local for local development.

Similar to the philosophy of Go, the root project should be in charge of determining the final version resolution among competing constraints. Producing a top-level master lock file will serve as a master manifest and provide repeatable builds.

corral fetch should then refer to the locked versions and fetch quickly and deterministically.

Corral uses two special locations for storing things:

• repo_cache: Central location (per user or workspace) for caching repositories (without indexes or workspaces)
• corral_base: Per-project location for holding the checked-out source of each dependency of the project.

Currently, these are both being put into . as:

• ./_repos
• ./_corral

I think these are good names, but we should add a flag / env var to control the repo cache location, defaulting to $HOME if possible. And, we should make sure we situate the corral_base next to the bundle.json in all cases. (ancestor of CWD or a dir passed on the CLI)

A couple of sub-issues related to how corral makes use of the semver solver:

• If the solver returns more than one solution to a version constraint, how should we choose? Many dependency tools just pick the latest solution, but we could have other options. At least pick one.
• Update should use the solver across the whole graph, and not just individual deps in isolation.
• Non-semver constraints could be added to the semver solver, as a literal comparison, to allow the solver to operate across the whole graph.

There are a number of to-do's that are listed in the README that should be issues.

This is a master list of issues that should be addressed in order for Corral to meet feature parity and be able to replace Stable.

Must-haves:

• #26 Must support simple tag / branch references
• #32 Remote GitLab & Bitbucket support
• #33 Local git repos
• #34 Local bundles (w/o vcs)
• #59 Update recursion is a must-have, master lock is a should-have
• #55 Windows builds

Should-haves:

• #7 Should have at least simple usage info in a howto
• #27 The -n option will help people learn how Corral works
• #28 It might be messy to move the repo cache later.
• #62 Don't want recursion crashes.

The fetch begins and downloads the dependent repos, but then fails before the checkout completes.

Currently, Corral has a mix of writing user output to env.out (which may be OK), and more developer-focused debug output to env.err and to a log stream. Also, the user output should support quiet/normal/verbose levels as described in the CLI options.

We should formalize the output usage and make the code consistent. I do think the user vs developer separation is useful, so maybe we can use two instances of the Log class:

• the current Log for developer messages could remain as-is, possibly renamed to dev_log.
• a new Log could be configured for user output, and use a leaner LogFormatter. The CLI options would map to log levels, like quiet=>err or warn, normal=>info, verbose=>fine.
• all env.err use should be updated to one of the above logs.

For local paths we should be able to skip the clone and fetch steps, and version/tag queries and checkout using symlink or copy or logical reference path in PONYPATH.

Maybe this can be done via a special "local" VCS.

I need to set up our standard CI.

With issue #23 closed, we should be a new release of Corral. Current plan: to do at the same time as the next ponyc release. See ponylang/ponyc#3363.

Local Override: Stable allows local bundles that are just [relative] path entries in the bundle, resulting in canonicalized paths in the PONYPATH. How can we do this while also tracking a remote location to allow local development of a published library? That is: use remote, fork remote and work on local, commit fork to remote and go back to using that. This could be a simple matter of tracking the local location as a temporary override. Or, we just let the developer update (remove+add) the bundle with the alternate locator. Or, use a more general alias technique like go mod?

My idea: Use a general alias technique in a separate file (e.g. local-overrides.json). This way the developer(s) could have this file but do not check it into the repo. So the users cloning the repo won't have the overrides (but could add them easily if needed).

For local repos we should be able to skip the clone and fetch steps, and just do a direct checkout.

Currently corral doesn't have the version command.

It should print corral 0.1.0 on corral version.

Make sure that no commands modify anything when -n (nothing == true) is set.

There's a problem right now with using corral with various dependencies. If that dependency has a "Stable" format bundle.json then corral will try to operate on it and choke.

I think that we should take 1 of 2 possible approaches

stable format support

For a while, we should add support to handle stable format bundle.json files to corral so corral can do its thing with them in terms of handling transitive dependencies.

don't use bundle.json

The other option is to make a clean break and change the name of bundle.json for corral to something like corral.json.

We currently have a copy of the core of https://github.com/btab/pony-semver library as package within Corral. We should propagate our changes back upstream, and change Corral to use this library via a git submodule.

We can use @btab's library directly and have him add a release tag for us. Or, we could use my fork https://github.com/cquinn/pony-semver.

Currently, logging for programmer debugging is on by default and noisy. And, the output for user consumption doesn't respect the --quiet or --verbose flags and gets all mixed in and badly formatted.

Proposal:

1. Add a new --debug flag for setting the programmer debugging log level, defaulting to LvlNone.
2. Create a second Log instance for user output, with the simple formatter that sets its level based on the quiet and verbose flags to one of 3: --quiet => Warn. (default) => Info. --verbose => Fine. Add this to Context.

Could also add colors to one or both loggers like Ponyup.

Currently,

using corral run ponyc results in an error rather than the project being built.

To ease the transition to using corral, @rkallos and I believe that corral run should be able to run without a bundle.json being present.

@cquinn care to weigh in?

Currently all the VCSes get passed Env which allows them to print to stdout, stderr etc.

Unfortunately this also gives me ambient auth which we probably do not want. Given that Env gives access to root it would allow for anything to be done, open files, open sockets, etc.

I think we should narrow down to passing only the minimally needed env usage and work from there and add additional access as needed.

It almost assuredly doesn't.

Given this will touch a lot of code, if you can't do this in a fairly quick go, to avoid merge conflicts, it would be best to do in short increments that can be done off of latest master and merged quickly.

Perhaps a module at a time.

This is related to the tree structure under _corral, and whether to flatten all transitive dependencies to the top level there (with name flattening) or nest them under the immediate deps. I am pretty sure that this is done (as flat) already but creating an issue from the README just in case.

Pretty sure this is completed, but will double-check.

Currently, the git cmd uses "regex" functionality. Once the ci branch is merged, we want to remove the vendored regex functionality that depends on PRCRE2 and use something that has no external dependencies to make bootstrapping corral easier.

To that end

• Regex using code should be rewritten
• README should have "how to install pcre2" removed
• "vendored" regex library should be removed

Release date is Monday May 4. I'm going to give us a day of nightlies to make sure everything is working before doing the release.

There's no version or version info right now. Will need to be able to set.

We need tests. We don't really have any. A good start would be to copy/borrow the things we were testing with Pony-stable over to corral. There will be some conversion needed but its a start.

When we locate the tag/version of a commit that we want during update, we should also record its commit hash in the lock file. We can then verify this when we fetch.

The Golang module system does this to avoid surprises caused by people moving tags around out from under people and quietly breaking builds.

If we decide to support these VCSs.

CmdFetch needs a mechanism to track transitive deps that it has visited to prevent re-visiting and potential infinite recursion.