oyvindberg / bleep Goto Github PK

For a platform to be "supported" by bleep, it needs graalvm native-image support, and github action runners.

If it doesn't have that, I personally will not do anything to make it work. However, you can make it possible to use it if you volunteer to add some fallback support where bleep runs on the JVM.

At least two things need to be done:

• bleep setup-ide needs to construct a java command with the relevant classpath instead of calling a native binary
• the launcher in Main needs to download relevant jar files and classload them

we have all the tools necessary to create a fantastic TUI now - time to experiment!

this worked in the template I used at https://github.com/note/teleport-scala

rewrites are already in place, but some surface is missing before scripts can do arbitrary rewrites before bootstrapping

note: rewrites are named, so any combination of rewrites will yield a new build folder.

for instance if you bootstrap a build with add-jmh and upgrade-deps the bloop folder would end up in something like ./.bleep/builds/add-jmv__upgrade-deps/.bloop

I have a project where I depend on native artifacts which have a different classifier per platform. So in a bleep.yaml, I have the following dependency:

    - module: org.bytedeco:pytorch:1.13.0-1.5.9-SNAPSHOT
      publication:
        classifier: linux-x86_64 # could also be macosx-x86_64 or another platform or even a variant like linux-x86_64-gpu
        ext: jar
        name: pytorch
        type: jar

I can set the classifier manually and it totally works fine, but now I'd like to automatically detect the current platform and set the classifier accordingly (JavaCPP already has a function that can be used for that). To make things more complicated, there's also a gpu suffix in the mix which I'd like to be able to append to the classifier as an additional manual configuration.

For sbt there's a plugin called sbt-javacpp which does exactly that (among a few other things).

I'm fairly new to bleep, so I'd like to ask whether this is doable and what would be the best way to do it. I.e. using build rewrites?

there are already bash auto completion in place - the same structure can hopefully be used for other shells.

should include a command similar to bleep install-tab-completions-bash

the most important pages are not written yet:

• describe model
• cross builds
• templates

more details / clarify:

• writing scripts
• porting sbt plugins

I'm much more inclined to provide a mechanism by which you can use anything which can output json/yaml.
Let's say we perform these steps on startup:

• look for bleep.yaml , use if it exists
• look for bleep.* with the executable bit set. if it exists run it, and pipe the output of the program through the normal json parser

wouldn't that be neat? That way you can for instance code the build in typescript, and you pay for it by having to start node on every invocation of bleep. Or well, we could certainly cache the output by the timestamp of the executable file actually.

Originally posted by @oyvindberg in #15 (comment)

When configuring authentication as described in #255, I had to include the setting compileServerMode to make the config valid. If I use only this as a config.yaml:

authentications:
  <some-resolver-url>:
    user: <the-user>
    password: <the-password>

I get:

📕 Couldn't parse json file /Users/kristof/Library/Application Support/build.bleep.bleep/config.yaml: Missing required field: DownField(compileServerMode)

I think it would be great if this config could somehow be merged with the defaults, so that it only needs to contain things that are changed by the user.

• do we need to string-template in any values?

• rename project
• combine projects (including moving files)
• add new project
• add cross-compilation to project)

Tried on an Ubuntu 22.04 machine.

❯ bleep import
📗 Will call sbt multiple times to retrieve information about the existing build. Grab some popcorn as it may take a while! [t => 0 ms]
📗 Calling sbt to discover projects... [t => 4 ms]
📙 [subprocess: sbt discover projects]: downloading sbt launcher 1.6.2 [t => 92 ms]
📙 [subprocess: sbt discover projects]: mkdir: cannot create directory '/.cache': Permission denied [t => 93 ms]
📙 [subprocess: sbt discover projects]: mkdir: cannot create directory '/.cache': Permission denied [t => 95 ms]
📙 [subprocess: sbt discover projects]: cat: /.cache/sbt/boot/sbt-launch/1.6.2/sbt-launch-1.6.2.jar.sha1: No such file or directory [t => 96 ms]
📙 [subprocess: sbt discover projects]: shasum: standard input: no properly formatted SHA checksum lines found [t => 109 ms]
📙 [subprocess: sbt discover projects]: failed to download launcher jar: https://repo1.maven.org/maven2/org/scala-sbt/sbt-launch/1.6.2/sbt-launch-1.6.2.jar (shasum mismatch) [t => 109 ms]
📕 Failed external command 'sbt discover projects' with exit code 2. See log file for exact command [t => 110 ms]

There seem to be some path issues. It does not recognize the sbt version and makes and absolute path out of the cache dir. I tried with multiple projects with the same result. I also tried to import one of the projects on a Mac machine and the import did work fine here.

I keep experimenting with bleep for my project, and running into smaller issues that could be totally my fault using it the wrong way but I think might still make sense to report.

I did an import of my sbt build which sets some java properties when running the main class in a forked JVM, like so:

    fork    := true,
    javaOptions ++= Seq(
      "-Dfoo=bar",
    ),

On import, bleep will generate s.th. like this:

    platform:
      jvmOptions: -Dfoo=bar
      name: jvm

but the java properties are not set on bleep run. So I tried setting jvmRuntimeOptions: "-Dfoo=bar" instead without really knowing the difference, but it still does not pick up the properties. Interestingly though, if jvmRuntimeOptions are set, and I run bloop run manually in.bleep/builds/normal, it does pick up the properties.

Here may be the wrong place to ask, because I know Bleep uses Bloop heavily for many things, but I'm still curious.

What is the caching story for Bleep? How does it prevent unnecessary computation taking place over and over? For example does it cache the results of tests?

Also, where is such cache located? Can it be saved/stored somewhere and then later and/or elsewhere unpacked/re-hydrated and reused?

https://json5.org/

I'm trying to check if I can set Java properties on my project with your fixes from #253 by running bleep from git, but it seems I'm missing something.

Here's what I did:

cs install bleep
git clone --recurse-submodules [email protected]:oyvindberg/bleep.git
bleep generate-resources
# edit bleep_git in bleep_dev.sh to repo dir

Now in my project I can run a main class without problems:

bleep run my_project --class Playground # works

But using the git version of bleep like so

/path/to/bleep-repo//bleep_dev.sh run my_project --class Playground

fails with the following error:

Error: LinkageError occurred while loading main class Playground
        java.lang.ExceptionInInitializerError: null

I also tried building a native bleep binary and ran that, but the error is the same. Any idea what I'm missing?

I think this direction with build variants produced by build rewrites is fairly promising, in that the builds are immutable (deriving a new build doesn't change the original build), and you have an original build which makes sense without the rewrites.

If we end up needing all this power, we could in theory lift the build variants into the build file as a first class thing:

build-variants:
  gpu: scripts/com.foo.ApplyGpuBuildRewrite 
projects: ...

ApplyGpuBuildRewrite would produce a new build variant output as yaml to stdout for instance.
Then you could tell bleep something like bleep --variant gpu compile, and it would pick that build variant.
There would just be some bookkeeping with running the script, caching the output, invalidating on changes to scripts sources and so on.

Thoughts? :)

Originally posted by @oyvindberg in #250 (comment)

• publish
• native image
• anything else?

Bleep should be able to build container images.
JIB is a great tool to do that.

Possible source of inspiration: https://github.com/schmitch/sbt-jib/

Open to PRs?

This will allow us to do source generation and other shenanigans.

It should be relatively easy to implement actually, some tree traversal in the bloop compile command to find the commands, and then tell bloop to compile the necessary steps in turn

seems we're downloading the index file quite often, which takes some hundreds of milliseconds at exactly the wrong time

scala-cli for instance has most of these integrations already

Using bleep 0.0.1-M21, bleep is unable to figure out the version number if the property is formatted with spaces: :

version = "3.6.1" 

additionally, it doesn't remove quotes before attempting to download the scalafmt binary, which leads to a 404. The version currently has to be written without spaces nor quotes.

For any command, Bleep prints the timing of each output, which I find very helpful. However, it can make log output hard to read sometimes. For example, this is the summary of a bleep test run:

📗 BSP: Execution took 0ms [t => 28910 ms]
📗 BSP: 1 tests, 1 passed [t => 28910 ms]
📗 BSP: All tests in foo passed [t => 28910 ms]
📗 BSP:  [t => 28910 ms]
📗 BSP: 5 tests passed. 0 tests failed. 0 tests ignored. [t => 28911 ms]
📗 BSP:  [t => 28911 ms]
📗 BSP: Executed in 26 s 410 ms [t => 28911 ms]
📗 BSP:  [t => 28911 ms]
📗 BSP: =============================================== [t => 29314 ms]
📗 BSP: Total duration: 0ms [t => 29314 ms]
📗 BSP: All 3 test suites passed. [t => 29314 ms]
📗 BSP: =============================================== [t => 29314 ms]
📗 Tests succeeded [t => 29341 ms]

which looks really crowded. The empty lines thrown in by the test framework are not that helpful any more.

One solution I could imagine would be to only print the time if it changed relatively to the previous output, which for this particular case would look like this:

📗 BSP: Execution took 0ms [t => 28910 ms]
📗 BSP: 1 tests, 1 passed
📗 BSP: All tests in foo passed
📗 BSP: 
📗 BSP: 5 tests passed. 0 tests failed. 0 tests ignored. [t => 28911 ms]
📗 BSP:
📗 BSP: Executed in 26 s 410 ms 
📗 BSP:
📗 BSP: =============================================== [t => 29314 ms]
📗 BSP: Total duration: 0ms
📗 BSP: All 3 test suites passed.
📗 BSP: ===============================================
📗 Tests succeeded [t => 29341 ms]

Another solution could be to move the time to the beginning of the line:

📗 BSP [28910 ms]: Execution took 0ms 
📗 BSP [28910 ms]: 1 tests, 1 passed 
📗 BSP [28910 ms]: All tests in foo passed 
📗 BSP [28910 ms]:  
📗 BSP [28911 ms]: 5 tests passed. 0 tests failed. 0 tests ignored. 
📗 BSP [28911 ms]:  
📗 BSP [28911 ms]: Executed in 26 s 410 ms 
📗 BSP [28911 ms]:  
📗 BSP [29314 ms]: =============================================== 
📗 BSP [29314 ms]: Total duration: 0ms 
📗 BSP [29314 ms]: All 3 test suites passed. 
📗 BSP [29314 ms]: =============================================== 
📗 [29341 ms] Tests succeeded

which I think would be less confusing to the user.

Or if we combine both:

📗 BSP [28910 ms]: Execution took 0ms 
📗 BSP           : 1 tests, 1 passed 
📗 BSP           : All tests in foo passed 
📗 BSP           :  
📗 BSP [28911 ms]: 5 tests passed. 0 tests failed. 0 tests ignored. 
📗 BSP           :  
📗 BSP           : Executed in 26 s 410 ms 
📗 BSP           :  
📗 BSP [29314 ms]: =============================================== 
📗 BSP           : Total duration: 0ms 
📗 BSP           : All 3 test suites passed. 
📗 BSP           : =============================================== 
📗 [29341 ms] Tests succeeded

WDYT?

In the json format this is easy.

-scala: "2.12",
+scala: ["2.12", "2.13"]

It's getting more complicated later.

generate bloop files for the whole matrix.

it's fine in a way, find a name convention, handle name collisions. The problem is really what bleep compile should do? bleep compile --scala 2.12? and manually/implicitly default to one version?

import from sbt

will need to run sbt bloopInstall for each scala version on import. or fork the bloop sbt plugin and enable it to output scala version dependent bloop directories.

Coursier can manage JVMs. As such a bleep file should be able to just specify one, and it should be downloaded and used automatically.

There is already functionality for this in the ported native image plugin.

• include projects via a git submodules
• script sbt bloopInstall for all
• import all
• run all tests

idea: use scala community build?

I think the most urgent thing to agree on is what test projects will look like. recognizability is important, simpleness (specifically no scopes) is important, small amount of boilerplate in bleep.json is important. These are mostly in conflict.

I have more thoughts here as well, mainly related to #2 #13 #14

In order to use authentication for URLs configured in resolvers, one needs to create a config file. By looking through the code¹ I found that https://github.com/dirs-dev/directories-jvm is used and that on my macOS machine I had to create a ~/Library/Application Support/build.bleep.bleep/config.yaml with the following contents:

authentications:
  <some-resolver-url>:
    user: <the-user>
    password: <the-password>

compileServerMode: 
  mode: shared

It would be great if this was documented on the website to lower the barrier for people trying out Bleep.

Alright, this is an interesting one : I'm attempting to run bleep at $work in a self-hosted GHA. I don't have control over the runners as the infra is provided by a completely isolated team.

• If I install bleep using the coursier channel, as you would expect, I essentially bump into this : the LIBC available on the runner is not matching the one that bleep was built against.
• So I'm like "well, this is CI anyway, let's try and run bleep on JVM, shall we, it's not the end of the world right now?" Unfortunately, when checking for the version number in the build-file (in order to download the correct version), bleep presumes it's running from a native-image, and invokes a coursier function that is unavailable on JVM (only implemented for graal).

The workaround I found is to use coursier to bootstrap the version of bleep referenced in the build file, something like :

BLEEP_VERSION=$(cat bleep.yaml | grep '$version' | cut -d ":" -f2 |   tr -d ' ')
cs bootstrap build.bleep:bleep-cli_2.13:$BLEEP_VERSION -M bleep.Main -o bleep

• - updated graalvm
• - ported to arm64
• - automate release

jvm/name contains both name and version
scripts contain both module and class name

One of the earliest ideas which surfaced around the concept of a light-weight json-like build was that builds are now composable, and you can combine them however you want

share templates within your organization

imports:
  myorg: https://myorg/shared-templates/stable-sha
projects:
  a:
    extends: myorg.mytemplate
  scripts:
    dependsOn: myorg.scripts

this way you can standardize on dependency versions and all the shenanigans bigger orgs tend to do. You could also share scripts this way.

This would be meant to check in permanently.

import external builds into your build

use case: mount multiple builds into your IDE so you can update them together seamlessly. This is incredibly useful for polyrepo programming.

Maybe something like this:

imports:
  circe: https://github.com/circe/circe.git
projects:
  a:
    dependsOn: circe.circe-core

This would depend on either circe having a bleep.yaml already, or that bleep runs the automatic import.

It's somewhat impractical to apply these changes to the build and ensure that they are not checked in, so I think it can be improved somewhat with

build variants based on build rewrites

With the proposed build variants idea you can have your "normal" build with normal dependencies, and a build rewrite which rewrites dependencies into imports. Then you can use that build variant to compile, mount in IDE (bleep setup-ide --build-variant inlined-deps jvm3) and so on.

Given this build file

build-variants:
  inlined-deps: scripts/InlineDeps
projects: 
  a:
    dependencies: io.circe::circe-core:1.4.0
    scala:
      version: 3.2.1
  scripts:
    dependencies: build.bleep::bleep-core:${BLEEP_VERSION}
    scala:
      version: 3.2.1

And this build rewrite (with hypothetical syntax):

package scripts
object InlineDeps extends bleep.InlineDepsRewrite {
  Map("io.circe" -> "https://github.com/circe/circe.git")
}

That build rewrite would then generate a build like this:

build-variants:
  inlined-deps: scripts/InlineDeps
imports:
  circe: https://github.com/circe/circe.git
projects: 
  a:
    dependsOn: circe.circe-core
    scala:
      version: 3.2.1
  scripts:
    dependencies: build.bleep::bleep-core:${BLEEP_VERSION}
    scala:
      version: 3.2.1

And suddenly you can have an amazing experience working on hundreds of git repos simultaneously by commenting in/out build inlines as needed. These build variants would also be convenient to check in.

These are surprisingly common use cases:

• compile all projects that have Scala version 2.13.*
• test all projects that have Scala version 2.12.* and platform JVM
• test all projects that have Scala platform Native
• publish locally all projects that have Scala version 3.*
• etc...

Even though it is unusual to want this, other tools can't do this well, not even sbt.

What is Bleep's story for this?

a bloop subcommand section here might be useful.
Detection of an already running bloop server should be done as well, we now always start a new one.

Originally posted by @hamnis in #76 (comment)

The release v0.14.3-RC2 or circe-yaml now contains a module circe-yaml-v12.
We can replace the replace the vendored circe-yaml code with this module.

look into what other build tools do with tags, passing test names, etc

Using :

• Bleep version 0.0.1-M22
• Macos Monterey 12.3.1
• openjdk version "17.0.3" 2022-04-19

Repro setup

in a temp folder (I need to tweak the template after generation)

> bleep build new -p jvm -p js -p native foobar
> rm -r tests 
> mv foobar-test tests

Then I remove the managed jvm because I get an error when unzipping the graal artifact referenced in the default template.

JS

> bleep test tests@js3
📗 bootstrapped in 73 ms [t => 119 ms]
📙 You build uses the default system JVM, which can change outside the build. for stable builds over time, let bleep manage your chosen JVM by adding it to bleep.yaml [t => 164 ms]
📗 BSP: Generated JavaScript file '/private/tmp/foobar/.bleep/.bloop/tests/js3/[email protected]' [t => 3306 ms]

Bleep just stays stuck there.

Native

> bleep test tests@native3
📗 bootstrapped in 73 ms [t => 120 ms]
📙 You build uses the default system JVM, which can change outside the build. for stable builds over time, let bleep manage your chosen JVM by adding it to bleep.yaml [t => 166 ms]
📗 BSP: Detecting test frameworks in Scala Native projects is not yet supported [t => 1092 ms]
📗 BSP: Missing configured test frameworks in tests@native3 [t => 1092 ms]
📗 BSP: =============================================== [t => 1092 ms]
📗 BSP: Total duration: 0ms [t => 1092 ms]
📗 BSP: No test suites were run. [t => 1092 ms]
📗 BSP: =============================================== [t => 1092 ms]
📕 tests tests@native3 failed status code ERROR [t => 1111 ms]

Trying to set a testFramework explicitly in the build (under template-native) doesn't change the error.