oyvindberg / bleep Goto Github PK
View Code? Open in Web Editor NEWA bleeping fast scala build tool!
License: MIT License
A bleeping fast scala build tool!
License: MIT License
For a platform to be "supported" by bleep, it needs graalvm native-image support, and github action runners.
If it doesn't have that, I personally will not do anything to make it work. However, you can make it possible to use it if you volunteer to add some fallback support where bleep runs on the JVM.
At least two things need to be done:
bleep setup-ide
needs to construct a java command with the relevant classpath instead of calling a native binaryMain
needs to download relevant jar files and classload themwe have all the tools necessary to create a fantastic TUI now - time to experiment!
this worked in the template I used at https://github.com/note/teleport-scala
rewrites are already in place, but some surface is missing before scripts can do arbitrary rewrites before bootstrapping
note: rewrites are named, so any combination of rewrites will yield a new build folder.
for instance if you bootstrap a build with add-jmh
and upgrade-deps
the bloop folder would end up in something like ./.bleep/builds/add-jmv__upgrade-deps/.bloop
I have a project where I depend on native artifacts which have a different classifier per platform. So in a bleep.yaml, I have the following dependency:
- module: org.bytedeco:pytorch:1.13.0-1.5.9-SNAPSHOT
publication:
classifier: linux-x86_64 # could also be macosx-x86_64 or another platform or even a variant like linux-x86_64-gpu
ext: jar
name: pytorch
type: jar
I can set the classifier manually and it totally works fine, but now I'd like to automatically detect the current platform and set the classifier accordingly (JavaCPP already has a function that can be used for that). To make things more complicated, there's also a gpu suffix in the mix which I'd like to be able to append to the classifier as an additional manual configuration.
For sbt there's a plugin called sbt-javacpp which does exactly that (among a few other things).
I'm fairly new to bleep, so I'd like to ask whether this is doable and what would be the best way to do it. I.e. using build rewrites?
there are already bash auto completion in place - the same structure can hopefully be used for other shells.
should include a command similar to bleep install-tab-completions-bash
the most important pages are not written yet:
more details / clarify:
I'm much more inclined to provide a mechanism by which you can use anything which can output json/yaml.
Let's say we perform these steps on startup:
wouldn't that be neat? That way you can for instance code the build in typescript, and you pay for it by having to start node on every invocation of bleep. Or well, we could certainly cache the output by the timestamp of the executable file actually.
Originally posted by @oyvindberg in #15 (comment)
When configuring authentication as described in #255, I had to include the setting compileServerMode
to make the config valid. If I use only this as a config.yaml
:
authentications:
<some-resolver-url>:
user: <the-user>
password: <the-password>
I get:
๐ Couldn't parse json file /Users/kristof/Library/Application Support/build.bleep.bleep/config.yaml: Missing required field: DownField(compileServerMode)
I think it would be great if this config could somehow be merged with the defaults, so that it only needs to contain things that are changed by the user.
Tried on an Ubuntu 22.04 machine.
โฏ bleep import
๐ Will call sbt multiple times to retrieve information about the existing build. Grab some popcorn as it may take a while! [t => 0 ms]
๐ Calling sbt to discover projects... [t => 4 ms]
๐ [subprocess: sbt discover projects]: downloading sbt launcher 1.6.2 [t => 92 ms]
๐ [subprocess: sbt discover projects]: mkdir: cannot create directory '/.cache': Permission denied [t => 93 ms]
๐ [subprocess: sbt discover projects]: mkdir: cannot create directory '/.cache': Permission denied [t => 95 ms]
๐ [subprocess: sbt discover projects]: cat: /.cache/sbt/boot/sbt-launch/1.6.2/sbt-launch-1.6.2.jar.sha1: No such file or directory [t => 96 ms]
๐ [subprocess: sbt discover projects]: shasum: standard input: no properly formatted SHA checksum lines found [t => 109 ms]
๐ [subprocess: sbt discover projects]: failed to download launcher jar: https://repo1.maven.org/maven2/org/scala-sbt/sbt-launch/1.6.2/sbt-launch-1.6.2.jar (shasum mismatch) [t => 109 ms]
๐ Failed external command 'sbt discover projects' with exit code 2. See log file for exact command [t => 110 ms]
There seem to be some path issues. It does not recognize the sbt version and makes and absolute path out of the cache dir. I tried with multiple projects with the same result. I also tried to import one of the projects on a Mac machine and the import did work fine here.
cs install --channel https://raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json bleep
https://raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json
No modified time in response
Exception in thread "main" java.lang.Exception: Error decoding /home/ondra/.cache/coursier/v1/https/raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json (https://raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json): Expected object: CursorHistory(List(El(CursorOpDownField(repositories),true)))
at coursier.install.Channels.$anonfun$find$25(Channels.scala:185)
cs install --channel https://raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json /
https://raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json
No modified time in response
https://raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json
100.0% [##########] 1.3 KiB (37.7 KiB / s)
Exception in thread "main" java.lang.Exception: Error decoding /home/ondra/.cache/coursier/v1/https/raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json (https://raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json): Expected object: CursorHistory(List(El(CursorOpDownField(repositories),true)))
at coursier.install.Channels.$anonfun$find$25(Channels.scala:185)
cs install --channel https://raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json .
https://raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json
No modified time in response
Exception in thread "main" java.lang.Exception: Error decoding /home/ondra/.cache/coursier/v1/https/raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json (https://raw.githubusercontent.com/oyvindberg/bleep/master/channel/bleep.json): Expected object: CursorHistory(List(El(CursorOpDownField(repositories),true)))
at coursier.install.Channels.$anonfun$find$25(Channels.scala:185)
cs install --channel https://raw.githubusercontent.com/oyvindberg/bleep/master/channel bleep
https://raw.githubusercontent.com/oyvindberg/bleep/master/channel
No modified time in response
Exception in thread "main" java.io.FileNotFoundException: /home/ondra/.cache/coursier/v1/https/raw.githubusercontent.com/oyvindberg/bleep/master/channel (Is a directory)
at java.base/java.io.FileInputStream.open0(Native Method)
I keep experimenting with bleep for my project, and running into smaller issues that could be totally my fault using it the wrong way but I think might still make sense to report.
I did an import of my sbt build which sets some java properties when running the main class in a forked JVM, like so:
fork := true,
javaOptions ++= Seq(
"-Dfoo=bar",
),
On import, bleep will generate s.th. like this:
platform:
jvmOptions: -Dfoo=bar
name: jvm
but the java properties are not set on bleep run
. So I tried setting jvmRuntimeOptions: "-Dfoo=bar"
instead without really knowing the difference, but it still does not pick up the properties. Interestingly though, if jvmRuntimeOptions
are set, and I run bloop run
manually in.bleep/builds/normal
, it does pick up the properties.
Here may be the wrong place to ask, because I know Bleep uses Bloop heavily for many things, but I'm still curious.
What is the caching story for Bleep? How does it prevent unnecessary computation taking place over and over? For example does it cache the results of tests?
Also, where is such cache located? Can it be saved/stored somewhere and then later and/or elsewhere unpacked/re-hydrated and reused?
I'm trying to check if I can set Java properties on my project with your fixes from #253 by running bleep from git, but it seems I'm missing something.
Here's what I did:
cs install bleep
git clone --recurse-submodules [email protected]:oyvindberg/bleep.git
bleep generate-resources
# edit bleep_git in bleep_dev.sh to repo dir
Now in my project I can run a main class without problems:
bleep run my_project --class Playground # works
But using the git version of bleep like so
/path/to/bleep-repo//bleep_dev.sh run my_project --class Playground
fails with the following error:
Error: LinkageError occurred while loading main class Playground
java.lang.ExceptionInInitializerError: null
I also tried building a native bleep binary and ran that, but the error is the same. Any idea what I'm missing?
I think this direction with build variants produced by build rewrites is fairly promising, in that the builds are immutable (deriving a new build doesn't change the original build), and you have an original build which makes sense without the rewrites.
If we end up needing all this power, we could in theory lift the build variants into the build file as a first class thing:
build-variants:
gpu: scripts/com.foo.ApplyGpuBuildRewrite
projects: ...
ApplyGpuBuildRewrite
would produce a new build variant output as yaml to stdout for instance.
Then you could tell bleep something like bleep --variant gpu compile
, and it would pick that build variant.
There would just be some bookkeeping with running the script, caching the output, invalidating on changes to scripts
sources and so on.
Thoughts? :)
Originally posted by @oyvindberg in #250 (comment)
> bleep setup-ide
๐ bootstrapped in 4 ms [t => 5 ms]
๐ command failed [t => 5 ms]java.nio.file.NoSuchFileException: /home/ondra/Projects/bleep/.bsp
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:429)
Bleep should be able to build container images.
JIB is a great tool to do that.
Possible source of inspiration: https://github.com/schmitch/sbt-jib/
Open to PRs?
This will allow us to do source generation and other shenanigans.
It should be relatively easy to implement actually, some tree traversal in the bloop compile
command to find the commands, and then tell bloop to compile the necessary steps in turn
seems we're downloading the index file quite often, which takes some hundreds of milliseconds at exactly the wrong time
scala-cli for instance has most of these integrations already
Using bleep 0.0.1-M21, bleep is unable to figure out the version number if the property is formatted with spaces: :
version = "3.6.1"
additionally, it doesn't remove quotes before attempting to download the scalafmt binary, which leads to a 404. The version currently has to be written without spaces nor quotes.
For any command, Bleep prints the timing of each output, which I find very helpful. However, it can make log output hard to read sometimes. For example, this is the summary of a bleep test
run:
๐ BSP: Execution took 0ms [t => 28910 ms]
๐ BSP: 1 tests, 1 passed [t => 28910 ms]
๐ BSP: All tests in foo passed [t => 28910 ms]
๐ BSP: [t => 28910 ms]
๐ BSP: 5 tests passed. 0 tests failed. 0 tests ignored. [t => 28911 ms]
๐ BSP: [t => 28911 ms]
๐ BSP: Executed in 26 s 410 ms [t => 28911 ms]
๐ BSP: [t => 28911 ms]
๐ BSP: =============================================== [t => 29314 ms]
๐ BSP: Total duration: 0ms [t => 29314 ms]
๐ BSP: All 3 test suites passed. [t => 29314 ms]
๐ BSP: =============================================== [t => 29314 ms]
๐ Tests succeeded [t => 29341 ms]
which looks really crowded. The empty lines thrown in by the test framework are not that helpful any more.
One solution I could imagine would be to only print the time if it changed relatively to the previous output, which for this particular case would look like this:
๐ BSP: Execution took 0ms [t => 28910 ms]
๐ BSP: 1 tests, 1 passed
๐ BSP: All tests in foo passed
๐ BSP:
๐ BSP: 5 tests passed. 0 tests failed. 0 tests ignored. [t => 28911 ms]
๐ BSP:
๐ BSP: Executed in 26 s 410 ms
๐ BSP:
๐ BSP: =============================================== [t => 29314 ms]
๐ BSP: Total duration: 0ms
๐ BSP: All 3 test suites passed.
๐ BSP: ===============================================
๐ Tests succeeded [t => 29341 ms]
Another solution could be to move the time to the beginning of the line:
๐ BSP [28910 ms]: Execution took 0ms
๐ BSP [28910 ms]: 1 tests, 1 passed
๐ BSP [28910 ms]: All tests in foo passed
๐ BSP [28910 ms]:
๐ BSP [28911 ms]: 5 tests passed. 0 tests failed. 0 tests ignored.
๐ BSP [28911 ms]:
๐ BSP [28911 ms]: Executed in 26 s 410 ms
๐ BSP [28911 ms]:
๐ BSP [29314 ms]: ===============================================
๐ BSP [29314 ms]: Total duration: 0ms
๐ BSP [29314 ms]: All 3 test suites passed.
๐ BSP [29314 ms]: ===============================================
๐ [29341 ms] Tests succeeded
which I think would be less confusing to the user.
Or if we combine both:
๐ BSP [28910 ms]: Execution took 0ms
๐ BSP : 1 tests, 1 passed
๐ BSP : All tests in foo passed
๐ BSP :
๐ BSP [28911 ms]: 5 tests passed. 0 tests failed. 0 tests ignored.
๐ BSP :
๐ BSP : Executed in 26 s 410 ms
๐ BSP :
๐ BSP [29314 ms]: ===============================================
๐ BSP : Total duration: 0ms
๐ BSP : All 3 test suites passed.
๐ BSP : ===============================================
๐ [29341 ms] Tests succeeded
WDYT?
In the json format this is easy.
-scala: "2.12",
+scala: ["2.12", "2.13"]
It's getting more complicated later.
it's fine in a way, find a name convention, handle name collisions. The problem is really what bleep compile
should do? bleep compile --scala 2.12
? and manually/implicitly default to one version?
will need to run sbt bloopInstall
for each scala version on import. or fork the bloop sbt plugin and enable it to output scala version dependent bloop directories.
Coursier can manage JVMs. As such a bleep file should be able to just specify one, and it should be downloaded and used automatically.
There is already functionality for this in the ported native image plugin.
sbt bloopInstall
for allidea: use scala community build?
I think the most urgent thing to agree on is what test projects will look like. recognizability is important, simpleness (specifically no scopes) is important, small amount of boilerplate in bleep.json is important. These are mostly in conflict.
I have more thoughts here as well, mainly related to #2 #13 #14
In order to use authentication for URLs configured in resolvers
, one needs to create a config file. By looking through the code1 I found that https://github.com/dirs-dev/directories-jvm
is used and that on my macOS machine I had to create a ~/Library/Application Support/build.bleep.bleep/config.yaml
with the following contents:
authentications:
<some-resolver-url>:
user: <the-user>
password: <the-password>
compileServerMode:
mode: shared
It would be great if this was documented on the website to lower the barrier for people trying out Bleep.
Btw. I found it relatively easy to do, thanks for the clean code base! โฉ
Alright, this is an interesting one : I'm attempting to run bleep at $work in a self-hosted GHA. I don't have control over the runners as the infra is provided by a completely isolated team.
The workaround I found is to use coursier to bootstrap the version of bleep referenced in the build file, something like :
BLEEP_VERSION=$(cat bleep.yaml | grep '$version' | cut -d ":" -f2 | tr -d ' ')
cs bootstrap build.bleep:bleep-cli_2.13:$BLEEP_VERSION -M bleep.Main -o bleep
jvm/name
contains both name and version
scripts
contain both module and class name
One of the earliest ideas which surfaced around the concept of a light-weight json-like build was that builds are now composable, and you can combine them however you want
imports:
myorg: https://myorg/shared-templates/stable-sha
projects:
a:
extends: myorg.mytemplate
scripts:
dependsOn: myorg.scripts
this way you can standardize on dependency versions and all the shenanigans bigger orgs tend to do. You could also share scripts this way.
This would be meant to check in permanently.
use case: mount multiple builds into your IDE so you can update them together seamlessly. This is incredibly useful for polyrepo programming.
Maybe something like this:
imports:
circe: https://github.com/circe/circe.git
projects:
a:
dependsOn: circe.circe-core
This would depend on either circe having a bleep.yaml
already, or that bleep runs the automatic import.
It's somewhat impractical to apply these changes to the build and ensure that they are not checked in, so I think it can be improved somewhat with
With the proposed build variants idea you can have your "normal" build with normal dependencies, and a build rewrite which rewrites dependencies into imports. Then you can use that build variant to compile, mount in IDE (bleep setup-ide --build-variant inlined-deps jvm3
) and so on.
Given this build file
build-variants:
inlined-deps: scripts/InlineDeps
projects:
a:
dependencies: io.circe::circe-core:1.4.0
scala:
version: 3.2.1
scripts:
dependencies: build.bleep::bleep-core:${BLEEP_VERSION}
scala:
version: 3.2.1
And this build rewrite (with hypothetical syntax):
package scripts
object InlineDeps extends bleep.InlineDepsRewrite {
Map("io.circe" -> "https://github.com/circe/circe.git")
}
That build rewrite would then generate a build like this:
build-variants:
inlined-deps: scripts/InlineDeps
imports:
circe: https://github.com/circe/circe.git
projects:
a:
dependsOn: circe.circe-core
scala:
version: 3.2.1
scripts:
dependencies: build.bleep::bleep-core:${BLEEP_VERSION}
scala:
version: 3.2.1
And suddenly you can have an amazing experience working on hundreds of git repos simultaneously by commenting in/out build inlines as needed. These build variants would also be convenient to check in.
These are surprisingly common use cases:
Even though it is unusual to want this, other tools can't do this well, not even sbt.
What is Bleep's story for this?
a bloop subcommand section here might be useful.
Detection of an already running bloop server should be done as well, we now always start a new one.
Originally posted by @hamnis in #76 (comment)
The release v0.14.3-RC2 or circe-yaml now contains a module circe-yaml-v12
.
We can replace the replace the vendored circe-yaml code with this module.
look into what other build tools do with tags, passing test names, etc
Using :
in a temp folder (I need to tweak the template after generation)
> bleep build new -p jvm -p js -p native foobar
> rm -r tests
> mv foobar-test tests
Then I remove the managed jvm
because I get an error when unzipping the graal artifact referenced in the default template.
> bleep test tests@js3
๐ bootstrapped in 73 ms [t => 119 ms]
๐ You build uses the default system JVM, which can change outside the build. for stable builds over time, let bleep manage your chosen JVM by adding it to bleep.yaml [t => 164 ms]
๐ BSP: Generated JavaScript file '/private/tmp/foobar/.bleep/.bloop/tests/js3/[email protected]' [t => 3306 ms]
Bleep just stays stuck there.
> bleep test tests@native3
๐ bootstrapped in 73 ms [t => 120 ms]
๐ You build uses the default system JVM, which can change outside the build. for stable builds over time, let bleep manage your chosen JVM by adding it to bleep.yaml [t => 166 ms]
๐ BSP: Detecting test frameworks in Scala Native projects is not yet supported [t => 1092 ms]
๐ BSP: Missing configured test frameworks in tests@native3 [t => 1092 ms]
๐ BSP: =============================================== [t => 1092 ms]
๐ BSP: Total duration: 0ms [t => 1092 ms]
๐ BSP: No test suites were run. [t => 1092 ms]
๐ BSP: =============================================== [t => 1092 ms]
๐ tests tests@native3 failed status code ERROR [t => 1111 ms]
Trying to set a testFramework explicitly in the build (under template-native
) doesn't change the error.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.