Consider using GJF 1.7 about ktfmt

[tyvsmith]

It would break hermeticism for us, because we don't have a widely adopted mechanism for managing multiple JRE environments in our monorepo, so it would break for us as it would depend on externalities on the dev machine if I made this work today without undertaking the dev provisioning work first.

Regarding GraalVM's native-image, it was an interesting exploration. I chatted with some maintainers as well, and it doesn't seem like anyone has gotten kotlin-compiler to work with native-image. By using the embeddable artifact instead, the run-time init wasn't too verbose.

Graal Update

This is based on a lot of trial and error, I was learning the tool at the same time, so who knows how correct it is.

Generate config

java -agentlib:native-image-agent=config-output-dir=config -jar ktfmt-0.16-SNAPSHOT-jar-with-dependencies.jar TestFile.kt

Add additional reflection config options to reflect-config.json

in addition to what was generated

{
  "name":"org.jetbrains.kotlin.com.intellij.openapi.command.CommandListener",
  "allDeclaredFields":true,
  "allPublicMethods":true
},
{
  "name":"org.jetbrains.kotlin.com.intellij.openapi.vfs.VirtualFileListener",
  "allDeclaredFields":true,
  "allPublicMethods":true
},

Provide Substitutions for Unsafe access

import com.oracle.svm.core.annotate.TargetClass;
import com.oracle.svm.core.annotate.RecomputeFieldValue;
import com.oracle.svm.core.annotate.RecomputeFieldValue.Kind;
import com.oracle.svm.core.annotate.Alias;
import com.oracle.svm.core.annotate.Substitute;
import org.jetbrains.kotlin.com.intellij.util.concurrency.AtomicFieldUpdater;

@TargetClass(AtomicFieldUpdater.class)
final class Target_org_jetbrains_kotlin_com_intellij_util_concurrency_AtomicFieldUpdater {   
    @Alias @RecomputeFieldValue(kind = Kind.AtomicFieldUpdaterOffset)
    private long offset;
}
public class Substitutions { }

Build native image

Download JNA and put it on the CP (needed because 🤷 )

native-image \
    -cp substitutions:libs/jna-5.5.0.jar \ 
    -H:+ReportExceptionStackTraces \
    -H:+ReportUnsupportedElementsAtRuntime \
    -H:+TraceClassInitialization \
    -H:IncludeResourceBundles=com.sun.tools.javac.resources.compiler \
    -H:ConfigurationFileDirectories=config \
    -H:Class=com.facebook.ktfmt.MainKt  \
    -H:Name="ktfmt" \
    --initialize-at-build-time \
    --initialize-at-run-time=org.jetbrains,com.facebook \
    --no-fallback \
    --no-server \
    --report-unsupported-elements-at-runtime \
    --verbose \
    --allow-incomplete-classpath \
    -jar ktfmt-0.16-SNAPSHOT-jar-with-dependencies.jar

Run

and provide first System prop that causes a crash

./ktfmt-0.16-SNAPSHOT-jar-with-dependencies -Djava.runtime.version="11.0.7+10-jvmci-19.3-b10"

Now it'll start crashing with Exception in thread "main" java.lang.IllegalStateException: Resource not found: /org/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment$Companion.class even though it's defined in resource-config.json.

This is where I stopped and was told by a maintainer they heard a few teams try and give up, and jetbrains tried and also gave up a couple years ago. I reached out for an update from them, but haven't heard yet.

I expect a cascading series of continuing errors, that would need to be whack-a-moled. Anything resource loading, system props, file access, weird reflection issues.

Next steps

We'll probably maintain a fork until we can upgrade our repo or get a better system for JDK distribution.

On a separate note, this project should probably be using kotlin-compiler-embeddable which is a slightly reduced artifact designed for tools like this. I have a branch I've moved over and can send up a PR if that's the direction y'all want to go.

[cgrushko]

ktfmt actually depends on code added in GJF 1.8 (as a matter of fact, 1.8 was released because we added code to GJF and wanted it to be in maven central :) )

I see 3 options -

1. Run ktfmt under Java 11, regardless of the migration. If 11 is installed, you can simply do
   $(/usr/libexec/java_home -v11)/bin/java -jar ktfmt-<VERSION>-jar-with-dependencies.jar File.kt
2. Build ktfmt and GJF from source. This is actually what we do internally because we set it up before everything was in Maven Central. If you're using Buck, we can probably share the BUCK files if it helps. I'm not sure we have GJF 1.8 internally, though - it could be we never imported anything newer than the changes we ourselves made, so it might be that GJF 1.8 isn't buildable under JDK 8.
3. Create a native executable using GraalVM. An issue on GJF suggested it was straightforward to do, so it might be the case for ktfmt as well.

What do you think?

[tyvsmith]

Thanks, After posting this I pulled down the project and forced it to gjf 1.7 and saw the API incompatible changes that would require option 2 with some modifications.

Internally, I've already started exploring option 1 by automating our provisioning on developer machines to guarantee jdk 11 install, option 1.

I hadn't looked into GraalVm before, I'll take a look.

[cgrushko]

I'm going to close this Issue, since we can't directly use GJF 1.7, but please do update if you run into trouble / make progress.

[tyvsmith]

1. Explored some options with this, but not ideal as it violates constraints around the build system being hermetic.
2. This seems like the next path I'll go down, was trying to avoid to keep off the forked path. Can you share your Buck file for reference?
3. Explored this, trivial for Google Java Format. Not Trivial for ktmt because of the dep on kotlin-compiler -> intellij-community. Will need a number substitutions/FieldRecomputations and significant number of custom feature init rules.

[cgrushko]

Re running ktfmt under Java 11 - how does it violate hermeticism? you could check in a java 11 into the repo and run ktfmt through a wrapper script? I don't know the details, I'm just imagining how it would look like.

Re sharing BUCK files - I took a look, and it'll require a lot of sanitizing, and the result will be underwhelming, meaning the adjustments needed to be made for your repo's deps will probably not be worth it vs writing the rules from scratch.

Re GraalVM - if you have the time, please list some issues you ran into - I think it'll be beneficial for others who might want to look into this.

---

BTW, what happens when you change google-java-format's compiling options to Java 1.7? does it use new language features?

[tyvsmith]

Changing to target 1.8 didn't work, since it needed to use the shaded EP jar from before, which was intertwined a bit. Additionally, looked like new language feature support was a motivating factor of the change originally, so likely would have hit that too. So I've just deployed a fork internally to maintain until we can bump to jdk11 for real, since the impl on this is pretty low priority atm.

[cgrushko]

Thanks for the detailed update!
What do you mean by fork? depending on a "GJF version 1.75"?

[cgrushko]

@tyvsmith re kotlin-compiler-embeddable - what's the difference in practice?
We can switch to it, but it might require some wiring on the internal repo.

[tyvsmith]

Thanks for the detailed update!
What do you mean by fork? depending on a "GJF version 1.75"?

I built the last GJF hash prior to jdk 11 requirement and bundled that into the fatjar as a dep instead of the one from maven central. We then deploy that ktfmt fatjar to our artifactory for consumption in the repo. Unless new GJF api's are introduced that are used in this project, keeping updated should be straightforward.

@tyvsmith re kotlin-compiler-embeddable - what's the difference in practice?
We can switch to it, but it might require some wiring on the internal repo.

The embeddable artifact is shaded and designed to be used as a single jar in independent tools. The regular one is used within Intellij and has a variety of fragmented namespaces. I know the Buck project has gone back and forth on which to use. I believe Gradle uses the embeddable.

The shading can cause oddities, because the compiler plugins then need to be the embeddable versions or not. If you look in the kotlin repo, you'll see distributions for both, such as the jvm-abi-plugin.

https://discuss.kotlinlang.org/t/kotlin-compiler-embeddable-vs-kotlin-compiler/3196

[cyw3]

It would break hermeticism for us, because we don't have a widely adopted mechanism for managing multiple JRE environments in our monorepo, so it would break for us as it would depend on externalities on the dev machine if I made this work today without undertaking the dev provisioning work first.

Regarding GraalVM's native-image, it was an interesting exploration. I chatted with some maintainers as well, and it doesn't seem like anyone has gotten kotlin-compiler to work with native-image. By using the embeddable artifact instead, the run-time init wasn't too verbose.

Graal Update

This is based on a lot of trial and error, I was learning the tool at the same time, so who knows how correct it is.

Generate config

java -agentlib:native-image-agent=config-output-dir=config -jar ktfmt-0.16-SNAPSHOT-jar-with-dependencies.jar TestFile.kt

Add additional reflection config options to reflect-config.json

in addition to what was generated

{
  "name":"org.jetbrains.kotlin.com.intellij.openapi.command.CommandListener",
  "allDeclaredFields":true,
  "allPublicMethods":true
},
{
  "name":"org.jetbrains.kotlin.com.intellij.openapi.vfs.VirtualFileListener",
  "allDeclaredFields":true,
  "allPublicMethods":true
},

Provide Substitutions for Unsafe access

import com.oracle.svm.core.annotate.TargetClass;
import com.oracle.svm.core.annotate.RecomputeFieldValue;
import com.oracle.svm.core.annotate.RecomputeFieldValue.Kind;
import com.oracle.svm.core.annotate.Alias;
import com.oracle.svm.core.annotate.Substitute;
import org.jetbrains.kotlin.com.intellij.util.concurrency.AtomicFieldUpdater;

@TargetClass(AtomicFieldUpdater.class)
final class Target_org_jetbrains_kotlin_com_intellij_util_concurrency_AtomicFieldUpdater {   
    @Alias @RecomputeFieldValue(kind = Kind.AtomicFieldUpdaterOffset)
    private long offset;
}
public class Substitutions { }

Build native image

Download JNA and put it on the CP (needed because 🤷 )

native-image \
    -cp substitutions:libs/jna-5.5.0.jar \ 
    -H:+ReportExceptionStackTraces \
    -H:+ReportUnsupportedElementsAtRuntime \
    -H:+TraceClassInitialization \
    -H:IncludeResourceBundles=com.sun.tools.javac.resources.compiler \
    -H:ConfigurationFileDirectories=config \
    -H:Class=com.facebook.ktfmt.MainKt  \
    -H:Name="ktfmt" \
    --initialize-at-build-time \
    --initialize-at-run-time=org.jetbrains,com.facebook \
    --no-fallback \
    --no-server \
    --report-unsupported-elements-at-runtime \
    --verbose \
    --allow-incomplete-classpath \
    -jar ktfmt-0.16-SNAPSHOT-jar-with-dependencies.jar

Run

and provide first System prop that causes a crash

./ktfmt-0.16-SNAPSHOT-jar-with-dependencies -Djava.runtime.version="11.0.7+10-jvmci-19.3-b10"

Now it'll start crashing with Exception in thread "main" java.lang.IllegalStateException: Resource not found: /org/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment$Companion.class even though it's defined in resource-config.json.

This is where I stopped and was told by a maintainer they heard a few teams try and give up, and jetbrains tried and also gave up a couple years ago. I reached out for an update from them, but haven't heard yet.

I expect a cascading series of continuing errors, that would need to be whack-a-moled. Anything resource loading, system props, file access, weird reflection issues.

Next steps

We'll probably maintain a fork until we can upgrade our repo or get a better system for JDK distribution.

On a separate note, this project should probably be using kotlin-compiler-embeddable which is a slightly reduced artifact designed for tools like this. I have a branch I've moved over and can send up a PR if that's the direction y'all want to go.

I get the same error Exception in thread "main" java.lang.IllegalStateException: Resource not found: /org/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment$Companion.class.

Is there any progress here?