The tokens in tokens.csv are all lowercase about astminer

[egor-bogomolov]

Which task do you run?
I guess we can add an option to split them straight-away as it is done in parse task.

[v587su]

I use the pathContexts task.

[egor-bogomolov]

Okay, I will add the functionality for this!

[v587su]

Thanks a lot.

[v587su]

I removed the ' toLowerCase() 'in astminer/src/main/kotlin/astminer/common/TreeUtil.kt line62. But after building , the tokens are still in lowercase.
@egor-bogomolov Could you tell me where should I modify?

[egor-bogomolov]

@v587su I assume that you are modifying the library and building the CLI then. In order to see changes, you should point CLI to the local build of the library. To do so, modify astminer-cli/build.gradle.kts. If you need further help with it, don't hesitate to ask.

[v587su]

@egor-bogomolov Sry, I'm not familiar with gradle. I tried many times. And change the dependency of astminer-cli to implementation(files("astminer-0.5.4.jar")).
But it seems that I build the library in a wrong way. Is this command "./gradlew build" right? Should I point the CLI to the jar file or the build dir?
The error log of the final cli-0.3.jar:

Exception in thread "main" java.lang.NoClassDefFoundError: com/github/gumtreediff/client/Run
        at astminer.parse.java.GumTreeJavaParser.<init>(GumTreeJavaParser.kt:12)
        at cli.PathContextsExtractor.<init>(PathContextsExtractor.kt:35)
        at cli.MainKt.main(Main.kt:13)
Caused by: java.lang.ClassNotFoundException: com.github.gumtreediff.client.Run
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 3 more

Thanks again.

[egor-bogomolov]

Hi, @v587su!
The correct way to add dependency is as follows: compile(files("astminer-0.5.4.jar"))
Alternatively, to avoid copying jars from library to CLI, you can set the dependency as compile(files("../build/astminer/libs/astminer-0.5.4.jar")). Then, you should go to the root astminer directory, run ./gradlew build and changes will be immediately in CLI.

Sorry for not elaborating on it previously :(

[v587su]

@egor-bogomolov Thanks for your warm heart. It seems that CLI and the local library are incompatible. When I run ./gradlew shadowJar, an error comes up:

> Task :compileKotlin FAILED
e: /astminer/astminer-cli/src/main/kotlin/cli/Code2VecExtractor.kt: (113, 74): Too many arguments for public constructor Code2VecPathStorage(outputFolderPath: String) defined in astminer.paths.Code2VecPathStorage
e: /astminer/astminer-cli/src/main/kotlin/cli/Code2VecExtractor.kt: (113, 85): Too many arguments for public constructor Code2VecPathStorage(outputFolderPath: String) defined in astminer.paths.Code2VecPathStorage
e: /astminer/astminer-cli/src/main/kotlin/cli/PathContextsExtractor.kt: (108, 74): Too many arguments for public constructor Code2VecPathStorage(outputFolderPath: String) defined in astminer.paths.Code2VecPathStorage
e: /astminer/astminer-cli/src/main/kotlin/cli/PathContextsExtractor.kt: (108, 85): Too many arguments for public constructor Code2VecPathStorage(outputFolderPath: String) defined in astminer.paths.Code2VecPathStorage

FAILURE: Build failed with an exception.

[egor-bogomolov]

I recommend you pull the latest changes from our master branch. Recently I've updated the API of storages and it looks like it is updated in your CLI (call with 3 parameters) but not in the library (constructor with a single parameter).

[egor-bogomolov]

As you can see in Code2VecPathStorage.kt, in master branch it is already updated.

[v587su]

It is my fault. The latest version is 0.5.5. However, I use 0.5.4.jar.
After fixing this, I build it successfully. But I get the same error while using cli-0.3.jar.

Exception in thread "main" java.lang.NoClassDefFoundError: com/github/gumtreediff/client/Run
        at astminer.parse.java.GumTreeJavaParser.<init>(GumTreeJavaParser.kt:12)
        at cli.PathContextsExtractor.<init>(PathContextsExtractor.kt:32)
        at cli.MainKt.main(Main.kt:13)
Caused by: java.lang.ClassNotFoundException: com.github.gumtreediff.client.Run
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 3 more

My java version is 11.0.3.

[egor-bogomolov]

Could you please fork the repository and push your changes there?

[v587su]

I make no changes. Just to test the process.

[egor-bogomolov]

Does the error occur when you use remote version of the library? I.e., if you just clone the repository, run ./gradlew shadowJar and try to run java -jar build/shadow/cli-0.3.jar?

[v587su]

It is ok with remote version. I guess it is caused by my java version. What is yours?

[egor-bogomolov]

java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)

[egor-bogomolov]

I was able to reproduce your error, looking for a fix.

[egor-bogomolov]

I've made a commit to master where I added shadowJar task to the library. To make things work with local library follow these steps:

1. Go to astminer/ directory and run ./gradlew shadowJar
2. Go to astminer/astminer-cli and add dependency implementation(files("../build/shadow/lib-0.5.5.jar")) in build.gradle.kts
3. Run ./gradlew shadowJar, this time in CLI, to create a runnable jar.

[v587su]

@egor-bogomolov Thanks! It works!

[v587su]

@egor-bogomolov Hi, the local library still has bugs. The result generated by cli-0.3.jar has no content in path_contexts.csv. And the lines of tokens.csv are fewer than the online version.

[egor-bogomolov]

What do you mean by the local and online libraries? Did you check the corresponding versions? What project do you test it on?

[v587su]

Forgive my poor English.
I build the local library and point it to the CLI. The result of CLI seems wrong. There is no content in path_contexts.csv. Besides, tokens.csv have fewer tokens than the remote version of the library.
The version of local library is 0.5.5.
I test on my own data. My data is ok with the remote version. I believe it can be reproduced on any data.

[SpirinEgor]

Since the issue discussion, we significantly improved the astminer. Check our new documentation to see how it works now. If you still have any questions, feel free to open an issue for that.