System Image about spine-toolbox

[PekkaSavolainen]

This should be fixed on both master and 0.8-dev branches.

[soininen]

Do you have the latest Toolbox? The button has been fixed quite recently.

[danajhen]

Yes, I have the version installed, both SpineOpt and Spinetoolbox. However, if I click the button to create a system image of the project, I get the following error:

[hannesfelipe]

I also have problems creating a system image using spinetoolbox. I am using version 0.7.2 and getting the following error message:

[07-12-2023 17:21:38] ERROR: The following package names could not be resolved:
[07-12-2023 17:21:38] * CompatLinearAlgebraExt (not found in project, manifest or registry)
[07-12-2023 17:21:38] * DistributionsTestExt (not found in project, manifest or registry)
[07-12-2023 17:21:38] Suggestions: PowerModelsDistributionStateEstimation
[07-12-2023 17:21:38] * FillArraysPDMatsExt (not found in project, manifest or registry)
[07-12-2023 17:21:38] * FillArraysSparseArraysExt (not found in project, manifest or registry)
[07-12-2023 17:21:38] * FillArraysStatisticsExt (not found in project, manifest or registry)
[07-12-2023 17:21:38] Stacktrace:
[07-12-2023 17:21:38] [1] pkgerror(msg::String)
[07-12-2023 17:21:38] @ Pkg.Types C:\Users\USERNAME\AppData\Local\Programs\Julia-1.9.4\share\julia\stdlib\v1.9\Pkg\src\Types.jl:69
[07-12-2023 17:21:38] [2] ensure_resolved(ctx::Pkg.Types.Context, manifest::Pkg.Types.Manifest, pkgs::Vector{Pkg.Types.PackageSpec}; registry::Bool)
[07-12-2023 17:21:38] @ Pkg.Types C:\Users\USERNAME\AppData\Local\Programs\Julia-1.9.4\share\julia\stdlib\v1.9\Pkg\src\Types.jl:1003
[07-12-2023 17:21:38] [3] ensure_resolved
[07-12-2023 17:21:38] @ C:\Users\USERNAME\AppData\Local\Programs\Julia-1.9.4\share\julia\stdlib\v1.9\Pkg\src\Types.jl:954 [inlined]
[07-12-2023 17:21:39]
[07-12-2023 17:21:39] Process to create Run SpineOpt_JuliaSysimage.dll failed

[PekkaSavolainen]

@hannesfelipe I could not recreate your error message but got this one:

(Conda) Python 3.9
spinetoolbox: 0.7.2 from PyPi
Julia: 1.8.5
SpineOpt: 0.6.16
SpineInterface: 0.11.13

[08-12-2023 14:14:37] Process to create Run SpineOpt_JuliaSysimage.dll successfully started.
This process might take a while, but you can keep using Spine Toolbox as normal in the meantime.
[08-12-2023 14:18:09] 	Starting program C:/Julia-1.8.5/bin/julia.exe
[08-12-2023 14:18:09] 	Arguments: --project=C:/data/JuliaProjects/SpineOptProject -e using Pkg;
project_dir = dirname(Base.active_project());
cp(joinpath(project_dir, "Project.toml"), joinpath(project_dir, "Project.backup"); force=true);
cp(joinpath(project_dir, "Manifest.toml"), joinpath(project_dir, "Manifest.backup"); force=true);
try
    modules = split("StatsFuns TableTraits SentinelArrays Parsers DiffResults Rmath Compat JuMP SnoopPrecompile DataStructures NaNMath Rmath_jll Calculus OrderedCollections Requires CodecZlib HiGHS_jll HypergeometricFunctions FillArrays DataValueInterfaces CodecZstd StaticArrays EnumX ChangesOfVariables StatsBase MathOptInterface HiGHS OpenLibm_jll SpineInterface PDMats ArrowTypes InverseFunctions CodecLz4 Reexport Zlib_jll JSON Tables Zstd_jll DocStringExtensions ForwardDiff InlineStrings Distributions BenchmarkTools ExprTools IrrationalConstants RecipesBase PooledArrays URIs IteratorInterfaceExtensions Bzip2_jll Mocking Lz4_jll PrecompileTools DataAPI StatsAPI CompilerSupportLibraries_jll SpecialFunctions BitIntegers CommonSubexpressions SpineOpt DiffRules MacroTools SortingAlgorithms Scratch LoggingExtras CodecBzip2 LogExpFunctions ChainRulesCore OpenSpecFun_jll CEnum DualNumbers QuadGK DensityInterface TimeZones ConcurrentUtilities StaticArraysCore JLLWrappers TranscodingStreams Arrow MutableArithmetics TZJData Preferences Missings", " ");
    Pkg.add(modules);
    Pkg.add("PackageCompiler");
    @eval import PackageCompiler
    Base.invokelatest(
        PackageCompiler.create_sysimage,
        Symbol.(modules);
        sysimage_path="C:/data/SpineToolboxData/Projects/SpineOptProject_III/Run SpineOpt_JuliaSysimage.dll",
        precompile_statements_file="C:\\data\\SpineToolboxData\\Projects\\SpineOptProject_III\\.spinetoolbox\\items\\tool_1\\precompile_statements_file.jl"
    )
finally
    cp(joinpath(project_dir, "Project.backup"), joinpath(project_dir, "Project.toml"); force=true);
    cp(joinpath(project_dir, "Manifest.backup"), joinpath(project_dir, "Manifest.toml"); force=true);
end
[08-12-2023 14:18:09] 	Execution in progress...
[08-12-2023 14:18:11] Updating registry at `C:\Users\ttepsa\.julia\registries\SpineRegistry`
[08-12-2023 14:18:13] ERROR: GitError(Code:ERROR, Class:SSL, Your Julia is built with a SSL/TLS engine that libgit2 doesn't know how to configure to use a file or directory of certificate authority roots, but your environment specifies one via the SSL_CERT_FILE variable. If you believe your system's root certificates are safe to use, you can `export JULIA_SSL_CA_ROOTS_PATH=""` in your environment to use those instead.)
[08-12-2023 14:18:13] Stacktrace:
[08-12-2023 14:18:13] [1] set_ssl_cert_locations(cert_loc::String)
[08-12-2023 14:18:13] @ LibGit2 C:\Julia-1.8.5\share\julia\stdlib\v1.8\LibGit2\src\LibGit2.jl:1023
[08-12-2023 14:18:13] [2] initialize()
[08-12-2023 14:18:13] @ LibGit2 C:\Julia-1.8.5\share\julia\stdlib\v1.8\LibGit2\src\LibGit2.jl:989
[08-12-2023 14:18:13] [3] (::LibGit2.var"#162#163")()
[08-12-2023 14:18:13] @ LibGit2 C:\Julia-1.8.5\share\julia\stdlib\v1.8\LibGit2\src\LibGit2.jl:975
[08-12-2023 14:18:13] [4] lock(f::LibGit2.var"#162#163", l::ReentrantLock)
[08-12-2023 14:18:13] @ Base .\lock.jl:185
[08-12-2023 14:18:13] [5] ensure_initialized
[08-12-2023 14:18:13] @ C:\Julia-1.8.5\share\julia\stdlib\v1.8\LibGit2\src\LibGit2.jl:971 [inlined]
[08-12-2023 14:18:13] [6] LibGit2.GitRepo(path::String)
[08-12-2023 14:18:13] @ LibGit2 C:\Julia-1.8.5\share\julia\stdlib\v1.8\LibGit2\src\repository.jl:9
[08-12-2023 14:18:13] [7] update(regs::Vector{Pkg.Registry.RegistrySpec}; io::Base.PipeEndpoint, force::Bool)
[08-12-2023 14:18:13] @ Pkg.Registry C:\Julia-1.8.5\share\julia\stdlib\v1.8\Pkg\src\Registry\Registry.jl:401
[08-12-2023 14:18:13] 
[08-12-2023 14:18:13] Process to create Run SpineOpt_JuliaSysimage.dll failed

So toolbox did create the precompile_statements_file.jl and loaded_modules.txt files for me but creating the actual DLL failed. Your listing above does not show the whole traceback so I cannot tell if the precompile file was actually created or not?

I suggest you try creating the sysimage manually, which worked for me. The instructions are in https://github.com/Spine-tools/SpineOpt.jl#compilation-into-a-julia-system-image . It might be easier to debug what's going on, if an error message pops up there.

[PekkaSavolainen]

Yes, I have the version installed, both SpineOpt and Spinetoolbox. However, if I click the button to create a system image of the project, I get the following error:

@danajhen Please try creating the sysimage manually and see what happens.

[hannesfelipe]

@PekkaSavolainen I tried it manually. My ""file.jl contains the following:

--trace-compile=file.jl
run_spineopt(C:\Users\USER\Spine\Simple_YT_System.spinetoolbox\items\in\In.sqlite)

After running the command "create_sysimage(; sysimage_path="SpineOpt.dll", precompile_statements_file="file.jl")", I have a .ddl file and try to use it in SpineToolbox. However, this seems not working.

Also I don't know how to use the image via step 4: Start Julia with --sysimage=SpineOpt.dll to use the generated image. As I don't know where to write this comment in the Toolbox.

[PekkaSavolainen]

@hannesfelipe Step 4 happens here in Toolbox. Click the right-hand side button and navigate to your dll.

[hannesfelipe]

@PekkaSavolainen that's what I did. But there is no acceleration of the first run after a restart of Spinetoolbox. The first run still takes several minutes for my simple model.

Anything I have to add to the text in my file.jl?

[hannesfelipe]

I also add the whole traceback when using the option to generate the system image directly in Spinetoolbox. I think if this way would work again it would be easier for users like me.

[08-12-2023 14:18:22] DAG 1/1 completed successfully
[08-12-2023 14:18:48] Process to create Run SpineOpt_JuliaSysimage.dll successfully started.
This process might take a while, but you can keep using Spine Toolbox as normal in the meantime.
[08-12-2023 14:22:06] Opening Tool specification editor for Run SpineOpt
[08-12-2023 14:23:34] Starting program C:/Users/USER/AppData/Local/Programs/Julia-1.9.4/bin/julia.exe
[08-12-2023 14:23:34] Arguments: -e using Pkg;
project_dir = dirname(Base.active_project());
cp(joinpath(project_dir, "Project.toml"), joinpath(project_dir, "Project.backup"); force=true);
cp(joinpath(project_dir, "Manifest.toml"), joinpath(project_dir, "Manifest.backup"); force=true);
try
modules = split("StatsFuns TableTraits SentinelArrays Parsers DiffResults Rmath FillArraysStatisticsExt Compat JuMP SnoopPrecompile DataStructures NaNMath Rmath_jll Calculus OrderedCollections Requires CodecZlib HiGHS_jll HypergeometricFunctions FillArrays DataValueInterfaces CodecZstd EnumX StatsBase MathOptInterface HiGHS OpenLibm_jll SpineInterface PDMats ArrowTypes CodecLz4 SuiteSparse Reexport Zlib_jll FillArraysSparseArraysExt JSON Tables Zstd_jll DocStringExtensions ForwardDiff InlineStrings Distributions Statistics BenchmarkTools ExprTools IrrationalConstants PooledArrays URIs IteratorInterfaceExtensions Bzip2_jll Mocking Lz4_jll DistributionsTestExt PrecompileTools DataAPI StatsAPI CompilerSupportLibraries_jll SpecialFunctions BitIntegers CommonSubexpressions SpineOpt DiffRules MacroTools SortingAlgorithms Scratch LoggingExtras CompatLinearAlgebraExt CodecBzip2 LogExpFunctions OpenSpecFun_jll CEnum DualNumbers QuadGK TimeZones ConcurrentUtilities StaticArraysCore JLLWrappers FillArraysPDMatsExt TranscodingStreams Arrow MutableArithmetics TZJData Preferences Missings", " ");
Pkg.add(modules);
Pkg.add("PackageCompiler");
@eval import PackageCompiler
Base.invokelatest(
PackageCompiler.create_sysimage,
Symbol.(modules);
sysimage_path="C:/Users/USER/Spine/Simple_YT_System/Run SpineOpt_JuliaSysimage.dll",
precompile_statements_file="C:\Users\USER\Spine\Simple_YT_System\.spinetoolbox\items\run_spineopt\precompile_statements_file.jl"
)
finally
cp(joinpath(project_dir, "Project.backup"), joinpath(project_dir, "Project.toml"); force=true);
cp(joinpath(project_dir, "Manifest.backup"), joinpath(project_dir, "Manifest.toml"); force=true);
end
[08-12-2023 14:23:34] Execution in progress...
[08-12-2023 14:23:41] ERROR: The following package names could not be resolved:
[08-12-2023 14:23:41] * CompatLinearAlgebraExt (not found in project, manifest or registry)
[08-12-2023 14:23:41] * DistributionsTestExt (not found in project, manifest or registry)
[08-12-2023 14:23:41] Suggestions: PowerModelsDistributionStateEstimation
[08-12-2023 14:23:41] * FillArraysPDMatsExt (not found in project, manifest or registry)
[08-12-2023 14:23:41] * FillArraysSparseArraysExt (not found in project, manifest or registry)
[08-12-2023 14:23:41] * FillArraysStatisticsExt (not found in project, manifest or registry)
[08-12-2023 14:23:41] Stacktrace:
[08-12-2023 14:23:41] [1] pkgerror(msg::String)
[08-12-2023 14:23:41] @ Pkg.Types C:\Users\USER\AppData\Local\Programs\Julia-1.9.4\share\julia\stdlib\v1.9\Pkg\src\Types.jl:69
[08-12-2023 14:23:42]
[08-12-2023 14:23:42] Process to create Run SpineOpt_JuliaSysimage.dll failed

[PekkaSavolainen]

Does this file C:\Users\USER\Spine\Simple_YT_System\.spinetoolbox\items\run_spineopt\precompile_statements_file.jl exist? If it does you can go directly to step 3 in creating the sysimage. file.jl refers to the Tool Spec main program file that does something like:

using SpineOpt

m = run_spineopt(ARGS...)

[hannesfelipe]

@PekkaSavolainen thank you! Using this file for the system image is working. 👍

Might be helpful to add this information to the description for the creation of the system image.

[PekkaSavolainen]

@PekkaSavolainen thank you! Using this file for the system image is working. 👍

Might be helpful to add this information to the description for the creation of the system image.

You're welcome! Yes, might be a good idea to add the lessons learned here into the User Guide but the feature itself has been updated in the 0.8-dev branch. However, I cannot test it there because that branch does not support SpineOpt yet.

[PekkaSavolainen]

@danajhen Did you try creating the sysimage manually?

[jkiviluo]

However, I cannot test it there because that branch does not support SpineOpt yet.

It should work, says Manuel. There are people testing it, so some issues may pop-up.

[manuelma]

Yeah, as @jkiviluo says, SpineOpt and SpineInterface repos have 0.8-dev branches that work well with toolbox-side 0.8-dev branches. If you check out all of them together you should be able to test it.

[PekkaSavolainen]

Thanks, I got SpineOpt working in 0.8-dev Spine Toolbox and I'm seeing the same error as @hannesfelipe above. Now all I need to is to fix the bug...

[danajhen]

@danajhen Did you try creating the sysimage manually?

Yes, thank you! All good now

[PekkaSavolainen]

So, I think I know what the problem here is. Basically, creating the sysimage works like this at the moment. First we run the tool code (e.g. run_spineopt(...)) wrapped up in Base.loaded_modules calls. Then we compare the before and after states of loaded modules to get the modules that were imported or used while running the original code. We write the names of the imported modules into a loaded_modules.txt file which is used in the next phase. In the next phase, we are trying to Pkg.add() the contents of loaded_modules.txt file but this does not work because Pkg.add() expects package names and not modules. For example, CompatLinearAlgebraExt is imported when running run_spineopt(...) and this is written to the loaded_modules.txt file but CompatLinearAlgebraExt is not a package but an extension in the Compat package.

There are two fixes that I can think of

1. We don't create the loaded_modules.txt file at all, and just create the sysimage like this

PackageCompiler.create_sysimage(;, sysimage_path="self.sysimage_path", precompile_statements_file="precompile_statements_file")

See https://julialang.github.io/PackageCompiler.jl/stable/refs.html

This looks like it works very well but I'm not sure what the downsides are. Maybe that there could be a lot of packages in the same environment than whatever package (e.g. SpineOpt) the current tool uses and the sysimage creation process might be slower?

2. Replace the process of creating loaded_modules.txt so that we somehow get a list of the packages that were used when the tool code runs but I'm not sure how to do that and could use some help here...

[PekkaSavolainen]

I think I got this. PR coming up tomorrow.