Comments (8)
I'm reasonably sure it can happen (we did get segfaults before we introduced this feature when people were rolling their own). I think it can come from the following combination of factors:
- we store multiple cache files for each package
- you can load packages from a different environment (both "moving up" the chain of environments, or by switching environments)
- with an environment variable, there's no record of whether the cache file was created with or without precompilation, and so the precompilation state cannot be used to check for consistency
- once we load a package, all future packages that depend on that package will use the in-memory version.
I think this is all you need to have bad things happen in practice: say A gets used by B & C; B gets precompiled without A being precompiled, but C gets precompiled with A being precompiled. If you load B before C, boom.
from snoopcompile.jl.
Hmm, you're right that as more and more packages use it, the "escape" I thought we had in https://discourse.julialang.org/t/psa-for-snoopprecompile-turning-off-extra-workload-for-specific-packages/92865/5?u=tim.holy doesn't really scale.
We should change this while also moving SnoopPrecompile to JuliaLang and possibly renaming it.
from snoopcompile.jl.
With a package preference, you would still need to re-precompile a package, even though the intention was to specify that you don't want it to run the precompilation workload in future invalidations. To avoid this, we could use an environment variable instead, e.g. JULIA_SKIP_SNOOPED_PRECOMPILE=PackageA,PackageB
with a coma-separated list of packages. One downside however is that re-enabling the run of precompilation workloads for a package will not cause the package to re-precompile. Would we have a way to manually invalidate a package and retrigger precompilation of affected packages in a project?
from snoopcompile.jl.
The whole point of using Preferences is to avoid the myriad problems that come from using an environment variable. In the worst case you can get segfaults if the compilation status of the system is inconsistent.
from snoopcompile.jl.
I see the value of Preferences in the general case, but here might we not assume that a package's logic should be unaffected by whether the precompilation workload is run or not? From my perspective, the related code would only control the amount of things that end up in the package image. That is based on the assumption that no methods, types or globals are defined in the @precompile_all_calls
block. Under this assumption, are the potential issues you mention still likely to happen?
Whether or not we'd like to make such an assumption would be another question.
In any case, having a preference per-package would be a clear improvement over the status quo.
from snoopcompile.jl.
Starting with 1.9, precompilation & package loading basically work like a (very flexible) C linker. That's how we handle the native code. If you expect to be able to link against things that are already precompiled (because they were available when you compiled the package), and then those objects disappear behind your back, bad things happen. Consistency across the entire set of dependencies is paramount, and there's no way to guarantee consistency if environment variables control what does and does not get precompiled.
from snoopcompile.jl.
I see, though I wonder if that situation would happen in practice. The only way I see is if the build of a whole project somehow relies on two independent builds of that same package, which produce different images; but otherwise, wouldn't the state of the compiled code for a given package be read directly from its package image after precompilation? Given that the effect of such an environment variable would always be tied to an invalidation of the whole chain of dependency downstream, I do not see how objects would be able to vanish without triggering recompilation of dependent code.
from snoopcompile.jl.
Fixed by PrecompileTools
from snoopcompile.jl.
Related Issues (20)
- SnoopPrecompile with pkgimages chokes on non-native code HOT 8
- `suggest` is producing `StackOverflowError`
- Add a `.JuliaFormatter.toml` and format?
- There are not tags for SnoopPrecompile in the repo HOT 10
- Clarify docs re: precompile and runtime dispatch HOT 2
- MethodError on `invalidation_trees` HOT 2
- confusing output of mt_backedges HOT 4
- A lazier approach to precompilation HOT 3
- can't SnoopPrecompile ModelingToolkit / DifferentialEquations :( HOT 5
- Loading ReverseDiff and UnicodePlots together and looking at their invalidations prints tens of thousands of lines HOT 3
- Failure to insert invalidations by invalidation_trees()
- Register new versions of SnoopCompileCore? HOT 1
- `MethodError` with `invalidation_trees` on Julia 1.8.2, works on 1.9.1
- `Illegal inttoptr` error on HPC HOT 3
- Error in showing output of `invalidation_trees`
- Why is Pkg.jl a dependency?
- Sorting on number of children seems to have regressed HOT 2
- Next breaking release HOT 10
- Support for cross-environment (de-)serialization of records and offline analysis HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from snoopcompile.jl.