Comments (5)
"Virtual memory used by Julia"
Where did this data come from?
Does this refer to the "Virtual Size" in Process Explorer?
I have 32GB of physical memory.
Windows seems to allocate 2~3TB of virtual memory space for Edge.
Maybe Edge has a much bigger problem than julia?
Maybe related: Unreasonably huge process virtual memory size reported by Process Explorer
from julia.
In general, virtual memory isn't really meaningful (at least on 64 bit systems). You have roughly infinite address space, available, so as long as you don't bring the memory into resident memory, it's all fairly imaginary.
from julia.
"Virtual memory used by Julia"
Where did this data come from? Does this refer to the "Virtual Size" in Process Explorer?
I have 32GB of physical memory. Windows seems to allocate 2~3TB of virtual memory space for Edge. Maybe Edge has a much bigger problem than julia?
Maybe related: Unreasonably huge process virtual memory size reported by Process Explorer
No, it's actually the Private Bytes
in Process Explorer, which is about 300MB in the screenshot of Edge. In addition, my workstation was initially configured to have total virtual memory of about 32GB, running a few instances of Julia in parallel will result in errors like "Cannot load xxx.dll due to insufficient virtual memory" (I cannot remember the exact message, but it's something like that). After raising the maximum limit of virtual memory to 1TiB, now I can open about 64 instances in parallel without actually doing any calculation.
So I think in my case, Julia seems to use too much virtual memory as soon as it starts up, and uses up the virtual memory at a very fast rate.
In general, virtual memory isn't really meaningful (at least on 64 bit systems). You have roughly infinite address space, available, so as long as you don't bring the memory into resident memory, it's all fairly imaginary.
Within a specific amount (like tens or hundreds of GB), I do agree that virtual memory is not a problem. However, for some unknown reason, on my workstation, the amount of virtual memory is extremely large and prevents other programs (including other instances of Julia) from running.
By the way, is there any way to know how the virtual memory is being used, so that I might be able to compare the usage on both systems, and see what might be wrong?
from julia.
What if you set the environment variable OPENBLAS_NUM_THREADS=1
before starting julia? Does that change anything?
from julia.
What if you set the environment variable
OPENBLAS_NUM_THREADS=1
before starting julia? Does that change anything?
It works! The amount of used virtual memory has decreased to about 180MB.
After trying some different values of OPENBLAS_NUM_THREADS
, it seems that even setting it to 256 (number of logical cores in the Workstation) cannot replicate the usage of 17GB virtual memory, but I think that might be another issue for openBLAS, instead of Julia.
OPENBLAS_NUM_THREADS |
Claimed virtual memory (~MB) |
---|---|
1 | 186 |
2 | 318 |
4 | 581 |
8 | 1106 |
16 | 2157 |
32 | 4259 |
64 | 8462 |
128 | 8462 |
256 | 8462 |
No value | 18670 |
Anyway, the problem has been solved, thanks to all you people for helping me with this!
from julia.
Related Issues (20)
- at-doc return type depends on whether REPL is loaded HOT 3
- Possible improvement in error messages for invalid array sizes HOT 1
- Looking up an instruction pointer (`StackTraces.lookup(::InterpreterIP)`) has dynamic dispatch
- Encounter a bug using DifferentialEquations HOT 5
- Should the test environment precompile (w/ `--check-bounds=yes`) when precompiling a package environment? HOT 4
- push!, append! AbstractVector implementations no longer work on Julia 1.11+
- 'make test' fails for julia-1.11.0-rc2 HOT 2
- `JULIA_NUM_THREADS` ignored when embedding Julia on a Windows system HOT 1
- Segmentation fault in Hermitian `eigen`/`eigvals` on nightly HOT 30
- Add ErrorHints stdlib HOT 5
- Load stdlib JLLs by path
- Consider dropping bfloat16 demotion pass HOT 1
- "Inferrable" `stdin` for use with trimmed programs
- `@test`'s use of kwargs is confusing and limits extensibility HOT 1
- Provide an error message if LD_PRELOAD is set during a Julia build or startup HOT 15
- Allow uplo='F' for Symmetric and Hermitian wrappers? HOT 2
- Shipped Windows binaries with new codesigning need `.ji` checksums updated HOT 3
- Emit OSC 9;9 escape sequences on Windows HOT 1
- Ambiguous behavior of Tuple constructor with mismatched array length HOT 2
- RetentionParameterEstimator hits incorrect assert in emit_masked_bits_compare HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from julia.