Comments (7)
In splitting.py, the left/right_indices_buffer will use up 8GB for 10^9 rows. If that causes swapping, the performance benefit of multithreading (which requires these buffers) are most probably not worth it. Would it be an option to disable this?
I don't see how we could have multithreading at that level anymore. You suggest disabling thread-based parallelism for the split_indices
operation? Maybe that could be an option. @NicolasHug might know better how LightGBM does for this part of the code.
from pygbm.
Would it be an option to disable this?
Technically yes... I suppose we could use a single-threaded quick-sort like partitioning scheme.
Is there also a method that could without the buffer
I don't think so, or at least no with the current strategy. those arrays are used to that sample indices don't overwrite each other
from pygbm.
@NicolasHug might know better how LightGBM does for this part of the code
I haven't checked again but I don't think they have an option to disable parallel splitting.
@maartenbreddels could you check if you have the same issue on LightGBM? Note that they are reusing allocated data like we plan to do in #81 so we need to take this into account
from pygbm.
There are some parallel in place partition algorithms: http://www.lsi.upc.edu/~lfrias/research/parpar/wea08.pdf
they don't appear super trivial, not something I'd do in 1 evening.
But I think one of the buffers could be avoided, that would already save a bit. Would you be interested in a PR that does either a single threaded split, or uses 1 buffer, or both? I can't promise I can do it, but if the 1 buffer PR makes the code less readable, and that is a reason not to merge, I won't bother.
could you check if you have the same issue on LightGBM?
I cannot use LightGBM as it is now, the current implementation makes at least 2 memory copies, my vaex-ml hack avoids 1 copy, but still the memory usage is excessive.
My plan is to see what is possible with pygbm (much easier to understand, and easier to edit), and possible see how they can be translated to lightgbm.
from pygbm.
I think a single-threaded version would be welcome and should not be too complicated to add.
I'd be curious to know how to avoid using one of the two arrays though!
from pygbm.
I think a single-threaded version would be welcome and should not be too complicated to add.
I'll open an PR for that, any guidelines for how this should be configurable?
I'd be curious to know how to avoid using one of the two arrays though!
I thought of using the sample_indices for the 'left' indices, and a scratchpad/buffer for the 'right' indices. Basically, sample_indices takes over the role of left_indices_buffer. That should work right?
from pygbm.
If you can make sure that no entry in samples_indices
gets overwritten before it's written into the other buffer then I guess so. But there's the "if" ^^
any guidelines for how this should be configurable?
Let go simple for now, you can try passing a parameter e.g. parallel_splitting
from BaseGradientBoosing
all the way down to SplittingContext.__init__
, and make split_indices
dispatch to either split_indices_parallel
or split_indices_single_thread
. We can work out the details later.
from pygbm.
Related Issues (20)
- API documentation is broken HOT 1
- All the examples require lightgbm HOT 1
- Allow score monitoring regardless of early stopping
- Optimize score loss computation
- Remove empty slice check (numba fixed the issue)
- Reuse grower (and thus the splitter) instead of creating a new one
- Updating to Scipy 1.2.0 breaks loss tests... HOT 2
- Avoid ordered_gradients? HOT 7
- Remove constant_hessian_value? HOT 1
- sum_gradient and sum_hessians computation in find_node_split_subtraction HOT 4
- Optimize categorical crossentropy gradient update HOT 3
- _update_raw_predictions() throws a deprecation warning HOT 1
- numba-integration-test failure HOT 6
- Status of this project? HOT 2
- Implement native support for missing values
- did you stopped development since since can not do better than lightGBM pr Xgboost pr catboost? HOT 4
- Implement histogram recycling to improve memory efficiency
- Recent Numba not usable with pygbm HOT 1
- Parallel splitting fails in nopython mode
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pygbm.