Comments (7)
It happened to me before, when data do not fit to RAM. Try to decrease dataset and check if you have the same error.
from otto_2015.
I have 64gb of ram, additionally I've set 256gb swapfile, but problem still occurs.
Now, I suspect this line: sys.path.append('/Users/ef/xgboost/wrapper'). I didn't update it.
from otto_2015.
This line doesn't matter.
I've tried to run directly in spider with hardcoded params - now, script stalls instead of segmentation fault. Still all cores used but usage in total is about 20%...
This is a similar issue:
dmlc/xgboost#209
from otto_2015.
Did you try to reduce dataset (take first 10000 lines only, for example)? My guess is that if it is not the memory problem, then xgboost stalls because of some problem with data.
from otto_2015.
I've shrinked both: test.csv (first 10k rows) and train.csv(random 10k rows - to have all labels) and there is no segmentation fault or stalling.
Btw. how much RAM do you used for full dataset?
from otto_2015.
I work on Mac with 32 Gb RAM. Try to increase dataset gradually. I suppose the problem with some line in the dataset, probably, some entries are not appropriate for xgboost (it could be the problem with NA values for example).
from otto_2015.
I've tried a few combinations, but this time except segmentation fault in logs, I got more meaningful message in RStudio:
train - all, test - 10k first
combine.preds("train_raw/model.xgbx", 10)
Reading epoch 0 ...
Reading epoch 1 ...
Reading epoch 2 ...
Reading epoch 3 ...
Reading epoch 4 ...
Reading epoch 5 ...
Reading epoch 6 ...
Reading epoch 7 ...
Reading epoch 8 ...
Reading epoch 9 ...
Error in rowSums(actual * predicted) :
error in evaluating the argument 'x' in selecting a method for function 'rowSums': Error in actual * predicted : non-conformable arrays
train - all random, test - all random
...
Error in rowSums(actual * predicted) :
error in evaluating the argument 'x' in selecting a method for function 'rowSums': Error in actual * predicted : non-conformable arrays
train - 60k random, test - 60k random
...
Error in rowSums(actual * predicted) :
error in evaluating the argument 'x' in selecting a method for function 'rowSums': Error in actual * predicted : non-conformable arrays
train - 30k random, test - 10k first
Error in cbind(data.pred, as.matrix(data.pred.epoch)) :
number of rows of matrices must match (see arg 2)
from otto_2015.
Related Issues (4)
- Why create a new NeuralNet instance every epoch? HOT 1
- utils.R - data paths HOT 1
- rgf.model HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from otto_2015.