Comments (5)
This issue will likely require many significant changes. The requirements I see from the top of my head:
- A
DataSet
abstraction that is compatible with "standard" data set loaders (torch and keras) as well as with pandas and numpy - The preprocessing part of VectorModel should be compatible. This is not easy: see comment above. We need to think how to deal with non-fitted preprocessors when a DataSet is passed
- Ideally, we would also want it to work with our caching utils. For that, semantic indices need to be part of DataSet somehow.
- Validation set splitting and cross-validation logic needs to be adjusted. How do we split a generator? Maybe instead we require to pass a validation loader (like in pytorch/keras) - what about cross validation then?
There will be many more difficulties which I cannot anticipate now. Certainly the issue should be split into multiple issues/PRs when the time to address it comes
from sensai.
It might be worthwhile to take a look at existing libraries such as NVTabular, which supports feature engineering and preprocessing with a particular (exclusive) focus on neural networks and which can be integrated with fastai.
I am not saying we shouldn't add more neural network features to sensAI, but if what is needed has already been done by others, we shouldn't reinvent the wheel.
If we go ahead with all this, it might make sense to restrict ourselves to being compatible only with finite datasets, e.g. support only torch.utils.data.Dataset
but not IterableDataset
(as is already the case for sensai.torch.torch_data.TorchDataSet
). It would make many things much easier, and it shouldn't be a practically relevant restriction for the vast majority of users.
from sensai.
A reasonable approach to handle this using current mechanisms (for torch models) is to have DataFrames which contain only meta-data (e.g. filenames/paths or other references to the actual data) and which do fully fit in memory and to make the TorchDataSet
implementation (injected via a TorchDataSetProviderFactory
) dynamically load the actual data in its iterBatches
method.
Maybe we do not need additional mechanisms to handle this sort of thing.
from sensai.
Yes, this sounds like a reasonable approach for many applications, where normalizers and feature extractors don't need to be fitted on the non-loaded data. What I originally had in mind was a support for training on generators of data frames (or arrays). Several libraries help building such generators, augmenting data on the way which can come in pretty handy. It might well be that these tools for data augmentation can also easily be used within an implementation of iterBatches
, thus removing my main motivation for generators.
We could either close this issue or put it on ice until one of us actually uses sensai for such data sets and shares hands-on experience (I would prefer the latter option)
from sensai.
The approach I described above has recently been added to sensAI with class TorchDataSetFromDataFramesDynamicallyTensorised
. To make use of it in torch vector models, one only needs to inject a TorchDataSetProviderFactory
that creates a provider which will in turn create such TorchDataSet
instances.
from sensai.
Related Issues (20)
- Support models that can input and predict multidimensional tensors HOT 1
- Write evaluation methods for TensorModel
- Build fails upon force-push to develop HOT 1
- geopandas cannot be installed under Windows HOT 2
- Add pytorch-lightning to sensai[torch] dependencies HOT 1
- Refactoring of the clustering package, bundling geo-analytics stuff in a separate optional package
- Do not track rst-files in repository HOT 4
- Fix publish package github workflow HOT 1
- Generalize SpanningTree and graph-related methods HOT 2
- Support passing separate validation data set to pytorch-lightning based models HOT 12
- Notebook tests can fail HOT 2
- Notebook Test Failure: PyTorch-Lightning ImportError HOT 2
- Rule-Based Models' fit interface
- Notebook test failure due to TypeError in nbconvert HOT 1
- Simplify/improve naming of evaluator/evaluation utility classes HOT 1
- Add tests for evaluation util classes
- Add tests for regression models HOT 1
- Add backwards compatibility tests from v0 to v1 HOT 1
- Update notebooks to v1 and re-enable them for the docs build
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sensai.