Comments (12)
Yes, your assumption is totally correct but in your code you're not doing it, your doing this:
timeseries = np.array(data[i:i+train+predict])
if scale: timeseries = preprocessing.scale(timeseries)
x_i = timeseries[:-1]
y_i = timeseries[-1]
i.e. you take the full "timeseries" sliding window including X_test and Y_Test and you standardize it using the preprocessing.scale() function.
Instead of writing timeseries = np.array(data[i:i+train+predict])
you should have written timeseries = np.array(data[i:i+train])
, take the mean and std from this data, and use it to standardize x_i and y_i
from deep-trading.
@JekyllQuant seems like you have right, just looking on the picture you can see that our predictions are too late. Does it mean that regression approach doesn't work at all? How do you think?
from deep-trading.
Hi, @JekyllQuant
Thank you for your interest, let's try to understand what's the problem.
This because Y_test, at test time, isn't known, so you cannot calculate the mean and the std used in the standardization process to normalize the full sliding window sample.
Let's say we already have some trained neural network for regression. We have some live data from stock market, close prices of last 20 days. They are definitely KNOWN.
x_i = timeseries[:-1]
We want to forecast 21st day price,
y_i = timeseries[-1]
,
but network outputs the scaled result and we want to restore it. My hypothesis is that I can just take mean and std from already KNOWN 20 days and assume that it's good enough estimate to restore the next day price. I do not use anything from the future, just statistical information from the past trying to restore future.
Correct me, if I am wrong please in my thoughts or implementation.
from deep-trading.
Sincere apologies for the very late reply Alex, I completely missed your post.
Does it mean that regression approach doesn't work at all? How do you think?
I cannot say much, but I wouldn't throw everything away. Maybe I would check if the information the NN is giving you can be useful.
For sure now it doesn't look quite good, aesthetically speaking.
from deep-trading.
Guys dont forget that you cant normalize data against different mean and std. it will give not correct results. So if you calculate mean and std for you first window, you should use then for all your data, including predicted data.
from deep-trading.
Hi Alex. Thanks for giving out your code, it's a very good example.
But, I still can't run volatility.py, because I CAN'T find "feature_extractor" module, Would you mind tell me where from it ?
thanks!
from deep-trading.
@cn3c3p added needed files, but I think they even weren't used in the code :)
from deep-trading.
@Rachnog which is the most effective algo after this fix? RNN?
Thanks
from deep-trading.
Consider the mean as M and inputs as x1,x2,.....xn. And we are trying to predict x_{n+1}.
After t
from deep-trading.
Hi, did anyone ever manage to fix this strange behaviour?
from deep-trading.
Hi, anyone have a solution to standardization of Test Datased in the goal to use this code in real world?
from deep-trading.
from deep-trading.
Related Issues (18)
- License is missing or I can't find it
- X_test2 and X_test1 missing
- Unable to execute Code due to => TypeError: 'generator' object is not subscriptable HOT 1
- undefined in train_nets.py
- simpler_forecasting table.csv is opened as a binary file
- Slice indices issue in Simple time series forecasting.ipython
- FileNotFoundError: File b'AAPL.csv' does not exist
- Wrong data preprocessing leading to better results | Multimodal project HOT 1
- Error in data-preprocessing in multivariate/multivariate.py
- Graphs not showing the training/test data
- StandardScaler() for OHLCV data HOT 1
- Input shape for simple_forecasting
- Indexing error in multimodel/process_data.py
- keras-cannot-import-name-np-utils
- Need versions of python used along with libraries HOT 2
- hyperopt.py not found
- Versions of dependent libraries needed - keras-2.0.8 fails HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deep-trading.