vrodriguezf / deepvats Goto Github PK
View Code? Open in Web Editor NEWDeep Visual Analytics for Time Series
License: Apache License 2.0
Deep Visual Analytics for Time Series
License: Apache License 2.0
In GitLab by @dmt on Jan 21, 2021, 19:36
Several warnings occur during the execution of the app, which may be due to incompatibilities with the hdbscan library update.
We have to find a way of deploying the project with docker-compose
in a way that the gpus of the server (if any) are usable by the container. In classic docker, it is as easy as adding the flag --gpus all
to the command docker run
. However, I do not know how to do this in compose.
This will be important in a few weeks because we are installing more GPU servers in the group and thus, this proejct should be moved to a gpu-based machine.
In GitLab by @lgs on Oct 15, 2020, 13:19
Merges patch-1 -> master
Update Dockerfile. Remove the lines to share data folder before switching to the user (it's no longer necessary)
In GitLab by @vrodriguezf on Mar 12, 2020, 18:57
Hi,
This issue is a way to keep track of the work done in this project and be aware of what is left.
I just made the initial commit which contains, as most important file, the notebook nbs/01_Timecluster_replication.ipynb
. In this notebook I explore the methodology proposed n the Timecluster paper with one piece of data from one of the longwalls.
In the last section of that notebook, I want to start exploring how to extend that methodoogy so that the expert cannot only see which time window corresponds to each point in the 2d space, but also the most important variables of that time window.
Also, I still have to find a way to validate the replication, since the data from the paper is not available.
In GitLab by @lgs on Dec 15, 2020, 18:06
Merges issue29_solved -> master
Clustering functionality has been added using reticulate and hdbscan (python). We have also added functionality to modify its parameters.
We have added reactivity to the embeddings value filters. A "renderUI to define the maximum points values to show of the embedding" has been created
The zoom functionality has been updated (now it works by clicking on a button)
dyShading plotting process updated. Previously there was a large overlap of rectangles when they were drawn in the dygraph, this is now resolved
In GitLab by @vrodriguezf on Nov 6, 2020, 11:37
In GitLab by @lgs on Oct 15, 2020, 13:24
Merges lgs-master-patch-09882 -> master
Updated docker-compose to fix the problem of /data/PACMEL-2019 permissions
In GitLab by @vrodriguezf on Mar 18, 2020, 18:20
Take the code from @pml and add it to the module 00_load
to be aligned in terms of preprocessing with the rest of the project.
In GitLab by @vrodriguezf on Mar 31, 2020, 13:46
Ensure that the validation set is properly selected, in terms of respecting the time indices. With time series, you cannot split your data into training and test randomly, you have to take one part of the series (for example, the last) as validation and leave the rest for training.
In GitLab by @vrodriguezf on Nov 6, 2020, 11:14
If possible, using the mouse wheel would be great for this, otherwise a couple of +/- buttons can make the trick.
In GitLab by @lgs on Oct 13, 2020, 17:55
Currently, when several users use the docker-compose up -d command within a single server, inconsistencies appear and containers are overwritten.
To avoid this it is necessary to define a project_name when docker-compose is invoked. To solve this and to automate the process it is proposed:
Use the .env file to define default project names. https://stackoverflow.com/a/44924392
Create a makefile that allows you to invoke docker-compose by automatically defining a name associated with the id of the user who invokes it (something like timeclusteR_extension-$(id -u) ) https://stackoverflow.com/a/59965344
In GitLab by @vrodriguezf on Oct 14, 2020, 12:09
null
In GitLab by @lgs on Dec 3, 2020, 12:56
New versions of nbdev do not work with the traditional Jupyter's Magic Lines.
i.e: %nbdev_export should be replaced by #remove as new documentation explains (https://pypi.org/project/nbdev/1.1.5/)
In GitLab by @vrodriguezf on Mar 18, 2020, 18:16
Just in the same way that the authors of the Timecluster paper show in this video, having a webapp of the replication and extension coded in this project would allow for easy sharing and insights.
Possible options for implementation include: voila (voila-ipyvuetify), R shiny
In GitLab by @lgs on Oct 30, 2020, 15:25
Create a new Jupyter Notebook that:
Right now, the only way of showing clusters in the visualization app is to click on the button "calculate and show clusters", which is what it is done in the original Timecluster paper. HOwever, in order to integrate some of the works of our colleagues in Poland, it would be interesting to add the possibility to get the clusters from a logged artifact.
The artifact would be a ReferenceArtifact
logged by the dr_run, just like it is done with the embeddings
, and it could be as simple as an array of cluster labels.
In GitLab by @vrodriguezf on Oct 24, 2020, 16:57
Right now, the validation set used during the encoder training is just a random 20% split of the input artifact, made automatically by Keras using the argument val_pct
. This implies that the validation data is used for the normalization of the training data, and this is a bad practice. The validation data must be normalized independently, using the means/stds of the training data.
The easiest way to solve this problem is to use a separate artifact as validation set, instead of splitting the one used for training. As an example, we can use days 1-10 for training. days 11-13 for validation, and days 14-15 for test. Note that, in the call to model.fit
, the argument validation_split
is no longer applicable, and we have to use the argument validation_data
(See docs)
Update Dockerfile-rstudio file to adapt it to new Python library updates.
In GitLab by @vrodriguezf on Nov 2, 2020, 10:55
Define a function plot_top_losses
(either in the DCAE
module or in utils
) with this header:
@patch([Model class in Keras])
def plot_top_losses(self:[Model class in Keras], k, largest=True, **kwargs):
"Take the validation data of model self, compute the model losses for every item there, and sort the results. If `largest` is True, the validation losses will be sorted from larger to lower. Once they are sorted, take the k first items based on this order and plot the predictions."
My idea is to call it like model.plot_top_losses
given a Keras model in the variable model
, therefore the @patch
. @path is a decorator from fastcore to add methods to a class on the fly.
In GitLab by @lgs on Oct 22, 2020, 17:13
Use data from the test_data artifact to check visually the performance of the DCAE.
In GitLab by @vrodriguezf on Oct 16, 2020, 16:19
In the Timecluster paper, the 2D embedding plot is a scattered connected plot, where each point is marked as a hollow square:
In the DR notebook it is more or less the same, although the plot is wrognly logged into weights and biases. This is how the plot looks in the notebook:
And this is how it looks one in wandb:
As it can be seen, the points are filled with a colour. The plot in shiny also has filled marks.
I'd like to have both the wandb plots and the shiny ones with hollow marks as well, so that the plot scales better when it has thousands of points.
In GitLab by @vrodriguezf on Oct 14, 2020, 13:30
There can be problems if, for example, the DR wants to use an artifact with 15 variables, but the associated DCAE has been trained with an artifact with 10 variables. The inputs wouldn't match.
There's already a scaffolding of the function check_compatibility
in the DR
notebook:
%nbdev_export
def check_compatibility(dr_ar:TSArtifact, dcae_ar:TSArtifact):
"TODO: Function to check that the artifact used by the DCAE and the artifact that is \
going to be passed through the DR are compatible"
ret = dr_ar.metadata['TS']['vars'] == dcae_ar.metadata['TS']['vars']
# Check that the dr artifact is not normalized
return ret
For now it just checks that the variables used are the same (in number and names). There are other things that has to be checked here such as:
TS.freq
inside the metadata.TS.has_missing_values
, it has to be false)In GitLab by @vrodriguezf on Mar 18, 2020, 18:14
According to the paper: The number of feature maps, size
of filter and depth of the model are set based on the recon-
struction error on validation set.
Use weights & biases to analyse which hyperparameters are best for the mining data.
In GitLab by @lgs on Oct 29, 2020, 10:49
null
In GitLab by @dmt on Jan 19, 2021, 10:28
Merges dmt-master-patch-62643 -> master
In GitLab by @vrodriguezf on Oct 14, 2020, 12:47
It is non sense that something as important as the creation of the dataset artifacts is in a draftsheet notebook. The easiest way I can think of moving this is to create a notebook 00_dataset_artifacts.ipynb
. We can even think of creating a function that is exported to the library and gather the functionality of the notebook, i.e., creating an TSArtifact from a given dataframe, with a specified configuration.
On a side note, I would rather remove the nomenclature 00_, 01_, for notebooks that don't belong explicitly to the project pipeline. Therefore, thinks like 00:load should be renamed as load.
In GitLab by @lgs on Dec 15, 2020, 17:46
null
In GitLab by @vrodriguezf on Apr 1, 2020, 18:53
In GitLab by @vrodriguezf on Mar 30, 2020, 15:55
From the Timecluster paper:
"For validation purpose, we also compare an automatic clustering approach. A hierarchical clustering method (HDB- SCAN) [12,20] is used to generate the most significant clusters as a density-based clustering algorithm. It requires only one parameter which represents the minimum size of the cluster. We use the sklearn package, and we use the hdbscan package as available on PyPi in order to determine the number of clusters"
Implementing this is useful to provide automatic clusters to other PACMEL projects such as the ones devoted to shapelet generation.
In GitLab by @lgs on Oct 29, 2020, 10:57
The plot_validation_ts_ae function takes the first element of each window and plots it, this way it is possible to see the whole time-series because we have only used a stride 1 during the DCAE training. If we use a value higher than 1, the graph will not make any sense.
We need to change the current code, to plot only one window selected in the function parameters (i.e. window_num):
for i,ax in zip(range(original.shape[2]),fig.axes):
ax.plot(original[:,0,i], label='Original Data') # Change this
ax.plot(prediction[:,0,i], label='Prediction') # Change this
In GitLab by @vrodriguezf on Oct 14, 2020, 13:18
I am not sure if running a single DCAE training (i.e., notebook DCAE.ipynb
) would log a config.yaml
into the wandb files of the run. Running it from a sweep does it for sure. This is important because the notebook DR
, when it restores a DCAE run at the beginning, needs of a config.yaml
file.
In GitLab by @lgs on Nov 17, 2020, 13:08
We have observed some anom
We have observed some anomalies in the JNK dataset. One of them is shown in the Figure. You can see that there is an abrupt change in all sensors for a minute (all signals except SM_ShearerLocation have the value 0).
We wonder why this stop is due to an error in data acquisition or are small machine stops for some operational reason? (important to remark that the sensors that measure temperature are also set to 0)
Translated with www.DeepL.com/Translator (free version)
In GitLab by @lgs on Oct 15, 2020, 17:16
Merges lgs-master-patch-73510 -> master
In GitLab by @vrodriguezf on Oct 14, 2020, 13:34
Execute, for a given DR artifact, the UMAP reducer (notebook DR
) multiple times with different configurations.
To do this automatically, the library papermill
has to be used, in combination with a wandb_group
to visualize the results of the experiment in a separated URL in wandb. We should discuss how to do this further.
Another option could be to create a sweep, just as with the DCAE case. However, I don't know if sweeps can be run without a metric to optimize...that's the case here, we are just exploring UMAp parameters, without a clear optimization metric.
In GitLab by @vrodriguezf on Oct 14, 2020, 13:06
Right now, the test artifact used in the Dimensionality Reduction (DR) is a subset of 1000 rows of the same artifact used for training the associated DCAE. Therefore, the data loaded from the artifact is already normalized correctly.
However, if the DR artifact is different than the DCAE one (the real and useful case), the DR one has to be normalized according to the means and stds contained in the metadata of the DCAE one. This implies that, for convenience, artifacts that are going to be used for DR should be logged without normalization. Otherwise, when using them in the DR, they should be first denormalized and normalized again with the DCAE metadata.
In GitLab by @vrodriguezf on Oct 20, 2020, 18:21
Modify server.R
, more specifically, the computation of the reactive tsdf
, so that when calling used_artifacts
, the correct artifact is looked for instead of assuming that the correct one is the first. The question is: how to find is it the correct one?
In GitLab by @lgs on Oct 29, 2020, 11:40
Update README with info. about useful gist for Jupyter Lab (shortcuts, etc)
In GitLab by @vrodriguezf on Mar 18, 2020, 18:04
Only one day of logs is being used in the nb 01_timecluster_replication
. It is likely that better patterns are found if many days are taken into account.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.