cmstrackerdpg / mlplayground Goto Github PK
View Code? Open in Web Editor NEWWeb application for collaborating to build ML models for Tracker DQM/DC
Web application for collaborating to build ML models for Tracker DQM/DC
Hi,
I have found a vulnerability within MLP login page, where unknown users can sign up for a new account with the procedure as follows:
Instead of having "Login with CERN" button directly taking users to CERN single sign-on, the button redirects to another page provided by django-allauth, which provides options for CERN sign-on, GitHub sign-on, and sign-in or sign-up for local accounts. We have to find a way to fix django-allauth directly.
Thanks,
Vichayanun
A login view is already implemented using the Django-provided LoginView
, but there is no direct link to the login URL (login/
).
Currently, there are only two from_csv
methods in the LumisectionHistogram1D
and LumisectionHistogram2D
models. This functionality should be moved to a separate module.
All data-X
attributes have been namespaced to data-bs-X
as of 5.0
The goal is to move away from Travis CI and have a more modular CI using Github Action.
Preliminary version should run:
The Task
will be a "container" which will group data so that it can be serialized and used by users for training and testing their models.
They should include:
ManyToMany
relationship with:
The main point to keep in mind is that a Task
should contain multiple histograms, and many histograms can belong to many different Task
s
@Abhit03 @XavierAtCERN please verify that a Task
should either aggregate a single type of histograms or a mix of them
Add the following minimal plots (either static or using altair)
Runs
Lumisections
Use generic display views for lists of items and items.
Using a yaml
configuration as @XavierAtCERN suggested, the user should be able to determine which datasets are to be dumped into the database when parsing Histogram Data Files
In order to run the Django project, ROOT
must be installed, along with the appropriate version of python and pyROOT
.
The Dockerfile
in the repository can create an UBI8 + Python3.8 + ROOT image, which can be used as the container to run the workflow in.
Currently, DIR_PATH_EOS_CMSML4DC
points to a single directory, but it will be useful to allow multiple ones.
Splitting them by :
could be a solution to retain the use of a single environmental variable.
Hi,
I just discovered that the histogram manager breaks when a histogram name containing a forward slash /
is selected in a filter. The URL below demonstrates the error message.
My initial explanation is that, since each row of the histogram manager contains the link to the visualiser in the form of ['visualize/(?P<runnr>[0-9]+)/(?P<lumisection>[0-9]+)/(?P<title>[^/]+)/\\Z']
, the histogram title containing a forward slash can break the pattern.
With this problem, we need some way to "sanitise" histogram titles so that it would not contain forward slashes that breaks the URL pattern defined in visualize_histogram/urls.py.
Thanks,
Vichayanun
Reader for the perRun/perLS data stored on EOS
Data integrity checks to keep track of potentially missing Runs/LS
Data discovery automation for operations (from RR, OMS, EOS, CertHelper, etc)
Logging of the above input tasks with potential (email/displayed) alarms/notifications.
Currently, each app has a separate /api
endpoint mounted on its URL (/lumisectionHistos1D/API
, /lumisectionHistos2D/API
).
Wouldn't it be clearer if all endpoints where under a common /api/
part of the URL?
E.g. (/api/lumisectionHistos1D/
, /api/lumisectionHistos2D/
).
Token authentication for now(?)
Currently, the only way to initiate the discovery of new DQMIO files is limited to running the discover_dqm_files
management command.
This should be executable via the API as well.
This functionality could be limited to staff
.
Optionally, a button may also be shown in the Histogram File Manager page.
The goal is to have a sensible url pattern to browse through runs / lumisections.
The preliminary pattern could be:
Hi,
MLP frontend redesign has been live for a few hours, but I already noticed a bug on the visualizer with the following error log:
TypeError at /visualize/316218/941/num_clusters_ontrack_PXBarrel/
unsupported operand type(s) for +: 'NoneType' and 'int'
This error is caused by some histograms not having x_bin
attribute. This attribute is required in this line: https://github.com/CMSTrackerDPG/MLplayground/blob/master/visualize_histogram/views.py#L62 Furthermore, those histograms also do not have x_min
and x_max
attributes, which is required in the plotting as well. I have checked the DB in production, and found that these histograms amount to 268 136 out of 447 754 1D histograms, more than half of the histograms in 1D lumisection database.
For the reference, here is an example link that breaks the backend: https://ml4dqm-playground.web.cern.ch/visualize/316218/941/chargeInner_PXLayer_1/
Upon further investigation, I found that these histograms are parsed from three 2018 CSV files, all parsed on 30 Sep 2022. The exact addresses for these files are as follows:
/eos/project/c/cmsml4dc/ML_2020/UL2018_Data/DF2018A_1D_Complete/ZeroBias_2018A_DataFrame_1D_1.csv
/eos/project/c/cmsml4dc/ML_2020/UL2018_Data/DF2018A_1D_Complete/ZeroBias_2018A_DataFrame_1D_10.csv
/eos/project/c/cmsml4dc/ML_2020/UL2018_Data/DF2018A_1D_Complete/ZeroBias_2018A_DataFrame_1D_11.csv
According to histogram file manager, these three files are the only ones that are parsed out of all files in 2018A era. For now, I will re-parse the file using the manager and see if this issue is fixed.
Thanks,
Vichayanun
Currently, on each parsing command issued for a HistogramDataFile, a new thread is spawned (from within Django) to parse the file.
Several issues with this approach:
The goal is to be able to access some information about run and lumisection histograms from the navbar. So far, a simple list of the variables available is enough. Will add more information and visualization (time series, ...) later.
Work on the front end to make the Parsing button functional
The goal is to fill the database with run certification taken from the Run Registry, relying on a static file as a starting point. Then write a preliminary frontend to view the fraction of lumisections GOOD or BAD for the various subdetectors / objects.
Example:
To filter lumisections of run 297056
, one has to query
https://ml4dqm-playground.web.cern.ch/api/lumisections/?run=13
instead of:
https://ml4dqm-playground.web.cern.ch/api/lumisections/?run=297056
Upon deployment, the ML Playground displays very long delays in rendering responses through the DRF API.
Initially, we were getting lots of Broken pipe
messages, which were due to the proxy cutting off the connection due to the server not responding in time. Then, by increasing the proxy timeout to 180s, we were able to proceed with debugging the issue.
daphne
with asgi
under settings.py
gunicorn
with wsgi
under settings.py
manage.py runserver
daphne
. Responses take up to 2 seconds, maximum, meaning that the database is not the problem.DEBUG
. No change.settings.py
, so that we can see what's going on in Django's mind while rendering that. You can see below that the queries themselves are fast. A long time is spent between the last SELECT
query and the start of the HTTP 200
response (~2.5 min).10.76.10.1:34256 - - [11/Apr/2022:13:27:40] "GET /api/histogram_data_files/?format=json&page=7" 200 17015
DEBUG - 2022-04-11 13:27:40,170 - http_protocol - HTTP response complete for ['10.76.10.1', 34256]
DEBUG - 2022-04-11 13:34:22,979 - http_protocol - HTTP b'GET' request for ['10.76.10.1', 34540]
DEBUG - 2022-04-11 13:34:22,990 - selector_events - Using selector: EpollSelector
DEBUG - 2022-04-11 13:34:23,118 - utils - (0.009) SELECT "django_session"."session_key", "django_session"."session_data", "django_session"."expire_date" FROM "django_session" WHERE ("django_session"."expire_date" > '2022-04-11T13:34:22.999654+00:00'::timestamptz AND "django_session"."session_key" = 'f3nubvfv7t7k6eltpk5e54ovzaijv89s') LIMIT 21; args=(datetime.datetime(2022, 4, 11, 13, 34, 22, 999654, tzinfo=datetime.timezone.utc), 'f3nubvfv7t7k6eltpk5e54ovzaijv89s'); alias=default
DEBUG - 2022-04-11 13:34:23,118 - utils - (0.009) SELECT "django_session"."session_key", "django_session"."session_data", "django_session"."expire_date" FROM "django_session" WHERE ("django_session"."expire_date" > '2022-04-11T13:34:22.999654+00:00'::timestamptz AND "django_session"."session_key" = 'f3nubvfv7t7k6eltpk5e54ovzaijv89s') LIMIT 21; args=(datetime.datetime(2022, 4, 11, 13, 34, 22, 999654, tzinfo=datetime.timezone.utc), 'f3nubvfv7t7k6eltpk5e54ovzaijv89s'); alias=default
DEBUG - 2022-04-11 13:34:23,127 - utils - (0.005) SELECT "auth_user"."id", "auth_user"."password", "auth_user"."last_login", "auth_user"."is_superuser", "auth_user"."username", "auth_user"."first_name", "auth_user"."last_name", "auth_user"."email", "auth_user"."is_staff", "auth_user"."is_active", "auth_user"."date_joined" FROM "auth_user" WHERE "auth_user"."id" = 1 LIMIT 21; args=(1,); alias=default
DEBUG - 2022-04-11 13:34:23,127 - utils - (0.005) SELECT "auth_user"."id", "auth_user"."password", "auth_user"."last_login", "auth_user"."is_superuser", "auth_user"."username", "auth_user"."first_name", "auth_user"."last_name", "auth_user"."email", "auth_user"."is_staff", "auth_user"."is_active", "auth_user"."date_joined" FROM "auth_user" WHERE "auth_user"."id" = 1 LIMIT 21; args=(1,); alias=default
DEBUG - 2022-04-11 13:34:23,136 - utils - (0.006) SELECT COUNT(*) AS "__count" FROM "histogram_file_manager_histogramdatafile"; args=(); alias=default
DEBUG - 2022-04-11 13:34:23,136 - utils - (0.006) SELECT COUNT(*) AS "__count" FROM "histogram_file_manager_histogramdatafile"; args=(); alias=default
DEBUG - 2022-04-11 13:34:23,138 - utils - (0.001) SELECT "histogram_file_manager_histogramdatafile"."id", "histogram_file_manager_histogramdatafile"."filepath", "histogram_file_manager_histogramdatafile"."filesize", "histogram_file_manager_histogramdatafile"."data_dimensionality", "histogram_file_manager_histogramdatafile"."data_era", "histogram_file_manager_histogramdatafile"."entries_total", "histogram_file_manager_histogramdatafile"."entries_processed", "histogram_file_manager_histogramdatafile"."granularity", "histogram_file_manager_histogramdatafile"."created", "histogram_file_manager_histogramdatafile"."modified" FROM "histogram_file_manager_histogramdatafile" LIMIT 50 OFFSET 300; args=(); alias=default
DEBUG - 2022-04-11 13:34:23,138 - utils - (0.001) SELECT "histogram_file_manager_histogramdatafile"."id", "histogram_file_manager_histogramdatafile"."filepath", "histogram_file_manager_histogramdatafile"."filesize", "histogram_file_manager_histogramdatafile"."data_dimensionality", "histogram_file_manager_histogramdatafile"."data_era", "histogram_file_manager_histogramdatafile"."entries_total", "histogram_file_manager_histogramdatafile"."entries_processed", "histogram_file_manager_histogramdatafile"."granularity", "histogram_file_manager_histogramdatafile"."created", "histogram_file_manager_histogramdatafile"."modified" FROM "histogram_file_manager_histogramdatafile" LIMIT 50 OFFSET 300; args=(); alias=default
DEBUG - 2022-04-11 13:37:01,528 - http_protocol - HTTP 200 response started for ['10.76.10.1', 34540]
DEBUG - 2022-04-11 13:37:01,530 - http_protocol - HTTP close for ['10.76.10.1', 34540]
DEBUG - 2022-04-11 13:37:01,531 - http_protocol - HTTP response complete for ['10.76.10.1', 34540]
Even after a 504
error, on the immediate next reload of the same page the page loads instantly (due to a 1-min cache of the responses), meaning that the response has already been cached from the first request, even if has not been returned to the client.
We need to 👽probe deeper👽 to what's going on inside Django between the end of the DB query and the start of the HTTP response.
Find a way to render an HTML page which:
FilePathField
in admin creates a dropdown)forms.py
does not seem a very valid option for this, however the FilePathField
seems useful (https://docs.djangoproject.com/en/4.0/ref/forms/fields/#filepathfield)HistogramDataFile
table.HistogramDataFiles
automatically (e.g. python manage.py discover_dqm_files
)The goal is to have a frontend stable enough to start developing functional tests.
While some information from OMS is actually available from the Run Registry and could be extracted in a similar way than certification, it would be better to access OMS information directly and compare both sources of information.
Histogram files (currently only in CSV format but also keeping future nanoDQM format in mind) are parsed via a management command.
For huge (>2GB) files, this may take a lot of time or may even crash. Having appropriate models for storing file parsing history/progress can help a lot with the app's bookkeeping.
After closing #30 , the discover_dqm_files
management command which relied on the file list generated by FilePathField
will not work.
The goal is to fill the database with lumisection certifications taken from the Run Registry, relying on a static files (one by subdetector/object) as a starting point. Then write a preliminary frontend to view the flag for each lumisection.
Required:
api/viewsets.py
api/serializers.py
api/routers.py
api/filters.py
(Optional)Move from foreign key id to value for APIs in order to make results more relevant. To do so, see here.
Currently only shows POST errors. Should also show successful POST, reset fields etc.
LumisectionHisto1D
and LumisectionHisto2D
are to be combined in a common model (LumisectionHistoBase
) that LumisectionHisto1D
, LumisectionHisto2D
will inherit from.
lumisection
date
title
data
entries
x_min
x_max
x_bin
source_data_file
Currently, the front-end queries the API every 5 seconds, leading to constant use of the ModelSerializer
which involves a lot of background function calls and high CPU usage.
Instead of writing custom serializers, caching the replies for 1 min should be a reasonable tradeoff of latency vs performance.
Remove unnecessary use of pandas when possible.
The goal is to use the bottom navbar to provide links to the urls allowing to cross-check the information provided (OMS, RR) or the location of the files used (eos, ...)
On the histogram file manager, filtering should be added for better management of the files
django-filters
and configure custom filteringA declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.