kescardoso / datasetbucket Goto Github PK

Sometimes, the PDF title isn't correct. We need to check that it is getting passed the correct string. It is possible this might be fixed when we fix the issue of the old datasets not being downloaded.

• Break down number of samples for each value
• Plot histograms/diagrams representing distributions for each feature
• Other analytics improvements

Add charts/graphs to the report to show the analysis visually

Add population demographics of each country to be used in the analysis of the data.
We want to compare the demographics in the data to the demographics in a given population, to check if the data is representative of the whole.

On add_dataset, the file is only successfully uploaded in local deployment.
After deploying our app to heroku, a fix is needed: either link files uploading to mongodb via b64 or via a cloud storage like s3 to handle user uploads in the deployed version (ultimately necessary).

edit and update Readme and Devpost descriptions

image analysis files need their paths updated. Images in a dataset are not currently getting analyzed or reported.

@kescardoso I want to delete the .vscode folder and .DS_Store in main, but I don't want to break the deployment. Can we delete it safely?

Need to rewrite time.sleep() statements into async statements to deal with multithreading.
time.sleep() is used in multiple files.

Maybe try:
https://realpython.com/async-io-python/

When a dataset is analyzed, the old zip + unzipped datasets need to be deleted.
Or, we need to delete the zip after we have sent the report

The results from the analysis will be available to download as a pdf, in addition to being displayed in the html.

Research how to deploy on Difinity
Once app is created, deploy

This fix can be achieved with jquery, a good example in this stackoverflow https://stackoverflow.com/questions/5757801/javascript-how-to-show-spinner-until-file-downloads

Categories:
after installing the location select functionality, bugs in category selection appeared

• previous multi selection not returning from db on edit view
• string and list not rendering properly (again!)

To be Implemented:
On datasets.html

• Dynamic Search
• Pagination

Build the initial app with flask.
Ensure that we can parse files well with flask

Variables names are in the incorrect format.
Code needs more comments

Add:

• Footer
• About/instructions page
• Credits

Improve user-friendly and aesthetic features:

• color key
• buttons
• typography

Develop user profile with more functionalities (not a priority, but cool if we have time)

• hyperlink author on datasets.html
• NavBar with active user session tag
• User info (name, position, photo, links, etc.)
• User tasks, activities, contributions, stats....

Some things left to figure out and learn 🤔

We want to include 3 diffterent formats supported:
{'root': {dict}} ,
{'content': {}, 'annotation':[{'labels':{}},{'points':{}}], 'extras':{}} , currently the only accepted format
{'keyword':{}, 'keyword':{}, 'keyword':{}}

Create Location functionality for datasets with dynamic query and db binding

Feature to be implemented as an alternative:

Figure out a way to add a link input to Add New Dataset, to include pdf reports from a cloud drop box.
Specially in case the JSON file handling doesn't integrate well to the project

Give the functionality of multiple choice for categories picking

Create the categories Query and management sessions:

• List categories with links
• Create Add New and Edit category forms

Need to be able to handle images and text files

Add about page to explain bias to the user and explain how we will analyze their data.

Extra Features:

• hyperlink categories and locations on datasets.html Post view, to point to datasets in that category (queried datasets page).

• Research installation for B64 string on a mongo-flask application
• How to handle file via mongodb string back and forth to python-flask

Currently, the analysis and recommendations generated by the program is lacking. For CSV files, it only calculates basic metrics (like mean and variance) and generates a few histograms. For JSON files, the program doesn't give much analysis as well.

We need people to improve on the PDF analysis/recommendations!

Need to decide how we want to parse and analyze data
Need to determine the best way to determine if a dataset is biased

Created a page with a text input form, to analyze datasets from a Kaggle command or url.

The page is not pragmatically functioning, it needs further developments and installation to link the HTML form to the bias auditor app.

Link to the deployed version: https://datasetbucket.herokuapp.com/analyse_data