nlp4all / nlp4all Goto Github PK

NLP4All is a learning platform for educational institutions to help students that are not in data-oriented fields to understand natural language processing techniques and applications.

Shell 0.20% JavaScript 2.12% Python 72.65% CSS 1.41% HTML 23.03% Dockerfile 0.50% Mako 0.10%

education educational-institutions learning-platform nlp educational-resources language learning platform machine-learning natural-language-processing nlp-pipeline nlp-pipelines

nlp4all's Introduction

nlp4all

This is the repository for NLP4All. NLP4All is built on Flask.

nlp4all's People

Contributors

Stargazers

Watchers

nlp4all's Issues

Refactor data import

The new implementation reads the schema and imports the data straight after upload.

It will need to be benchmarked but it's likely that even though reading from the filesystem is slow, it may just be quicker to read the schema by iterating over all the rows, and then only import the selected fields, because right now importing 400k tweets takes about 40 minutes and then the subsequent delete query (pre index, indexed version is being tested now) takes an additional 20 minutes if done in a single query, and > 60 minutes if done in individual queries.

While we're at it, consider using MongoDB for document storage, and join with a unique key (or the document ID)

If we go this route (probably more performant) that will require hooks on the init-db and drop-db as well as when deleting and adding data sources.

Update
Version with gin index on the document column actually takes longer both for import and for property deletion. This makes sense as it actually has to update more information at each step, and probably the indexing doesn't extend to such deep nesting (it could, if the structure was consistent). Seems like MongoDB may be the way to go.

UPDATE 2

MongoDB has now been implemented, which is now a 3 minute import. Key deletion still takes around 8 minutes, but that just leaves one remaining task. Process the schema BEFORE import, and only import the required keys. the whole process will be much quicker and probably total around the same (3 min)

add a /setup url to init db and set up an admin user

decouple models and utils completely

integrate gitguardian vulnerability scanner

spacey part one: add library, create helpers

address gitguardian security issues

this requires
1 - removing the exposed secrets
2 - removing them from the history and any branches which they appear (slightly complicated)
3 - doing the same in the upstream repo https://github.com/arthurhjorth/nlp4all (which hopefully will be defunct at some point once github detaches the fork)

organise views into directories

[spike] how much work to unify branches

Branch unification may save time later on, so it is worth spending max 2 hours on investigating what is involved and how difficult it is.

Nav-Bar does not show the active page selected

data importer

(user uploads) Are we only doing CSV and/or json? or did Arthur mention something else?

Dynamic schema generation

https://stackoverflow.com/questions/23036751/can-sqlalchemy-work-well-with-multiple-attached-sqlite-database-files

this might kill 2 birds with one stone, sqlite is not designed for concurrency at all

fix broken itsdangerous library

Burger Menu does not work

create style library

things like default buttons, page layouts, what does import page look like, dropdowns, etc etc

rename tweet* to text* in models

this is part of the larger body of work of getting the original functionality back into the site, especially changing from the old way of loading data to the new way

move datasourcemanager to postgres

I think the best option here, because we now have postgres, is to use a defined structure for required columns that we can pull in
id, data_source_id, text and category (maybe(?)), then user_data

the data source manager can then have an id, name, description, project(?), and structure which describes the fields within the jsonb user_data column.

Postgres supports JSONB format, which can be used for querying data within json columns, we can validate app-side before it goes to the database, but we have the structure for that data source defined.

See: https://medium.com/geekculture/postgres-jsonb-usage-and-performance-analysis-cdbd1242a018

also: https://docs.sqlalchemy.org/en/20/dialects/postgresql.html#sqlalchemy.dialects.postgresql.JSONB

https://github.com/NLP4ALL/nlp4all/security/code-scanning/14

https://github.com/NLP4ALL/nlp4all/security/code-scanning/12

can add integrity check