dsfsi / masakhane-web Goto Github PK

Description

I'm seeing this error when adding an additional model

It might be related to the time it takes to download the model.

The service starts with no language in memory, and we need to run curl --request GET 'http://127.0.0.1:5000/update to load the language pairs available in the database python manage.py all_languages in memory.

Edit the class AddResource to have default mode loaded when the command docker-compose up -d

Investigate, why the production backend service stops from time to time.

Description

Combine the steps of adding and updating loaded language models.

Tasks

• Combine add & update

Background

Masakhane Translate has a Manage CLI that let's you add language model info from the database, among other things. Language model info refers to the data stored in the database (Fig 1), the actual language models are downloaded in another step. The database seems to serve as a reference as to which database models are downloaded. The add_lang command (Fig 2) reads languages.json - which is just a file storing the language names & shorthands - and gets the name_tag composition to be stored in the database using the user-defined Language DB model. Take note of the database implementation using FlaskSQLAlchemy.

Once model info has been loaded into the database, the /update endpoint - which is linked to the AddResource Class (Fig 3) - must be hit which will go through the name_tags in the database and download the model files from Zenodo if necessary. Once the model has been downloaded it is loaded into memory with self.models which is a dictonary. It will also remove any models in memory that no longer have a reference in the DB.

Files

translate.py
model_load.py
init.py
manage.py

Screenshots

Figure 1 - Language table contents

Figure 2 - add_language command
Model info added to the database

Figure 3 - AddResource Class
Model info used to download models

Secure translate.masakhane.io/

The instructions on running the app through docker are not correct. To run the container we should use docker-compose up-d --build rather than docker-compose -f docker-compose.prod.yml up -d --build which is a command to run on production and it doesn't work on dev.

Description

Create documentation on guidelines to go about when updating dependencies.
It will probably involve going to the branch of the dependency upgrade and figuring if, and why, it might break and updating the code and testing before merging it into master.

Background

There is a dependancy bot in the repo that checks for out of date dependencies, once found it creates a new branch, updates the dependencies and creates a PR. Dependencies have a high chance of causing packages in the project to be invalid with one another and cause errors.

Tasks

Include specific tasks in the order they need to be done in. Include links to specific lines of code where the task should happen at.

• Figure out the method used for updating dependencies.
• Create documentation for contributors of the repo to know how to maintain dependencies.

Description

After learning about the backend of Masakhane Translate, I'd like to document explanations on the details of the system so future developers can get started easier.

Tasks

• Create a doc for explaining project structure and implementation
• Update nested Readme's
• Add more code comments to important files

It looks like the postprocessing doesn't remove BPE in the invalid (but still possible) case that there's no token after one ending in @@.

I ran into this with this English -> Swahili pair: "I don't speak Swahili well." -> "Mimi huzungumza Kiswahili kwa ukunju@@". Screenshot below:

Description

The issues #57 & #69 have been resolved in the PR #70
We need to merge the changes into production & ensure everything works as expected.

Tasks

• Review the production branch
• Merge master into prod after approval of #70
• Run tests and check all is working fine
• ask @Kabongosalomon to deploy on Masakhane.io

Integrate AfroTranslate package into the code base

It seems like the pre/post processing steps need to be revisited to make sure that the translation is case insensitive.

** Update the Readme - Running the system as a standalone on local machine gives - NO module named polyglot error when Installing required packages -pip install -r requirements.txt

See pull request by Julia

We need to have these configurations added

• IGBO -> English
• English -> IGBO
• English -> Setswana
• Hausa -> English
• English -> Hasua
• Yoruba -> English
• Swahili -> English

Description

For the models that are missing in Zenodo but are on Masakhane Web Translate, lets create the zip payloads to be uploaded to Zenodo. Once on Zenodo, the links in available models, should now point to Zenodo
Refactor

Files

A list of relevant files for this issue. This will help people navigate the project and offer some clues of where to start.

To Reproduce

If this issue is describing a bug, include some steps to reproduce the behavior.

Tasks

Include specific tasks in the order they need to be done in. Include links to specific lines of code where the task should happen at.

• Create the zip payloads to be uploaded to Zenodo.
• links in available models, should now point to Zenodo
• Refactor code to also Support Zenodo based downloads of models
• Commit, pull request

Afternoon @PeaceAz and @ntsakomtombeni

Be able to run the system on your local machine.

Run through Docker Compose

You will need to install docker on your local machine

Read the instructions in the README to get the system running locally.

I you have problems, please tag @Kabongosalomon

FOR SOMEONE WITH KNOWLEDGE OF REACT

Description

Create Docs explaining the structure of the frontend.

Background

The frontend/client is written in React and uses Webpack.

Files

/src/client

Tasks

Include specific tasks in the order they need to be done in. Include links to specific lines of code where the task should happen at.

• Document React project structure on Client README
• Documentation WebPack integration on Client README

Split the docker/standalone setup into it's own docs to shorten the README
README - Repo Description & Contributions
start_app.md - Instructions to run the app using docker/stand-alone
debug_errors.md - Doc to test the running app and solutions on potential errors

Display Masakhane favicon on the browsing tab