Comments (2)
@Zenthon @vukosim
Ishe has finished the Sentence Alignment on the gov-za multilingual so he'll jump ahead of me and work on Masakhane.
However, I do have an idea for masakhane on this bug so I just want to document it so Ishe might try it out.
We have the suspicion that the program runs out of memory when trying to add multiple models so perhaps we can try only dealing with one model at a time.
You can use python manage.py add_language
to add language refs to the database and download them. Perhaps the /update
endpoint should just be for downloading the models that have references in the database.
Note: when you python manage.py remove_language
you should delete the directory storing the model for data coherence.
Once the models are downloaded, the client should list what models are available for translation.
When the user picks a model - say eng-swa
only that model should be loaded into memory.
If the user picks another model - say eng-tiv
, the old model should be overwritten and the new one placed into memory.
My only concern with this approach is that it takes a long time to load a model and that might affect UX.
from masakhane-web.
@Kabongosalomon It seems that some of the language models config files are incorrect(their checkpoint files parameters are wrong or missing) hence why you can't add some language models such as the en-ln- model
I have tested however that you can add multiple languages at once.
Example:
- en-sw-JW300 (Swahili)
- en-tiv- (Tiv)
- en-iso- (Isoko)
All 3 of those models were loaded at the same time without issues
from masakhane-web.
Related Issues (20)
- Adding more languages to the online system HOT 3
- Adding info on JoeyNMT Involvement
- BPE markup appearing in output in edge case HOT 2
- Fix favicon
- Integrate Masakhane AfroTranslate package
- Server stopping bug HOT 2
- move from http to https
- Start the server with default languages. HOT 5
- Check the pre and post processing steps
- Get to run the system HOT 1
- update the README.md running with Docker part HOT 1
- Running system on standalone app/ Backend - Polyglot error
- Update Repo README.md to get all to work, split setup in docs folder
- Prepare 2nd payload for Zenodo HOT 7
- Deploying recent issues to production
- Documentation on the Backend system
- Combine Add_Language & Update
- Dependency Maintenance Documentation
- Documentation for the Client
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from masakhane-web.