Comments (10)
There is also IPFS.
from argos-translate.
No, the training data is very large. I don't even store it myself I just re-download from Opus every time I train a new model. If you're interested in the data go to Opus. There's also a generate_wiktionary_data.py
script in the training scripts.
from argos-translate.
I'm not too worried; the discussion on S.O. is pre-Microsoft acquisition. If you want to link it, go ahead.
from argos-translate.
It might be useful to consider using GitHub Releases to distribute the models, as there are no bandwidth limitations and since CloudFront is used to deliver the assets.
from argos-translate.
Related to this once better distribution is in place we'll want to add the ability to manage model downloads from the GUI.
from argos-translate.
GitHub has (so far, crossing fingers) been pretty good at hosting binary data (up to ~2GB in size per file) without limits, although the party might end sometimes in the future.
Otherwise a central server with CDN works pretty well too. For cheap, there's https://www.digitalocean.com/products/spaces/ (although careful on the outbound transfer fees!) or https://wasabi.com/ (no egress fees)
from argos-translate.
A CDN is probably the best long term solution (in addition to providing torrents) and Google Drive is definitely not ideal. Last weekend when there was a lot of traffic from Hacker News it seems that some people weren't able to download models. I'm tempted to link directly to the githubusercontent url in the LibreTranslate Models repo but that may increase the chance that we can't continue to use GitHub.
https://stackoverflow.com/questions/38768454/repository-size-limits-for-github-com
from argos-translate.
@davidak https://www.reddit.com/r/ipfs/comments/l43u9u/seeking_open_source_contribution_to_distribute/
from argos-translate.
I wonder if the raw data is also stored anywhere.
Or do we only have the compiled model files?
from argos-translate.
We're now sharing one model pair (English-Chinese) via IPFS (available on package index) as a test. A volunteer is currently pinning for us and we can expand if people find this useful.
from argos-translate.
Related Issues (20)
- ArgosTransate for python doesn't translate anything from English to French HOT 1
- Argos_translate no longer works offline? HOT 1
- multilingual-rag using argos-translate HOT 1
- The difficulty of generating a proper LLM for translation from web scraping...
- Feature Request: Allow installing without `nvidia-cuda` packages. HOT 3
- Pipe mode, line-by-line (stdin/stdout)
- Support for tamil language
- no pip install
- Does not support Python 3.12 HOT 1
- How to switch usage to GPU instead of CPU? HOT 1
- BUG: pip install argostranslate does not work on Python 3.12 HOT 3
- Problem in Greek Translation
- How to use custom model?
- Stanza version >=1.1.1 breaks a few langauges HOT 2
- Support for Brazilian Portuguese
- [BUG] SIGSEGV
- Title
- Upgrade SentencePiece for Windows Support on Python 3.12+ HOT 2
- Translation of pre-split and pre-tokenized sentences
- Translation from Spanish -> English missing words
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from argos-translate.