View Code? Open in Web Editor
NEW
Topic modeling idiomatic contexts.
Python 99.27%
Shell 0.73%
idiom2collocations's Introduction
idiom2collocations's People
Contributors
Watchers
idiom2collocations's Issues
Why?
As for tfidf, you only save the result. you'd better save the model so that you can use it for measuring quality of an example sentence.
How?
designate a new path for the model
save the model
save the model, as well as the topics - but this time just save the top 10 topics. It's just for viewing the model.
why?
if I do this locally:
takes 10 mins for 4 splits
I have 2900 splits -> 10 * 725 mins = 7250mins = 5 days.
well, you certainly should not wait for that amount of time.
goal?
resort to cloud computing. Submit jobs at google cloud.
how?
submit pyspark jobs to google cloud.
μ½μ§..
κΈ°μ‘΄μ elasticsearch μμ¬
brew install elastic/tab/elasaticsearch-full
curl http://localhost:9200 λ‘ build_snapshot false νμΈ
/usr/local/Cellar/elasticsearch/[version]/bin/elasticsearch-plugin install mapper-annotated-text
κ·Έλ¬λ©΄ μ€μΉκ° λ κ².
μ΄ μ΄νμ ν€λ°λλ ..fullλ‘ μ€μΉνμ.
Why?
We need a way to do:
How?
Goal: sentence -> quality
So? how does the function measure the quality?
TFIDF
Start with a tfidf model. What do I need?
you must first have the tfiddf lookups for each idiom.
it will be better to save them as ndjson, then. That will suit me better than tsv.
LDA