eubinecto / idiom2collocations

Topic modeling idiomatic contexts.

I aim to kill 🐤🐤 with 🪨

🪨	🗓	🐤 do more with fewer weights?	🐤 solves a problem that matters at scale?
The Wisdomify Project	2021-now	Draws inductive biases from how we interpret Korean proverbs	A solution to a Tip-of-the-Tongue problem
The Idiomify Project	2022-now	Draws inductive biases from the science of idiom acquisition (The Idiombeddings Project)	A solution to the lack of idiomatic fluency of English-as-a-Foreign-Language writers
The Politely Project	2022-now	N/A, as it is currently rule-based.	A solution to ...

idiom2collocations's Issues

tfidf - just save the model

Why?

As for tfidf, you only save the result. you'd better save the model so that you can use it for measuring quality of an example sentence.

How?

• designate a new path for the model
• save the model
• save the model, as well as the topics - but this time just save the top 10 topics. It's just for viewing the model.

submitting pyspark jobs to google cloud

why?

if I do this locally:

• takes 10 mins for 4 splits
• I have 2900 splits -> 10 * 725 mins = 7250mins = 5 days.

well, you certainly should not wait for that amount of time.

goal?

resort to cloud computing. Submit jobs at google cloud.

how?

submit pyspark jobs to google cloud.

• https://cloud.google.com/sdk/gcloud/reference/dataproc/jobs/submit/pyspark

annotated_text field

삽질..

1. 기존의 elasticsearch 삭재
2. brew install elastic/tab/elasaticsearch-full
3. curl http://localhost:9200 로 build_snapshot false 확인
4. /usr/local/Cellar/elasticsearch/[version]/bin/elasticsearch-plugin install mapper-annotated-text

그러면 설치가 될 것.

이 이후에 키바나도 ..full로 설치하자.

sentence quality

Why?

We need a way to do:

• sentence -> quality

How?

Goal: sentence -> quality

So? how does the function measure the quality?

TFIDF

Start with a tfidf model. What do I need?
you must first have the tfiddf lookups for each idiom.
it will be better to save them as ndjson, then. That will suit me better than tsv.

LDA