laurieburchell / cs-lid-harder-than-you-think

Repository accompanying "Code-Switched Language Identification is Harder Than You Think" (EACL 2024))

Code-Switched Language Identification is Harder Than You Think

This repository accompanies the paper Code-Switched Language Identification is Harder Than You Think (EACL 2024). Any problems or suggestions, please raise an issue!

Before starting, create a conda envirnoment using requirements.txt. All scripts are intended to be run in the directory where they are located.

Downloading and preprocessing the data

download-data.sh downloads and reformats our training and test data. Note that you need to email the authors to access two of the datasets: see appendix A of the paper for details.

The OpenLID training data is very large (>20GB). Make sure you have enough space available.

cd data/scripts
bash download_data.sh

We provide the clean-data.py script to preprocess the data prior to classification.

Models

We use three models in this paper:

• OpenLID
• MultiLID
• Franc

Run download-models.sh to obtain the pretrained OpenLID and MultiLID models. Franc is accessed using pyfranc. Scripts to obtain predictions from each model are located in models/<model_name>/get_<model_name>_preds.py.

Evaluation

Run calculate_metrics.py to generate the metrics reported in the paper. It expects the gold file and the predicted labels file to have one set of tab-separated labels per line. If the --franc argument is given, it will convert the gold and predicted labels to common alpha3 labels for better comparison.

cs-lid-harder-than-you-think's Issues

data preprocess suggestion

Hi Laurie,

The code was super easy to use, thanks for that.

just some suggesstions.

I had troubles with this line of code: reformat-itu-tureng.py#L38

The error is: f-string expression part cannot include a backslash
I solve it this way (not very clean 😄):

with open(args.out_file, 'w') as f:
    backslash_char = "\t"
    for s, t in zip(sents, tags):
        f.write(f"{s}{backslash_char}{'🍔'.join(t)}\n".replace("🍔", backslash_char))

• For turkish dataset there is no need to refer to the form ...
  you can access it here (TR-EN CS Corpus - with language tags.txt):

https://github.com/zeynepyirmibes/code-switching-tr-en

• For indonasian also there is need to refer to email
  you can access it here (825_Indonesian_English_CodeMixed.csv)

https://github.com/ir-nlp-csui/id-en-code-mixed