chirila / filelingr

This archive has scripts for working with linguistic fieldwork archival collections. It is a set of checks for files. It is meant mostly for fieldworkers who already have documentation collections and want to check what is in the folders, what's missing, or what's inconsistent in file naming conventions.

We are very happy to collaborate! And we'd like to know about features that would be useful.

To use the scripts, you need R and RStudio (both free). R can be downloaded from r-project.org (a list of mirrors is [here][https://cran.r-project.org/mirrors.html]). RStudio can be downloaded [here][https://www.rstudio.com/products/rstudio/]. Clone or download this repository, open the checkR.rmd file, and follow the instructions to install the required packages and tell the checkR script which directory you want to analyse.

filelingr's Issues

add sample files for testing

increase array of filetypes for audio

check for mp3, ogg, etc

Make shiny app

investigate shiny app for better ease of use

metadata file validation?

user specifies metadata file, script parses column and checks for completeness

more than two delimiters per line/delimiters in the wrong places

audio stats

table with

filename
filetype (.wav, mp3, etc)
sampling rate
length (mins, secs)
does .eaf file with same name exist?
does .textgrid with same name exist?
does .txt with same name exist?

check for files with suffixes like "-final" or "-rev"

add .meta files

checks for laMeta annotations

preserved elan offsets/check for settings files

check if Elan file is using offsets. If so, need settings files to preserve it

textgrids

TextGrid vs textgrid
other things needed?

check for English in transcript line

have user specify meta language and transcription tier, check for English/Spanish/etc in that tier

mfa and p2fa

two forced alignment algorithms produce two tiers. p2fa has phone is tier 1 and word as tier 2. For mfa, it's the opposite.

add .mp3 processing

open eaf file as text, search for = "" (or whatever the delimiter is for elan annotations)

return names of duplicate files

per Daisy Rosenblum's comment about transcripts from multiple sources, version control

list of characters used in eaf transcript lines

check for characters in files

parse list of characters (in particular field?), return non-ascii chars?
Mostly needed for legacy files that have been through multiple conversions, probably not needed for new data

automatically create subtitle files from eaf intervals

fork elan script

per reviewer, dalejbarr/elan hasn't been updated in 9 years

check for spaces vs _ vs - in filenames

Are there empty directories?

# Load the required package
library(fs)

# Specify the paths to the directories to check (don't do this, load it from the list of directories already loaded)
dir_paths <- c("path/to/dir1", "path/to/dir2", "path/to/dir3")

# Create a vector to store the empty directories
empty_dirs <- vector("character", length(dir_paths))

# Loop over the directory paths
for (i in seq_along(dir_paths)) {
  # Check if the directory is empty
  if (is_empty(dir_paths[i])) {
    # If the directory is empty, add it to the list of empty directories
    empty_dirs[i] <- dir_paths[i]
  }
}

# Remove any NULL values from the list of empty directories
empty_dirs <- empty_dirs[!is.null(empty_dirs)]

# Print the list of empty directories
print(empty_dirs)

maximum file name length

update elan package to install from forked version

https://github.com/chirila/dalejbarr-elan

add r history and ds_store to gitignore

is there a file called metadata

format
table called filenames

removing non-transcript items from transcription line

hard to check automatically

check for XXX in lines?
lg identification?

check for non-English in English gloss

have user specify language and tier name, check for "not that language" in that tier

print formatted text to file

write_yaml in yaml package

review filetypes

check other collections for other common filetypes (apart from elan etc)