j-hagedorn / trilogy Goto Github PK
View Code? Open in Web Editor NEWReference datasets for folktale motifs, tale types, and annotated texts
License: Other
Reference datasets for folktale motifs, tale types, and annotated texts
License: Other
Here is an example of the differences between the 'text' part of the dataset and the version on the Ashliman folktext website, the Ant and the Grasshopper:
an ant formerly a man - aesop (l’estrange, 1692) just has the moral bit at the end, not the actual story
the ant and the grasshopper - aesop (croxall, 1775) misses out the first para, same with the bewick 1818 version (same title), the James 1848, Jacobs 1894, and Bierce.
On the other hand, Jones (1912), and the other Bierce (the grasshopper and the ant) are complete.
Scrape and add tales from SuLaLune website to the tales dataset.
Clean to consistently reference nations or people groups in a consistent format
@salmonix identified the following issue in the consistency of variant depictions in the original documentation. For instance, in tale 1692 from atu_df.csv
: "A fool joins a band of robbers. They send him into a house to steal while the rest of them wait outside. He bungles the job in one of several ways [J2136]: He takes the robbers instructions literally. They tell him to bring something substantial (i.e. valuable), and he brings something heavy (e.g. a mortar) [J2461.1.7]. They tell him to bring something shiny (i.e. gold), and he brings a mirror [J2461.1.7.1]. The fool awakens the household. He wants to take more than he can carry, so he wakes the owner and asks him for help [J2136.5.6]. The fool finds a musical instrument and plays it loudly [J2136.5.7]. He decides to cook something to eat. Hearing the owner sigh in his sleep, the fool thinks he must be hungry, so he puts hot food in his mouth (hand) [J2136.5.5]."
Because the ATU original documents these in separate brackets, this is in our data as: "1692", 1, “J2136", "J2461.1.7", "J2461.1.7.1", "J2136.5.6","J2136.5.7","J2136.5.5". The problem with that this tale should look like: "1692",1,"J2461.1.7",["J2136.5.6","J2136.5.7","J2136.5.5"]. In order to clean this appropriately, we would need to identify textual markers that would let us to combine bracketed motifs and represent them as variants, rather than a prolonged sequence.
See p. 11, section 2.2.2 “In addition, the pages for the following tale types were unable to be scraped, due to errors generated in the R session: The Fisherman and His Wife, A Fool Does Not Count the Animal He Is Riding, Singing Thieves, Stages of Life, Straw, Bean, and Coal, The Wandering Jew”.
Includes:
Allow for joining with ATU ID
@sdaranyi asked that we: "List all terminals and roots. Group both lists into topics based on TMI labels. As L161 and L162 are frequently terminals, list all types for individual inspection where the motif string continues after them (or after any other terminal)". Planning to resolve this by doing two things:
atu_seq
dataset, creating a new column identifying the first and last motif within each tale_variant
. This will allow for easy filtering of these by people using the .csv file.atu_graph.graphml
file, adding two T/F fields to the node data: is_root
and is_leaf
@sdaranyi and @salmonix, please note that this approach will identify the first and last items present in the data. It will be an entirely different effort to identify what "should be" a terminal node, if the ATU does not identify these in the correct sequence.
Q: Under 'Boccaccio', 'Cat and Mouse', we find multiple tales with separate type numbers. For all other pages, we assign the same type to all tales in the page. This would require separate logic to assign separate types to each tale in a page. How valuable is this to our effort?
A (from @sdaranyi ): _From the Boccaccio label, I would leave out the Decameron Web. It is too concise to have a single type number. The rest have type numbers, so for "The boy who has never seen a woman", the subsumed three stories could be perhaps tagged as 1678_A, 1678_B and 1678_C. Does this violate your coding ideas? (For some reason, below the Boccaccio heading, "The boy..." is a standalone title as well, with the indication "Tales of types 1678 and 1459". I have no idea how 1459 entered the game. ) Likewise, "The enchanted pear tree" with its 8 variants could be labelled 1423_A to H. "The Three-Ring Parable", type 972, returns a 404 error at my end, and could be left out. The Griselda item sounds flat as a punctured tire yet has a number, so it should stay.
The "Cat and Mouse" category looks interesting, because in it, keywords (ie cat and mouse) clash with tale typology. I'd suggest to keep the existing type labels. If we run into evidence that keyword-level semantics is strong enough to blur type-level (topical) semantics, we can return to the problem._
Options:
Pasting your e-mail to track its resolution, @sdaranyi
"Hi Josh, I started to work with the AFT to exemplify on an existing corpus how things would look if people started using an existing Python workflow in Orange...:
This might either be a longer tale which can be segmented into different types, or a tale which partially fits 2 types (i.e. hybrid)
@sdaranyi noted that "Looking at today's pulled latest TMI, there's something wrong with the structure. Column 1 has mostly motif numbers only with labels in Column 3, but in a number of rows both the motif number and the label are present. Worse, based on this table you said there are 46222 individual motifs, whereas it has 54906 lines. Which one is the correct number? Chances are that this could change the proportion of motifs used in the ATU."
@sdaranyi , can you please provide examples where both the motif number and the label are present in the id
field? That would help me to find the issue.
@sdaranyi , my tmi
dataset still shows 46,222 unique values in the id
field, and 46,230 rows in the dataset, with the following items being duplicated (and needing to be resolved):
@sdaranyi and @salmonix , please identify any additional data cleaning issues you may find related to the tmi
dataset in the comments below, and I will try to resolve those quickly, if they are minor, rather than opening new issues to track them.
To do:
Add the English tales and their ATU tags from the Multilingual Folk Tale Database.
Separate and include tales with duplicate titles in the same tale type. These include:
Ant and Grasshopper, Bearskin, Beauty and the Beast, The Bremen Town Musicians, The Runaway Pancake, The End of the World, Devils Bridge Tales, Dividing Souls in the Graveyard, Fairy Cup Legends, Foolish Wishes, The Fox and the Cat, The Grateful Animals and the Ungrateful Man, Hanging Game, The Himphamp, Jack and the Beanstalk, The Language of Animals, The Lion in the Water, Llewellyn and His Dog Gellert, Town Mouse and Country Mouse
Remove any AFT pages with type numbers that are not found in the ATU.
Per reviewers:
To do:
Per request from @sdaranyi : "2-way clustering (biclustering, block clustering, co-clustering) on the AFT for first results. I know it exists in R because I used it a few years ago but it would be killing to find the relevant knowhow in my files right now. Its output is a heatmap, or it should be combined with one, the point being that we could immediately smuggle in the ATU thereby, without much ado. It would be nice to create a visual to reproduce the ATU contents, separating animal tales, magical tales etc into separate boxes so that eg these categories label the rows and the respective ATU numbers flagging texts label the columns (or vice versa, with more texts than categories). I'm not sure if the figure will make much sense for us, but for others not being familiar with this technique, it will. Plus herefrom we could move ahead toward text analysis vs motif string based block clustering in pursuit of motifs etc. So to say, an exemplification of the possibilities for future users with the willingness but no idea of what next."
@sdaranyi , you mentioned that you'd like to: "In a separate list, start collecting motifs that correspond to functions. Eg L161 and L162 would be F31 Wedding, maybe part of it F30 Punishment, whereas the root could be screened for F0 Initial situation, F1 Absentation (leaving from home)."
In order to do this, we would want to be able to join our motif table to a table of Proppian functions. Would we want any information about the functions in this table other than what's noted here (i.e. a table with propp_id
, definition
, function
)?
Clean patterns from atu_seq$motif
found using:
un <-
atu_seq %>%
select(motif) %>%
distinct() %>%
anti_join(tmi, by = c("motif" = "id"))
Clean patterns from aft$atu_id
found using:
un <-
aft %>%
select(atu_id) %>%
distinct() %>%
anti_join(atu_seq %>% select(atu_id) %>% distinct(), by = "atu_id")
The AT Uther updated had useful extra information ignored or condensed by the ATU, prominently on the motif strings of type variants. To integrate both sources could help us disambiguate too long motif chains, lasting often beyond the usual terminal. See eg suspicious cases where L161 is non-terminal, although quite often it is.
Need to address these issues, here (before expanding):
tmi
dataset dataset for all values in section ending "ff.", then resolve this with a between()
join to that portion of the tmi
dataset.between()
join to the tmi
dataset.A running list of tiny tweaks, filters and cosmetic changes:
Currently 802A* and 934D are duplicates
Dataframe contains multiple notes on possible combinations that require regex and unnesting
To do:
These include:
These include:
Tales missing with a manually assigned id: Ali Baba, Animal Brides and Animal Bridegrooms, Big Peter and Little Peter (norway133) , Friday, Frog Kings, The Hand of Glory, Hog Bridegrooms, The Husband Who Was to Mind the House, Mastermaid (norway120), Midwife for the Elves
Tales which appear to have a type in the URL but are missing: The Blood Brothers (type0303), The Three-Ring Parable, The Fox and the Crow (type0057)
Notes from review by @sdaranyi (thank you!):
Add pages with identified tales types to the list for scraping
This is based on an issue identified by @salmonix. In the example identified, the sequences of the tale variants are as follows for tale 1341A:
The text runs as following: "...The thieves kill him, too [J581, J2136]. (3) Two foolish slaves are recaptured because of their talkativeness [J581, J2136]..." The motifs identified are: J581 (Wisdom and Folly, Foolishness Of Noise-Making When Enemies Overhear) and J2136.1 (Wisdom and Folly).
@salmonix suggests that, in our cleared data perhaps we should only retain variants 2 and 3 above, and remove 1 and 4, where the same motif is repeated in a row.
Adding this as an issue for discussion: @sdaranyi and @salmonix, are we certain that we want to remove all tale variants where a motif is repeated 2 or more times in a row?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.