xachab / qumin Goto Github PK
View Code? Open in Web Editor NEWQuantitative modelling of inflection
Home Page: https://qumin.readthedocs.io/
License: GNU General Public License v3.0
Quantitative modelling of inflection
Home Page: https://qumin.readthedocs.io/
License: GNU General Public License v3.0
Hi Sacha ! I thought it could be useful to write here down some practical issues I found, just to remember they have to be fixed. This doesn't mean that I expect you to do it, I can also handle it! It's just for tracking.
When working on NTFS file systems, even on Linux-like OS, file names are limited to 255 chars.
However, Qumin stores files with really large names. In such cases, it could raise an OSError :
OSError: [Errno 36] File name too long: '../Results/JointPred/20240115/vlexique_short-20171031_paradigms.csv_v1.1.0-63-g6e7431d_20240115_12h03__patternsPhonsim.csv_v1.1.0-63-g6e7431d_20240115_12h14_onePredEntropies_beta0.5.csv'
There should be a fallback in case of such error, in order to save the computation results + a WARNING to the user.
Got following error on branch RemoveAliases, just by running qumin.lattice
on very basic test patterns.
Looks like collections (in kwargs) are passed to table_to_context
, but I don't know this error is normal behaviour, or if collections should theoretically be an argument for table_to_context
. Note also the very weird package imports at the top of the file qumin/lattice/lattice.py
I am not familiar with this part of the code, so I just put it there :
Traceback (most recent call last):
File "PATH/Qumin/bin/qumin.lattice", line 33, in <module>
sys.exit(load_entry_point('qumin', 'console_scripts', 'qumin.lattice')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "PATH/Qumin/qumin/make_lattice.py", line 164, in lattice_command
main(args)
File "PATH/Qumin/qumin/make_lattice.py", line 70, in main
lattice = ICLattice(pat_table.loc[list(microclasses), :], microclasses,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "PATH/Qumin/qumin/lattice/lattice.py", line 156, in __init__
self.context = table_to_context(dataframe, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: table_to_context() got an unexpected keyword argument 'collections'
I keep changing the insertion cost in the phonologically aware alignment function because it is hard to find a correct value.
Intuitively, it needs to be lower than half a bad substitution, but higher than half a good substitution. Indeed, if [t] and [d] are considered close, we might prefer a single substitution:
alignment: xyi
xyu
operations: SSS
To two insertions:
alignment: xyi-
xy-u
operations: SSII
In the case of two segments which are very distinct, for ex [t] and [a], we should prefer:
alignment: xyt-
xy-a
operations: SSII
to:
alignment: xyt
xya
operations: SSS
A lower cost favors many insertions and results in many concurrent alignments. A high cost forces alignments of dissimilar segments, and may result in not finding the right alignment.
The current heuristic is to multiply the mean cost in the entire segment inventory by a constant proportion, currently set to 1/3, but which has been set to 2 in the past:
Qumin/bin/representations/patterns.py
Line 24 in 52913e6
Then we do:
Qumin/bin/representations/patterns.py
Line 904 in 52913e6
While looking at the actual matrices of segment similarity, I noticed that there is often a vaguely bimodal distribution, with many similar pairs, and many dissimilar pairs. As a result, I believe the median is better than the mean.
We need the result to be lower than half the mean: a good default for the INS_PROP
might be to go back to 1/3 for the proportion. Since it is a constant, the program should accept it as a keyword argument in case the user wants to provide a custom number.
Some alternations differ only by the number of blanks, e.g. in Portuguese verbs:
[+son +stress -C]__ ⇌ [+son -C -stress]_ˈɐm_ʃ / X*_C+_u_
[+son +stress -C]_ ⇌ [+son -C -stress]ˈɐm_ʃ / X*[-lat]_u_
These could be expressed as:
[+son +stress -C]__ ⇌ [+son -C -stress]_ˈɐm_ʃ / X*_C*_u_
Currently, we generalize the entire alternation when this allows to merge more than one pattern. However, there may be several sites where generalization is possible, and only some which actually require it. For example:
Verbo | PresIndic:2 | PretPerfIndic:1 |
---|---|---|
criar | kɾˈiɐʃ | kɾiˈɐj |
continuar | kõtinˈuɐʃ | kõtinuˈɐj |
apoiar | ɐpˈojɐʃ | ɐpojˈɐj |
Are captured at first by three alternations:
ˈiɐʃ ⇌ iˈɐj
ˈuɐ ⇌ uˈɐj
ˈojɐʃ ⇌ ojˈɐj
We find two generalizations possible in the alternations:
We notice that generalizing can allow us to merge these three patterns, so we end up with:
[+son +stress -C][a-o-u-ɐ̃-õ-ɐ-ɔ]ʃ ⇌ [+son -C -stress][+J +dipht +son +stress -C] / X*C_
But only the first of the two generalizations were necessary, since all of these words do exactly /ɐ/ to /ˈɐj/. We want to generalize only what is necessary to obtain a single pattern:
[+son +stress -C]ɐʃ ⇌ [+son -C -stress]ˈɐj / X*C_
I don't want things like:
[+son +stress -C][a-o-u-ɐ̃-õ-ɐ-ɔ]ʃ ⇌ [+son -C -stress][+J +dipht +son +stress -C] / X*C_
It's either a list of phonemes as in [a-o-u-ɐ̃-õ-ɐ-ɔ]
or a list of features as in [+J +dipht +son +stress -C]
. This might be a good occasion to rewrite the way I do regular phonological changes in the alternation.
When the alternation has a regular phonological alternation, but the segments are not a natural class, the separator "-" is missing in the exported patterns.
Signalled by Olivier:
"je tombe sur une bizarrerie de la version actuelle du qumin: la convention pour l'usage des crochets n'est pas la même dans la partie contexte et dans la partie changement.
Exemple ci-dessous Sans doute est-ce un oubli? Je te le signale parce que suivant le reste de ton code ça peut créer des problèmes non triviaux..."
Example:
[iuy] ⇌ [jwɥ]ɔ̃ / [E-O-a-b-d-f-g-i-j-k-l-m-n-p-s-t-u-v-w-y-z-Ø-ɑ̃-ɔ̃-ŋ-œ̃-ɥ-ɲ-ʁ-ʃ-ʒ-ɛ̃]*[b-d-f-g-k-l-m-n-p-s-t-v-z-ŋ-ɲ-ʁ-ʃ-ʒ]_ <281>
The behaviour is that of Qumin V.1: always re-segment. However, it might be better to make it possible to respect the given segmentation. Unfortunately, Qumin has quirks regarding phonology, and the segmentation needed tends to be different from that which I use in later datasets (where I write tiered information, such as length, tones, stress, on the syllable's vowel).
Note that all paralex datasets now MUST have space separation:
"The value of the phon_form MUST be a sequence of space-separated segments, e.g. not "dominoːrum" but "d o m i n oː r u m"."
I think we need the following cases:
@JPapir : what do you think ? Is that reasonable ?
The line doing the splitting is here:
When installing Qumin with pypi, I get the following warning:
DEPRECATION: qumin is being installed using the legacy 'setup.py install' method,
because it does not have a 'pyproject.toml' and the 'wheel' package is not installed.
pip 23.1 will enforce this behaviour change. A possible replacement is to enable
the '--use-pep517' option.
Discussion can be found at https://github.com/pypa/pip/issues/8559
If I understand it well, we should use a pyproject.toml
file instead of setup.py
? Don't know it this is project related or only wrong setups on my side. Do not hesitate to close if irrelevant.
When computing absolute values, we should consider not only the number of pairs, but also the token frequencies of the paradigm cells.
Depends on #31
No reason to keep this custom class.
Currently, the doc lives on a server at the LLF, and the last generated version is old. I need to update the docs so they are current, re-generate, and setup somewhere to host the generated files. Maybe readthedocs ?
Remove svn use in:
Lines 47 to 68 in ba99305
SplitPatternDistribution does not accept a feature table yet.
Investigating why #30 was failing when pats.defective=False
, it turns out that v2.0 introduced a regression : whatever the value of pats.defective
, we get the same result including defective lexemes. Here is a sample patterns file with defective entries and pats.defective=False
. Lexemes p and q should be dropped.
lexeme,"('first', 'second')","('first', 'third')","('second', 'third')"
k,s_ ⇌ _p / p_s_ <2>,s_ ⇌ _k / p_s_ <1>,p ⇌ k / ps_ <1>
p,s_ ⇌ _p / p_s_ <2>,,
q,,,k ⇌ t / ps_ <1>
s,s ⇌ / ssp_ <1>,ss_ ⇌ _s / _p_s <1>,_p ⇌ p_ / _ss_ <1>
I never use pats.defective=False, so this explains why I didn't spot it before.
I suspect these 2 lines used to delete lexemes, whereas they only delete forms now that we use long format tables.
Do you confirm?
(to fix for #22)
When installing with PyPi, the following files are missing from the package:
If no mpld3 is not installed, this is fine. But if mpld3 is available, it raises the following error when qumin.lattice --html
:
JAVASCRIPT = _load_external_text("HighlightSubTrees.js")
File "/venv/lib/python3.9/site-packages/qumin/lattice/lattice.py", line 34, in _load_external_text
open(join(dirname(__file__), filename), "r", encoding="utf-8").readlines())
FileNotFoundError: [Errno 2] No such file or directory: '/venv/lib/python3.9/site-packages/qumin/lattice/HighlightSubTrees.js'
The Frequencies class (former Weights
from branch feature-overabundance), should have methods to:
Any other suggestions for this class can be added here.
There is still a lot of code which is not used at all, and was written "just in case". It should be removed to facilitate maintenance, since there should be very little new features implemented.
The calls to the Segment class in the alignments to obtain scores are costly. I could entirely avoid this by using a score matrix.
Repair the eval script.
Currently I segment forms, then replace each sequence of characters by 1-char aliases which look vaguely like the original sequence. This is a lot of complicated code for something simple. It would be better (and less confusing) to code strings on integers for internal calculations. All external outputs will be unchanged.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.