okgraph is a python3 library that performs unsupervised natural-language understanding (NLU).
It currently focuses on the following tasks:
- set expansion given one or a short set of words, continues this set with a list of other 'same-type' words (co-hyponyms);
- relation expansion given one or a short set of word tuples, continues this set with a list of tuples having the same implicit relation of the given tuples;
- set labeling given one or a short set of words, returns a list of short strings (labels) describing the given set (its type or hyperonym);
- relation labeling given one or a short set of word tuples, returns a list of short strings (labels) describing the implicit relation of the tuples in the given set.
Being unsupervised, it only takes a free (untagged) text corpus as input, in any space-separated language. Scriptio-continua corpora and languages needs third-party tokenization techniques (e.g. micter).
Please ensure a python version between 3.7 and 3.9 is being used. If you're using Windows, ensure the Windows 10 SDK have been installed by the Visual Studio Installer.
After cloning the repository (https://github.com/atzori/okgraph.git && cd okgraph
), run the followings commands from
the root directory of the downloaded project to install okgraph for development:
> python -m venv venv # be sure you are referring to an acceptable python version using
> source venv/bin/activate
(venv) > python -m pip install --upgrade pip setuptools devtools
(venv) > pip install -r requirements.txt # this may take several minutes
If you want to use it as a library inside one of your projects, just install it from your environment with:
(venv) > pip install ../path/to/downloaded/okgraph
To run the tests some text corpora are required. The following script will provide the required text corpora, along with their word-embeddings, corpus indexes and corpus dictionaries:
$ python tests/get_test_corpus_and_resources.py
This procedure may take a while: it will download the wiki-english-20171001 corpus from the
Gensim-data and use it to generate three corpora: text7.txt
,
text8.txt
and text9.txt
(obtained respectively from the first 107, 108 and 109
bytes of the wiki-english-20171001 corpus). Then, all the related resources (embeddings, index and dictionary) are
created. The brandly new corpora and their related resources can be found in tests/data
from the project directory.
This library uses Sphinx to automatically integrate the in-code comments within the library documentation. To obtain the code documentation run the following:
$ python docs/make_docs.py
The README, modules and packages will be automatically parsed to obtain an html documentation that can be found in
docs/build/html/index.html
from the project directory.
The first step is to create an OKgraph
instance by loading a text corpus. Any OKgraph
instance will also require a
word-embedding model, a corpus index and a corpus dictionary, but specifying any value for them is optional.
The word-embedding model can be obtained processing the specified corpus, or can be a pre-existent model.
The word-embedding model is available through one of the extension of the okgraph.embeddings.WordEmbeddings
abstract
class which introduces the word-embeddings and the operations that can be done with them. The
okgraph.embeddings.MagnitudeWordEmbeddings
class extends the okgraph.embeddings.WordEmbeddings
class and provides
the currently one and only implementation available for the word-embeddings, using the
Magnitude library.
The corpus index and corpus dictionary are strictly related to the corpus itself and are always obtained from its processing.
This example creates an OKgraph
instance based on the text8.txt
corpus:
from okgraph.core import OKgraph
okg = OKgraph("text8.txt")
The file text8.txt
will be set as corpus file and the word-embeddings, corpus index and corpus dictionary will
be searched using their default values starting from the same directory of the corpus (text8.magnitude
, indexdir/
and dictTotal.npy
). If found, they will be used as they are, otherwise they will be automatically generated processing
the corpus.
This example creates an OKgraph
instance based on the text8.txt
corpus with a specified word-embedding model:
from okgraph.core import OKgraph
okg = OKgraph("text8.txt", "model_file")
or equivalently:
import okgraph
okg = okgraph.OKgraph(corpus="text8.txt", embeddings="model_file")
The file text8.txt
will be set as corpus file and the word-embeddings is searched, starting from the same
directory of the corpus, in a file named model_file.magnitude
(when the extension is not provided, .magnitude
is
automatically appended). If found, the word-embeddings will be loaded, otherwise it will be generated processing the
corpus and stored as model_file.magnitude
.
If instead of a file name, the word-embeddings argument is a URL (starts with 'http://' or 'https://'), a remote version of the file will be used (to be fixed).
from okgraph.core import OKgraph
okg = OKgraph(corpus="text8.txt", embeddings="https://model_file")
The stream argument allows to stream the model, instead of downloading it:
from okgraph.core import OKgraph
okg = OKgraph(corpus="text8.txt", embeddings="https://model_file", stream=True)
This example creates an OKgraph
instance based on the text8.txt
corpus with a specified value for the
word-embeddings, the corpus index and corpus dictionary:
from okgraph.core import OKgraph
okg = OKgraph(corpus="text8.txt", embeddings="model_file",
index_dir="corpus_index/", dictionary_file="corpus_dictionary.npy")
The file text8.txt
will be set as corpus file and the word-embeddings, corpus index and corpus dictionary will
be searched using their specified values starting from the same directory of the corpus. If found, they will be used
as they are, otherwise they will be automatically generated processing the corpus and stored with the specified names.
When an OKgraph
instance is created loading a corpus, the word-embeddings, corpus index and corpus dictionary
are searched starting from the corpus directory to be loaded as they are, if they exist. To avoid this resources to be
loaded as they are and force their re-generation from the corpus processing, the force_init argument can be set
to True
:
from okgraph.core import OKgraph
okg = OKgraph(corpus="text8.txt", force_init=True)
This code will force the OKgraph
constructor to generate again the resources and overwrite them, if they exist.
The classes and methods in okgraph.preprocessing.*
are useful to parse and prepare a text corpus before creating
the OKgraph
instance. For instance, these methods helps, e.g., to convert xml (MediaWiki) and html format into
cleaned free text usable by okgraph.
Some methods are also useful to make it lowercase, for stemming, for co-occurrence (n-gram) tokenization, stop-words removal or idioms identification.
Some of these functions are taken from the Gensim preprocessing module and Gensim phrases module
(to be done)
Once an OKgraph
object has been instantiated, it can be used to execute the four task using its four methods:
set_expansion()
, relation_expansion()
, set_labeling()
and relation_labeling()
.
Every method takes four arguments:
seed
: is the generator used to compute the results. It will be: a list of strings in theset_expansion
andset_labeling
tasks; a list of string tuples in therelation_expansion
andrelation_labeling
tasks;k
(optional): is an integer specifying the limit to the number of results returned (setting it to-1
will avoid the limit). It is set to15
by default;algo
(optional): is a string specifying the name of the algorithm chosen as implementation of the task. Every task has its own default algorithm with default arguments, so this argument can be optionally not specified along with theoptions
argument;options
(optional): is a dictionary of the type {'argument': value} containing the values of the arguments requested by the chosen algorithm. Every task has its own default algorithm with default arguments, so this argument can be optionally not specified along with thealgo
argument;
To correctly execute a task it's important to know which implementations are available for each task, and so which
values can be assigned to the algo
and options
arguments. Every implementation has its own package, inside the
respective task package, containing a same-name module that implements the method task()
. The task()
method is
the one being called by the OKgraph
instance with the unpacked dictionary of arguments **options
.
The algo
arguments has to be the name of one of the packages and the options
arguments has to contain the packed
arguments for the respective task()
method.
All the set expansion algorithms can be found in the okgraph.task.set_expansion
package.
This is an example of usage with default values(using embeddings):
from okgraph.core import OKgraph
okg = OKgraph(corpus="text9.txt")
okg.set_expansion(["Italy", "France", "Germany"])
> e.g.: ["Spain", "Portugal", "Belgium", ...]
And another example using a specific algorithm (using embeddings):
from okgraph.core import OKgraph
okg = OKgraph(corpus="text9.txt")
okg.set_expansion(
seed=["Italy", "France", "Germany"],
k=15,
algo='centroid_boost',
options={"embeddings": okg.embeddings,
"step": 2,
"fast": False}
)
> e.g.: ["Spain", "Portugal", "Belgium", ...]
If you want to use a pretrained masked model you can use the fill-mask
algorithm, for example:
okg.set_expansion(
seed = ('italy', 'france', 'germany'),
k = 20,
algo = "fill_mask",
options = {}
)
All the relation expansion algorithms can be found in the okgraph.task.relation_expansion
package.
This is an example of usage with default values:
from okgraph.core import OKgraph
okg = OKgraph(corpus="text9.txt")
okg.relation_expansion([("Italy", "Rome"), ("Germany", "Berlin")])
> e.g.: [("Spain", "Madrid"),("Belgium", "Brussels"), ...]
And another example using a specific algorithm:
from okgraph.core import OKgraph
okg = OKgraph(corpus="text9.txt")
okg.relation_expansion(
seed=[("Italy", "Rome"), ("Germany", "Berlin")],
k=15,
algo="centroid",
options={"embeddings": okg.embeddings,
"set_expansion_algo": "centroid",
"set_expansion_options": {"embeddings": okg.embeddings},
"set_expansion_k": 15}
)
> e.g.: [("Spain", "Madrid"),("Belgium", "Brussels"), ...]
All the set labeling algorithms can be found in the okgraph.task.set_labeling
package.
This is an example of usage with default values:
from okgraph.core import OKgraph
okg = OKgraph(corpus="text9.txt")
okg.set_labeling(["Italy", "France", "Germany"])
> e.g.: ["country", "state", "nation", ...]
And another example using a specific algorithm:
from okgraph.core import OKgraph
okg = OKgraph(corpus="text9.txt")
okg.set_labeling(
seed=["Italy", "France", "Germany"],
k=15,
algo='intersection',
options={"dictionary": okg.dictionary,
"index": okg.index}
)
> e.g.: ["country", "state", "nation", ...]
All the relation labeling algorithms can be found in the okgraph.task.relation_labeling
package.
This is an example of usage with default values:
from okgraph.core import OKgraph
okg = OKgraph(corpus="text9.txt")
okg.relation_labeling([("Italy", "Rome"), ("Germany", "Berlin")])
> e.g.: ["capital", "capital_of", "soccer_team", ...]
And another example using a specific algorithm:
from okgraph.core import OKgraph
okg = OKgraph(corpus="text9.txt")
okg.relation_labeling(
seed=[("Italy", "Rome"), ("Germany", "Berlin")],
k=15,
algo='intersection',
options={"dictionary": okg.dictionary,
"index": okg.index}
)
> e.g.: ["capital", "capital_of", "soccer_team", ...]
The classes and methods in okgraph.evaluation.*
evaluate the performance of algorithms in each task based on several
benchmarks.
(to be done)
The followings are examples of use of the embeddings in okgraph:
from okgraph.core import OKgraph
okg = OKgraph(corpus="text9.txt")
# This is a WordEmbeddings class, specifically a MagnitudeWordEmbeddings
# instance
emb = okg.embeddings
# Obtain the vector representation of "town"
emb.w2v("town")
# Obtain the 5 closest word to "town"
emb.w2w("town", 5)
# Obtain the 5 words closest to the given vector representation (compatible with
# the embeddings)
v: ndarray
emb.v2w(v, 5)
# Obtain the 5 vector representations of the 5 words closest to the given vector
# representation (compatible with the embeddings)
v: ndarray
emb.v2v(v, 5)
More can be found in the okgraph.embeddings
module.
Tools that may be useful (not mandatory):
- hatch (commands reference)
- git flow (simple guide) and also git-flow-completion
To send a contribution:
- git checkout master
- git flow init -d (to set the default settings)
- git flow feature start my-cool-feature (use an appropriate feature name, for bugs use git flow bugfix start ...)
- add, commit and push your work (it will be in branch feature/my-cool-feature)
- follow the link suggested after the push to create a new push request to "develop" branch and start a discussion with the maintainer
- the maintainer will merge your work into develop (or master in case of new releases)
To implement a new task with an algorithm named new_implementation
, the new_implementation
package must be created
inside the task package. The new_implementation
package will have to contain the new_implementation.py
module
containing the task()
method, where the task()
method will effectively contain the algorithm.
For example, to implement a new relation labeling algorithm named my_rel_label_alg
, the following path will have
to exist: okgraph/task/relation_labeling/my_rel_label_alg/my_rel_label_alg.py
.
Look at existing methods for practical examples, e.g.: /tasks/set_expansion/centroid/centroid.py
All the project has been documented using the in-code Google Style Python Docstrings. The comments are extracted by Sphinx to automatically generate the library documentation. To update the documentation to match your updates, run the following script:
$ python docs/make_docs.py
To run the tests, from the root directory, run:
python -m unittest discover tests/ -v
If the data required to run the tests has not been acquired yet, the
tests/get_test_corpus_and_resources.py
script will be executed before testing.