mideind / greynirserver Goto Github PK

Display percentages as a bar graph (or pie?) and amounts relative to a set of standard yardsticks, such as the price of a liter of milk, the price of a 3-course meal at a restaurant, average monthly wage, the price of a common medium size car, the price of a 2-bedroom apartment, a well-known item in the fiscal budget, the GDP of Iceland, the GDP of the Nordic countries, the GDP of Europe, and the GDP of the world.

Add the ability to display a graphical timeline of appearances (in articles) of persons or entities. The timeline should be zoomable from a level of years down to days. For example, the user should be able to select Bjarni Benediktsson and see a graphical timeline of his frequency of appearances in news articles, zoomable down to individual articles (by headline). It might be interesting to allow two or more such timelines to be displayed in parallel (at synchronized zoom levels) for comparison purposes.

We at the Citizens Foundation in Iceland (Íbúar ses) are adding back an old feature to our Better Iceland website where we fetch issues from Alþingi to allow citizens to debate them. I'd be interested in using something like Reynir to help citizens understand laws better by providing some analysis and maybe key points from the law.

I wanted to know if you have had any experience using Reynir to parse laws from Alþingi?

Our citizens participation app we use for Better Iceland and Better Reykjavík is Open Source here:
https://github.com/rbjarnason/your-priorities-app

We are also working on an EU funded project called Active Citizen that aims to empower citizens democratically with AI and VR. We see Reynir fitting into this work as well and plan experiments with a dataset from Better Reykjavík.
https://github.com/rbjarnason/active-citizen
http://www.citizens.is/active

The article.api does not understand redirects and returns an error.

Eftirfarandi villa kom upp: 'NoneType' object has no attribute 'domain'

Steps to reproduce

Returns a valid response(http):
curl -d "url=http://www.ruv.is/frett/segir-nafn-svavars-gudnasonar-eydilagt" https://greynir.is/article.api/v1

Returns an invalid response(https):
curl -d "url=https://www.ruv.is/frett/segir-nafn-svavars-gudnasonar-eydilagt" https://greynir.is/article.api/v1

Note that the URL is an example taken from https://greynir.is/apidoc.

Allow users to go back to a previous query/URL/text using the browser back button or a dedicated back button.

For titles that are composite words with possessive prefixes, store two titles in the database. E.g., "fjármálaráðherra" -> "ráðherra fjármála"; "stjórnarþingmaður" -> "þingmaður stjórnar"; "framkvæmdastjóri" -> "stjóri framkvæmda"; "landsliðsfyrirliði" -> "fyrirliði landsliðs" ("liðsfyrirliði lands"?); "skorarformaður" -> "formaður skorar"

Use the topic vector engine to implement similarity queries where articles most similar to a given article are found and returned.

A primitive way to do this is to load the topic vectors of all parsed articles into memory (or use memory maps) and to scan them by brute force, calculating the cosine distance between each of them and the given search vector. Each vector is by default an array of 200 long (8-byte) floats, for a total of 1600 bytes per article. This means ~160 MB of memory for 100.000 articles and ~800 MB for 500.000 articles. (For these purposes, short (4-byte) floats are however quite sufficient.)

A more efficient, but slightly less accurate, way is to constrain the search to articles that have the same topics as the given article.

When attempting to find a composition of an unknown word, such as "samkeppnisyfirvöld", allow for an intervening "s" between known parts: "samkeppni" + "s" + "yfirvöld", at least if other solutions are not found.

Composite word generator should not create new conjunctions ('samtengingar') or other definitely closed word classes. For example, 'Bellen' should not be construed as 'Bell-en' where 'en' is a conjunction.

Einkunn á undan nafnorðum er óþarflega margræð þar sem bæði lo og so_lhþt eru leyfð þar en sömu orðmyndirnar eru yfirleitt til í báðum orðflokkum. Athuga má að búa til nýtt mark (elo?) fyrir þetta þar sem markarinn sér um að velja lo ef það er til en so LHÞT annars.

Add a Google Analytics cookie and log user queries to be able to look at traffic and query patterns and optimize accordingly.

After parsing, go through the resulting tree and token list to resolve relative date references and convert them to absolute dates ("í dag" -> the article's publication date, etc.) Use the absolute dates in further processing of the parse trees.

Scraper does not decode unicode escapes (\u0093, etc.) in article headings

  "stats": {
    "ambiguity": 1.0905077326652577, 
    "num_parsed": 1, 
    "num_sentences": 1, 
    "num_tokens": 8, 
    "parse_time": 0.18756818771362305, 
    "tok_time": 0.019917011260986328, 
    "total_time": 0.20748519897460938
  }

Describe the meaning of the stats keys in the docs, where appropriate.

Similarly to title lookup for people, show entity definitions with entity names where available.

got an error when importing ord.csv:

ERROR:  value too long for type character varying(16)
CONTEXT:  COPY ord, line 155, column ordfl: "nafnháttarmerki),"

nafnháttarmerki is 15 chars long so it should have been imported properly.

Changed to varchar(20) and successfully imported ord.csv

busla@708d680#diff-f94887202ef9aceae03b95bf25704909

Hi Vilhjálmur,

I'm working on a site called tala.is which aims to help people learning Icelandic to look up and correct inflections.

I'd like to use the data in Verbs.conf in tala.is. I haven't been able to find a data set on which cases verbs direct before, so I was very excited to find your project. Here's an example of how the data would be use in tala. I'd be happy to credit your project at the bottom.

I'd be really interested in talking about how our projects could work together to help people learn Icelandic.

Would you be willing to grant me permission to use the Reynir data in my own project?

Save and re-use token matching cache arrays both within individual sentences and across entire documents. This means that the same token (by token text & potentially kind) will only get its token-terminal match array filled once per terminal across the entire document, for a potentially large saving on token-terminal matching calls as tokens are typically repeated often within a document. The downside is a memory buffer of NumTerminals bytes per unique token within the document. This is not a problem, but if it were, the storage could be reduced to two bits per terminal (known/unknown, match/nomatch) for a total of NumTerminals / 4 bytes per unique token.

When a person with a non-Icelandic name occurs in a text, let his/her surname on its own refer to the same person as the full name including the surname. E.g., if "Hillary Clinton" occurs in an article, let the surname "Clinton" on its own refer to her. - Additionally, allow dropping of possessive 's' at end of non-Icelandic names if that causes the name to match a previously encountered non-possessive name. ("Clintons" would then match "Clinton".)

Correct cases are not always stored for percentage & amount tokens

For known place names, show a pop-up map when hovering over the name. Use four levels of map, if available: city/town, Iceland, continents, and world. Potentially show a summary map of all places mentioned in document when hovering over icon/link to that effect.

Allow multiple concurrent threads in scraper so that the CPU can keep working while threads wait for URL fetches - which can be quite slow, up to seconds per article.

The parsegrid view of parse trees shows the complete parse with direct correspondence to the Reynir CFG. This is a pretty complex view that often involves multiple stacked nonterminals on top of each other, and various artifacts of the way the CFG is constructed. Using e.g. utilities in treeutils.py, alternative, simpler views could be offered in the web UI that are closer to standard ways of representing parse trees.

In reducer, give additional points for combinations where prepositions are attached to verbs to form pairs that are found in the library. Example: "tala/ræða/þrasa um [þf]", "fara/ferðast/langa til [ef]", "komast/falla/detta í [þf]", "gangast við [þgf]", "bera undir [þf]"

I would love for the article.api endpoint to be expanded to parse a url, even though it isn't on a domain that http://greynir.is monitors.

The specific usecase I have in mind (for me), is entity extraction for http://umsagnir.gogn.in

Here's an example from the greynir.is web:

https://greynir.is/page?url=http%3A%2F%2Fumsagnir.gogn.in%2Fthing%2F146%2Fthingmal%2F80%2Fumsagnir%2F155

So a post request with either a url or text, return: json.

When attaching prepositions such as "á" and "í", consider whether the verb being attached to denotes movement or stationary placement, i.e. "fara á ballið" vs. "vera á ballinu". A noun in most (all?) cases does not attach to a movement preposition ("á"/"í" + þf.).

Merge capitalized unknown words into sequences and handle as a single token - they are probably entity names, possibly foreign.

Many sentences end with ", sagði Páll [Pálsson] [í samtali við Morgunblaðið]", i.e. ", said X [in an interview with The Times]". These tend to be interpreted by Greynir as an explanation of the preceding noun phrase, rather than an independent sub-sentence. By tuning the reducer, the latter can be given additional priority so that it wins the competition over an explanation (which is currently highly prioritized).

No checks for empty dicts are made in templates/genders.html:
https://github.com/vthorsteinsson/Reynir/blob/master/templates/genders.html#L26
https://github.com/vthorsteinsson/Reynir/blob/master/templates/genders.html#L41

Calculating empty vars results in:

Traceback (most recent call last):
  File "/Users/levy/Code/Reynir/venv/site-packages/flask/app.py", line 2000, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/levy/Code/Reynir/venv/site-packages/flask/app.py", line 1991, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/levy/Code/Reynir/venv/site-packages/flask/app.py", line 1567, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/levy/Code/Reynir/venv/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/levy/Code/Reynir/venv/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/levy/Code/Reynir/venv/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/levy/Code/Reynir/venv/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/levy/Code/Reynir/venv/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/levy/Code/Reynir/venv/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/levy/Code/Reynir/venv/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/levy/Code/Reynir/main.py", line 135, in decorated_function
    resp = f(*args, **kwargs)
  File "/Users/levy/Code/Reynir/main.py", line 754, in genders
    return render_template("genders.html", result = result, total = total)
  File "/Users/levy/Code/Reynir/venv/site-packages/flask/templating.py", line 134, in render_template
    context, ctx.app)
  File "/Users/levy/Code/Reynir/venv/site-packages/flask/templating.py", line 116, in _render
    rv = template.render(context)
  File "/Users/levy/Code/Reynir/venv/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/Users/levy/Code/Reynir/venv/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/levy/Code/Reynir/venv/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/Users/levy/Code/Reynir/templates/genders.html", line 2, in top-level template code
    {% extends "container-normal.html" %}
  File "/Users/levy/Code/Reynir/templates/container-normal.html", line 2, in top-level template code
    {% extends "container.html" %}
  File "/Users/levy/Code/Reynir/templates/container.html", line 83, in top-level template code
    {% block container %}
  File "/Users/levy/Code/Reynir/templates/container-normal.html", line 9, in block "container"
    {% block content %}
  File "/Users/levy/Code/Reynir/templates/genders.html", line 45, in block "content"
    <td class="count female">{{ (total.kvk / (total.kvk + total.kk) * 100) | format_is(1) }}%</td>
  File "/Users/levy/code/pypy3/lib_pypy/_decimal.py", line 665, in method
    self, other, strict=False)
  File "/Users/levy/code/pypy3/lib_pypy/_decimal.py", line 1335, in method
    mpd_func(res._mpd, a._mpd, b._mpd, ctx, status_ptr)
  File "/Users/levy/code/pypy3/lib_pypy/_decimal.py", line 1621, in __exit__
    self.context._add_status(status)
  File "/Users/levy/code/pypy3/lib_pypy/_decimal.py", line 1253, in _add_status
    raise exception
decimal.InvalidOperation

master...busla:master#diff-0a08a47dcc2d0c9b0f69ac11b12feec3

After parsing and reduction, parse trees need to be stored in the database in a helpful format for further processing down the line. Consider JSON and other alternatives. Also consider grammar augmentation for markup of stored trees, if necessary.

Reynir currently does not POS tag sentences for which no parse tree is found. Adding a rudimentary independent POS tagging feature would add usefulness and improve POS tagging accuracy scores. A separate POS tagger could inter alia build on BÍN and trigram statistics, and eventually an RNN language model.

Create a separate tokenizer module for Icelandic by splitting the existing module from Reynir and moving it into an installable Python package on PyPi. This would facilitate its use in independent NLP projects for Icelandic and promote standardization in the field. Such a module might conceivably have a broader license than GPLv3.

The reducer should encourage sentence trees where 1st and 2nd person pronouns - "ég"/"við" and "þú"/"þið" - have the same gender throughout the sentence. (Ideally, this would apply to some extent across entire articles.)

Get this error when POST´ing.

Running Greynir on Docker.

172.18.0.1 - - [02/Dec/2016 15:58:53] "POST /query.api HTTP/1.1" 500 -
greynir_web | Traceback (most recent call last):
greynir_web |   File "/usr/local/site-packages/flask/app.py", line 1994, in __call__
greynir_web |     return self.wsgi_app(environ, start_response)
greynir_web |   File "/usr/local/site-packages/flask/app.py", line 1985, in wsgi_app
greynir_web |     response = self.handle_exception(e)
greynir_web |   File "/usr/local/site-packages/flask/app.py", line 1540, in handle_exception
greynir_web |     reraise(exc_type, exc_value, tb)
greynir_web |   File "/usr/local/site-packages/flask/_compat.py", line 33, in reraise
greynir_web |     raise value
greynir_web |   File "/usr/local/site-packages/flask/app.py", line 1982, in wsgi_app
greynir_web |     response = self.full_dispatch_request()
greynir_web |   File "/usr/local/site-packages/flask/app.py", line 1614, in full_dispatch_request
greynir_web |     rv = self.handle_user_exception(e)
greynir_web |   File "/usr/local/site-packages/flask/app.py", line 1517, in handle_user_exception
greynir_web |     reraise(exc_type, exc_value, tb)
greynir_web |   File "/usr/local/site-packages/flask/_compat.py", line 33, in reraise
greynir_web |     raise value
greynir_web |   File "/usr/local/site-packages/flask/app.py", line 1612, in full_dispatch_request
greynir_web |     rv = self.dispatch_request()
greynir_web |   File "/usr/local/site-packages/flask/app.py", line 1598, in dispatch_request
greynir_web |     return self.view_functions[rule.endpoint](**req.view_args)
greynir_web |   File "/usr/src/app/Reynir/main.py", line 471, in query
greynir_web |     auto_uppercase = q.islower() if auto_uppercase else False))
greynir_web |   File "/usr/src/app/Reynir/tokenizer.py", line 1590, in disambiguate_phrases
greynir_web |     token = next(token_stream)
greynir_web |   File "/usr/src/app/Reynir/tokenizer.py", line 1735, in recognize_entities
greynir_web |     token = next(token_stream)
greynir_web |   File "/usr/src/app/Reynir/tokenizer.py", line 1179, in parse_phrases_2
greynir_web |     next_token = next(token_stream)
greynir_web |   File "/usr/src/app/Reynir/tokenizer.py", line 1001, in parse_phrases_1
greynir_web |     next_token = next(token_stream)
greynir_web |   File "/usr/src/app/Reynir/tokenizer.py", line 741, in annotate
greynir_web |     w, m = db.lookup_word(t.txt, at_sentence_start, auto_uppercase)
greynir_web |   File "/usr/src/app/Reynir/bindb.py", line 272, in lookup_word
greynir_web |     return self._lookup(w, at_sentence_start, auto_uppercase, self._meanings_func)
greynir_web |   File "/usr/src/app/Reynir/bindb.py", line 425, in _lookup
greynir_web |     cw = Wordbase.dawg().slice_compound_word(w)
greynir_web |   File "/usr/src/app/Reynir/dawgdictionary.py", line 252, in dawg
greynir_web |     Wordbase._dawg = Wordbase._load()
greynir_web |   File "/usr/src/app/Reynir/dawgdictionary.py", line 246, in _load
greynir_web |     return Wordbase._load_resource("ordalisti") # Main dictionary
greynir_web |   File "/usr/src/app/Reynir/dawgdictionary.py", line 229, in _load_resource
greynir_web |     dawg.load(pname)
greynir_web |   File "/usr/src/app/Reynir/dawgdictionary.py", line 102, in load
greynir_web |     with codecs.open(fname, mode='r', encoding='utf-8') as fin:
greynir_web |   File "/usr/local/lib-python/3/codecs.py", line 896, in open
greynir_web |     file = builtins.open(filename, mode, buffering)
greynir_web | FileNotFoundError: [Errno 2] No such file or directory: '/usr/src/app/Reynir/resources/ordalisti.text.dawg'

Provide an easy entry point into the database of people names and titles, where connected articles, entities and people can be further explored

Implement a tree pattern matching engine allowing patterns to be specified flexibly and matched groups to be extracted during processing.

After parsing, do a pass through the result tree and token list to assign meanings to personal pronouns ("hann", "hún", "það" in different cases and numbers), by finding the objects to which they refer. Use these dereferenced objects when constructing person titles, entity definitions, etc.

I HTTP POST´ed the following text to /posteg.api (note that I´m not separating the text into paragraphs like instructed in the docs).

Vistvæna merkið, sem hefur verið notað frá því að reglugerð tók gildi um vistvæna framleiðslu árið 1998, er skýrt dæmi um grænþvott, segir Birgitta Stefánsdóttir, sérfræðingur hjá Umhverfisstofnun. Grænþvottur, er notað yfir það þegar fyrirtæki merkir vörur þannig að þær líti út fyrir að vera umhverfisvænar þó að þær séu það ekki.
Neytandinn verður að vera gagnrýninn
Birgitta segir ljóst að ýmis orð eru notuð til að tefla á móti löggildum vottunum um umhverfislegt ágæti. Dæmi um orð af þessu tagi eru vistvænt, grænt, og náttúrulegt og í þeim felast fullyrðingar sem erfitt er að skilgreina eða færa rök fyrir. Því beri neytendum að hafa varann á. Birgitta segir mikilvægt að neytandinn spyrji sig til dæmis: „Hvað gerir hnetur náttúrulegri en aðrar hnetur?“ Það sé jafnframt mikilvægt að neytandi kynni sér hvað merkingar á umbúðum þýði í raun og veru og hvort þau merki séu löggild, eins og það sem er lífrænt: „Sem eru alls ekki grænþvottur heldur lögverndað orð og búið að skilgreina algjörlega hvað það orð þýðir.“ Birgitta telur vistvænan landbúnað í raun vera hefðbundinn landbúnað og hefði í raun átt að vera skilgreining á lágmarkskröfum í landbúnaði en ekki eitthvað millistig milli hefðbundins landbúnaðar og lífræns. Neytandi fær ekki nægilegar upplýsingar. Merki Evrópusambandsins fyrir lífræna ræktun. „Það er ekkert sem bannar framleiðendum að nota þessi merki, þótt þau þýði ekki neitt. Vandamálið er að neytandinn veit það ekki,“ segir Birgitta. Þórunn Arna Árnadóttir, sem er sviðsstjóri hjá Neytendastofu, segir jafnframt að þeim berist ábendingar frá neytendum ef merkingar eru ekki fullnægjandi en þá þarf neytandinn líka að vita þegar svo er ekki. Birgitta segir það vissulega geta verið erfitt að upplýsa sig en þó sé oftast hægt að nálgast upplýsingar sækist neytandinn eftir því. Á heimasíðu Umhverfisstofnunar má til dæmis skoða mörg þeirra vottuðu lífrænu merkja sem finna má á Íslandi. Með því að velja vöru eða þjónustu með áreiðanlegum umhverfismerkjum geta neytendur verið vissir um að þeir eru að velja rétt. Á Íslandi er ein starfandi vottunarstofa fyrir lífrænar vörur: Tún. Merki Túns, eins og annarra vottaðra lífrænna vara, er trygging neytandans fyrir því að varan sé framleidd samkvæmt alþjóðlegum stöðlum um lífrænar vörur. Sjaldgæft að ríkið eigi hlut í grænþvotti. Birgitta segir að það sé erfitt að vinna á móti grænþvotti. Grænþvottur sé mjög algengur en nú vinni Evrópusambandið gagngert að því að sporna á móti honum. Hún segir algengasta grænþvottinn þannig að fyrirtæki búa sér til eigin umhverfisstaðal með merkjum þar sem fyrirtækin sjálf kynna sig sem umhverfisvæn fyrirtæki. Það er því óalgengt að yfirvöld spili eins stórt hlutverk í slíku eins og varð raunin þegar búin var til merkingarlaus vottun sem var einungis villandi fyrir neytendur. Fræg merki hluti af grænþvotti. Á heimasíðu Umhverfisstofnunar er bent á alþjóðleg merki sem að geta reynst villandi fyrir neytendur, því jafnvel þau geti talist til grænþvotta. Meðal þeirra er merki World Wildlife fund, sem er pandabjörn. Merkið segir ekkert um umhverfislegt ágæti vörunnar sem ber myndina, eða framleiðslu hennar, heldur einungis að framleiðandi vörunnar hefur styrkt sjóðinn. Sjóðurinn þjóni sínu hlutverki en varan getur verið framleidd á mjög óumhverfisvænan hátt. Vill að það verði bannað að nota vistvæna merkið. Birgitta segir að lítið hafi breyst eftir að reglugerð um vistvæna framleiðslu var felld úr gildi síðasta sumar. Hún telur að það hefði verið eðlilegt að banna notkun merkisins: „Eina sem hefur breyst er að þessi reglugerð er ekki lengur á bakvið þetta merki, sem er samt ekki svo mikill munur því reglugerðin var ekki sterk til að byrja með.“ Hún segist enn vonast til þess að gerðar verði ráðstafanir til þess að notkun merkisins verði bönnuð. Hún telur að á Íslandi væri eðlilegt að einungis væru notuð tvö merki, annað hvort að varan sé lífræn og/eða íslensk.

Got HTTP 500 and the following response:
Eftirfarandi villa kom upp:

Presently images are found using a Google custom search that is slow and can cost money. Once a search has been performed, cache the result URL in a database table and use it in subsequent searches until stale - say for 7 up to 30 days. Optionally cache the whole image file in a BLOB column.

If somebody has a title of X 'og' Y (X 'and' Y), create two title records in persons table, for X and Y.

Consider whether to display articles in the web UI with options to show various grammatical features by color coding, highlighting, boxing or other visuals. As an example, cases, verb forms, noun phrases, verb phrases, prepositional phrases, etc. could be shown in this way. This might aid users in visualizing and learning Icelandic grammar.

Allow parsed articles to be output in CoNLL-U format, cf. the Universal Dependencies project. This requires the definition and adaptation of the CoNLL-U format for Icelandic.

Add heuristics to favor attachment of prepositions to closest possible verb/noun, unless there are good reasons not to.

Use Bootstrap or a similar framework to implement a simpler, leaner, mobile-ready front-end UI. Emphasize the query bar and query results, with details and deeper features available in drill-downs and via menus.

A search for "rúv" and "ríkisútvarpið" returned no results.

That noun is quite common in Icelandic media.

A title query of 'framkvæmdastjóri Já' should not return 'framkvæmdastjóri Jáverks'. The title query should filter on an exact match OR like('title %'), note the space before the percent sign.

Be able to infer relations and associations between articles, persons and entities by storing and comparing their contexts, i.e. by indexing nouns and verbs in articles and being able to explore similar articles by word co-occurrence statistics. This could also potentially help to disambiguate between different persons having the same name.

Scrape a trusted currency exchange rate website daily to find reference rates. Perhaps store the ISK equivalent with the parse tree (as is done with person names) to have access to the original rate at the time the article was published.