leonieweissweiler / cistem Goto Github PK

Looking for complete stemmer implementations for german I stumbled upon this. Why not add a web assembly export? I know its very experimental, but it would open up a lot of platforms already. ;) (pretty please)

If the word as input is not normalised to NFC, the following parts will not work:

• transformation of äöü to aou
• length() calculation
• substr($original, - $rest_length)

Working on "whatever" normalised would be complicated.

Hi!

I recognized, the first line of "cistem.cpp" says

#include "Cistem.hpp"

I assume it should be

#include "cistem.hpp"

Best,
Florian

Even in case-sensitive mode, the output is always lower case.
Is that on purpose?

I made a Rust translation of CISTEM for a project of mine: cistemrs. Probably you don't want to merge it into the repo, but maybe it's useful for other people using the language.

This issue is more of a comment and can be closed immediately. Thanks for the stemmer! It works very well for information retrieval. :)

Thank you for the useful stemmer. I have stumbled upon an issue:
There should be some exceptions for removing the prefix "ge".
E.g.
"Geschlecht" is stemmed to "schlecht"
"Gesellschaft" is stemmed to "sellschaft"
"gesamt" is stemmed to "samt"
"genau" is stemmed to "nau"
etc.

The difference of transformation between stem() and segments() is:

• removement of prefix ge-
• transformation of äöü to aou
• transformation of ß to ss

Example:

#!perl
use strict;
use warnings;
use utf8;

binmode(STDOUT,":encoding(UTF-8)");
binmode(STDERR,":encoding(UTF-8)");

use lib qw(../CISTEM);

use Cistem;

my @words = qw/geheilwässert/;

for my $word (@words) {
  for my $case_sensitive (0..1) {
    print 'Cistem::stem(',$word,',',$case_sensitive,'): ',
    Cistem::stem($word,$case_sensitive),"\n";
  }

  for my $case_sensitive (0..1) {
    print 'Cistem::segment(',$word,',',$case_sensitive,'): ',
    join('-',Cistem::segment($word,$case_sensitive)),"\n";
  }
}

Which results in:

~/github/perl/CISTEM-test$ perl cistem.t
Cistem::stem(geheilwässert,0): heilwass
Cistem::stem(geheilwässert,1): heilwass
Cistem::segment(geheilwässert,0): geheilwäss-ert
Cistem::segment(geheilwässert,1): geheilwäss-ert

I would expect the same segmentation:

ge-heilwass-ert

This would also allow sharing most of the code.

I guess having constants with the same value is not intended. In fact DOLLAR1_PATTERN does not match any "$1" occurences.