Not working after the last update to v26 about fulltextsearch

[lukas-staab]

I tried again and got a working workaround. I got it running with the following versions:

Full text search - Elasticsearch Platform 25.0.0 (downgraded)
Full text search 26.0.0
Full text search - Files 26.0.0
Full text search - Files - Tesseract OCR 25.0.0 (26 not yet released)

How I did it step by step:

• deactivated ES Platform 26 App on NC webinterface
• deleting / moving out the 26 App files like: mv ./nextcloud/apps/fulltextsearch_elasticsearch fulltextsearch_elasticsearch-26.back
• Downloading v25: wget https://github.com/nextcloud-releases/fulltextsearch_elasticsearch/releases/download/25.0.0/fulltextsearch_elasticsearch-25.0.0.tar.gz (btw: this release is not listed on github, i just guessed the url based on the old ones)
• unextract it: tar -xf fulltextsearch_elasticsearch-25.0.0.tar.gz
• moving the v25 version in: mv fulltextsearch_elasticsearch ./nextcloud/apps/
• activating the v25 app in the webinterface (you might have to confirm that you really want to activate it)

Hope it helps someone!

[Happyfeet01]

i use this Version.

[Happyfeet01]

@ostasevych no security issue. Like the Version installed as service, both Versions listen to 127.0.01:9200. This means only to localhost. You can block this port for outgoing traffic with UFW.

[benjaminfrombe]

+1

[Happyfeet01]

Not working here:

`lltextsearch:check
Full text search 26.0.0

• Search Platform:
  Elasticsearch 26.0.0 (Selected)
  {
  "elastic_host": [
  "http://internesuche:********@127.0.0.1:9200"
  ],
  "elastic_index": "internesuche",
  "fields_limit": "10000",
  "es_ver_below66": "0",
  "analyzer_tokenizer": ""
  }

• Content Providers:
  Deck 1.9.0
  []
  Files 26.0.0
  {
  "files_local": "1",
  "files_external": "0",
  "files_group_folders": "0",
  "files_encrypted": "0",
  "files_federated": "0",
  "files_size": "10",
  "files_pdf": "1",
  "files_office": "1",
  "files_image": "0",
  "files_audio": "0",
  "files_chunk_size": "2",
  "files_fulltextsearch_tesseract": {
  "version": "25.0.0",
  "enabled": "1",
  "psm": "4",
  "lang": "de,eng",
  "pdf": "1",
  "pdf_limit": "0"
  }
  }`

I hope anyone can maintain these App again.

`lars@livesystem-debian:~$ sudo -uwww-data php8.2 /var/www/nextcloud/occ fulltextsearch:test

.Testing your current setup:
Creating mocked content provider. ok
Testing mocked provider: get indexable documents. (2 items) ok
Loading search platform. (Elasticsearch) ok
Testing search platform. fail
In Test.php line 304:

Search platform (Elasticsearch) down ?`

[mahoneyr]

Not sure if this is the same issue - with occ fulltextsearch:test I'm getting "No alive nodes. All the 1 nodes seem to be down." Have verified that I can browse to elasticsearch from externally and curl to it from server (I'm using a docker deployment and can curl from inside container).

[lukas-staab]

I have the same issue as the original author. I can curl elasticsearch instance and index, but NC Test does not work. With the 25 Fulltext Search / ES-Plugin on a 26 Nextcloud it worked (i added a manual exception to install), but since I upgraded the app to 26 it broke. I tried to manually downgrade by replacing the 26 code by the 25 code but failed to get Nextcloud running again. If this hints someone to a working workaround please let me know how you did it :)

[mahoneyr]

Thanks - downgrading to v25 with these instructions worked for me also

[ostasevych]

Thanks - downgrading to v25 with these instructions worked for me also

Meanwhile, what to do to make the platform v26 working?

[Happyfeet01]

hmmm not working for me @lukas-staab

[mahoneyr]

Is this actually an issue for fulltextsearch_elasticsearch?

I have it (temporarily) working by reverting to v25 and then reinstalling v26 through the app. Based on log files, I'm wondering if there's an issue with the files populating /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/psr in v26?

[Happyfeet01]

That fixed it for me ☺️https://github.com/R0Wi/elasticsearch-nextcloud-docker

[ostasevych]

That fixed it for me ☺️https://github.com/R0Wi/elasticsearch-nextcloud-docker

How? Is it working with the Full Text Search Platform version 26.0.0 in your case? As in my case it said failed.
Which ES client version do you use?

[ostasevych]

i use this Version.

Great! As far as I understand your app version is 26.0.0 and the ES Client id ocker is v.8.6.1, am I right?
Meanwhile, how did you manage to migrate your index?

[Happyfeet01]

Yes, correct @ostasevych.
I had no index anymore. I index the files for the first time.

[ostasevych]

Yes, correct @ostasevych. I had no index anymore. I index the files for the first time.

Meanwhile, what about security concerns? Is it so safe to trust to dockerised solution comparing to the legacy installed on the server side? I mean, might the index be accessible from outside in this case? What is your opinion?

[ostasevych]

Yes, correct @ostasevych. I had no index anymore. I index the files for the first time.

Need some help here. I have a lot of files, and the indexing process is quite long, so I've started it from remote ssh terminal several times due to drop/downs.

So, the curl shows that the data are indexed

curl -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "query_string" : {
            "query" : "test",
            "default_field" : "title"
        }
    }
}
'

But the
reply of the command:

sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:search userA test
is just
search

and no data in the webUI :(

So, how to check whether it works?

UPD After restarting the indexing I am getting tons of unknown errors:

Are any tools or methods to debug fulltextsearch indexing and querying?

[lukas-staab]

this could be the underling error: nextcloud/fulltextsearch_elasticsearch#256

@Happyfeet01 are you mayben not using basic auth with your docker setup? i did before i downgraded (also did not use docker)

[Happyfeet01]

No basic Auth. Only Cluster Name and Address. But your linked issue seems the reason.

Before I tried this Docker Method I using the auth for Elasticsearch.

[ostasevych]

Again about reindexing. I am still struggling to make the full text search working on my instance:

• NC 26.0.3 (bare metal)
• All fulltextsearch apps are update
• Using groufolders app
• Elasticsearch via dockerised container: 8.6.1

Reindexing has been done with the following commands:

$ sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:stop
$ curl -X DELETE localhost:9200/my-index
$ sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:reset
$ sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:index

Searching with curl in index "my-index" for the word "Ivanova" (our member) gives (user data are stripped and scratched):

$ curl -X GET "localhost:9200/my-index/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "title": "Ivanova"
    }
  }
}
'
{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 6,
      "relation" : "eq"
    },
    "max_score" : 9.4341955,
    "hits" : [
      {
        "_index" : "my-index",
        "_id" : "files:6811",
        "_score" : 9.4341955,
        "_ignored" : [
          "content.keyword"
        ],
        "_source" : {
          "owner" : "user1",
          "groups" : [
            "GroupA",
            "GroupB",
            "GroupC",
            "GroupD"
          ],
          "circles" : [ ],
          "metatags" : [
            "files_group_folders"
          ],
          "source" : "files_group_folders",
          "title" : "GroupA/Scanned/Ivanova.pdf",
          "users" : [ ],
          "content" : "Ivanova document"
		  "tags" : [ ],
          "attachment" : {
            "date" : "2022-04-06T06:29:11Z",
            "content_type" : "application/pdf",
            "format" : "application/pdf; version=\"A-2b\"",
            "modified" : "2022-04-06T06:29:11Z",
            "language" : "uk",
            "creator_tool" : "ABBYY FineReader 14",
            "content_length" : 1612
          },
          "provider" : "files",
          "subtags" : [ ],
          "parts" : {
            "comments" : ""
          },
          "links" : [ ],
          "share_names" : {
            "user0" : "GroupA/Scanned/Ivanov.pdf",
            "user1" : "Scanned/Ivanov.pdf",
            "user2" : "GroupA/Scanned/Ivanov.pdf",
            "user3" : ""
          },
          "hash" : "aeca335860b2f59954q5e7fd34b174a1"
        }
      },
      {
        "_index" : "my-index",
        "_id" : "files:6812",
        "_score" : 9.4341955,
        "_ignored" : [
          "content.keyword"
        ],
        "_source" : {
          "owner" : "user0",
          "groups" : [
            "GroupA",
            "GroupB",
            "GroupC",
            "GroupD"
          ],
          "circles" : [ ],
          "metatags" : [
            "files_group_folders"
          ],
          "source" : "files_group_folders",
          "title" : "GroupA/Scanned/Ivanova2.pdf",
          "users" : [ ],
          "content" : "Ivanova........................................................... ...................",
          "tags" : [ ],
          "attachment" : {
            "date" : "2022-04-01T08:00:50Z",
            "content_type" : "application/pdf",
            "format" : "application/pdf; version=\"A-2b\"",
            "modified" : "2022-04-01T08:00:50Z",
            "language" : "uk",
            "creator_tool" : "ABBYY FineReader 14",
            "content_length" : 1260
          },
          "provider" : "files",
          "subtags" : [ ],
          "parts" : {
            "comments" : ""
          },
          "links" : [ ],
          "share_names" : {
            "user0" : "GroupA/Scanned/Ivanova2.pdf",
            "user1" : "Scanned/Ivanova2.pdf",
            "user2" : "GroupA/Scanned/Ivanova2.pdf",
            "user3" : "",
          },
          "hash" : "e09e889376ebe62b907b8023f37d21a9"
        }
      },
      {
        "_index" : "my-index",
        "_id" : "files:1576",
        "_score" : 9.30954,
        "_source" : {
          "owner" : "user0",
          "groups" : [
            "GroupA",
            "GroupB",
            "GroupC",
            "GroupD"
          ],
          "circles" : [ ],
          "metatags" : [
            "files_group_folders"
          ],
          "source" : "files_group_folders",
          "title" : "GroupA/Scanned/Ivaniva CV.pdf",
          "users" : [ ],
          "content" : "",
          "tags" : [ ],
          "attachment" : {
            "date" : "2022-06-02T07:57:28Z",
            "keywords" : "Scanned image",
            "content_type" : "application/pdf",
            "author" : "NAPS2",
            "format" : "application/pdf; version=1.4",
            "modified" : "2022-06-02T07:57:28Z",
            "language" : "lt",
            "title" : "Scanned image",
            "creator_tool" : "NAPS2",
            "content_length" : 4
          },
          "provider" : "files",
          "subtags" : [ ],
          "parts" : {
            "comments" : ""
          },
          "links" : [ ],
          "share_names" : {
            "user0" : "GroupA/Scanned/Ivanova CV.pdf",
            "user5" : "GroupA/Scanned/Ivanova CV.pdf",
            "user3" : ""
          },
          "hash" : "53f4a218a7ac0f31648efc0834c35199"
        }
      },
      {
        "_index" : "my-index",
        "_id" : "files:518673",
        "_score" : 8.218001,
        "_ignored" : [
          "content.keyword"
        ],
        "_source" : {
          "owner" : "user0",
          "groups" : [
            "GroupD"
          ],
          "circles" : [ ],
          "metatags" : [
            "files_group_folders"
          ],
          "source" : "files_group_folders",
          "title" : "GroupA/Scanned/Ivanova Bulletin.docx",
          "users" : [
            "user8"
          ],
          "content" : "Ivanova Bulletin text text",
          "tags" : [ ],
          "attachment" : {
            "date" : "2023-06-13T09:19:00Z",
            "content_type" : "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
            "author" : "Administrator",
            "modifier" : "User6",
            "modified" : "2023-06-14T13:32:00Z",
            "language" : "uk",
            "content_length" : 1701,
            "print_date" : "2023-06-14T13:30:00Z"
          },
          "provider" : "files",
          "subtags" : [ ],
          "parts" : {
            "comments" : ""
          },
          "links" : [ ],
          "share_names" : {
            "user0" : "GroupA/Bulls/Ivanova Bull.docx",
            "user9" : "GroupA/Bulls/Ivanova Bull.docx"
          },
          "hash" : "99a498dbb1db6f68a6be3793e30a9476"
        }
      },
      {
        "_index" : "my-index",
        "_id" : "files:6806",
        "_score" : 7.622202,
        "_source" : {
          "share_names" : {
            "user0" : "GroupA/Scanned/Photos/Ivanova",
            "user2" : "Scanned/Photos/Ivanova",
            "user3" : ""
          },
          "owner" : "user0",
          "users" : [ ],
          "groups" : [
            "GroupA",
            "GroupB",
            "GroupC",
            "GroupD"
          ],
          "circles" : [ ],
          "links" : [ ],
          "metatags" : [
            "files_group_folders"
          ],
          "subtags" : [ ],
          "tags" : [ ],
          "hash" : "",
          "provider" : "files",
          "source" : "files_group_folders",
          "title" : "GroupA/Scanned/Ivanova",
          "parts" : [ ],
          "content" : ""
        }
      },
      {
        "_index" : "my-index",
        "_id" : "files:466569",
        "_score" : 5.747198,
        "_ignored" : [
          "content.keyword"
        ],
        "_source" : {
          "owner" : "user0",
          "groups" : [
            "GroupD"
          ],
          "circles" : [ ],
          "metatags" : [
            "files_group_folders"
          ],
          "source" : "files_group_folders",
          "title" : "GroupA/Applications/Ivanova.docx",
          "users" : [
            "user4",
            "user5",
            "user6",
            "user7"
          ],
          "content" : "Application of Ivanova text text text",
          "tags" : [ ],
          "attachment" : {
            "date" : "2023-03-10T12:25:00Z",
            "content_type" : "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
            "author" : "user5",
            "modifier" : "user10",
            "modified" : "2023-05-17T17:55:09Z",
            "language" : "uk",
            "content_length" : 1550
          },
          "provider" : "files",
          "subtags" : [ ],
          "parts" : {
            "comments" : ""
          },
          "links" : [ ],
          "share_names" : {
            "user0" : "GroupA/Applications/Ivanova.docx",
            "user6" : "GroupA/Applications/Ivanova.docx",
            "user3" : "",
          },
          "hash" : "f8c12747f9b240ed696385d2aff7f0fe"
        }
      }
    ]
  }
}

Next, I search for Ivanova on behind of a user0

$ sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:search user0 Ivanova
search
> Files
 - 518673 score:0
 - 480179 score:0
 - 514585 score:0
 - 527182 score:0
 - 531276 score:0
 - 363692 score:0

The documents are stored in the groupfolder, which is accessible to user0, user1, user2, user3, user10, but that's what I see when searching with the help of NC fulltextsearch app:

$ sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:search user10 Ivanova
search
> Files

$ sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:search user3 Ivanova
search
> Files
 - 97091 score:0

Obviously, users in webUI cannot see the proper output.

Would you be so kind to explain, why it is not working as expected? What should I do to make it working? Thanks!

[lukas-staab]

Are you aware that Elasticsearch 8 is not offical supported? I would start there. see https://github.com/nextcloud/fulltextsearch_elasticsearch

[meonkeys]

Or is it? https://github.com/nextcloud/fulltextsearch_elasticsearch/wiki#installation-of-elasticsearch-via-docker points to https://github.com/R0Wi/elasticsearch-nextcloud-docker, which suggests elasticsearch version 8.6.1.

[ostasevych]

Or is it? https://github.com/nextcloud/fulltextsearch_elasticsearch/wiki#installation-of-elasticsearch-via-docker points to https://github.com/R0Wi/elasticsearch-nextcloud-docker, which suggests elasticsearch version 8.6.1.

Exactly!

[lukas-staab]

Are you aware that Elasticsearch 8 is not offical supported? I would start there. see https://github.com/nextcloud/fulltextsearch_elasticsearch

There was an update to the Readme. Seems like ES-plugin 26+ supports ES 8. If the 26 plugin does not work that means you still need to use 7, otherwise 8 is (newly) working.

Also the update to ES 8 might solve some issues people had here when going up to ES 26 plugin

[overcuriousity]

Tried it yesterday, it did not work with ES plugin 26. and elasticsearch 8.6.1 or newer. Maybe it changed with nc version 27.0.2, which I downloaded a couple hours ago. not tested yet.