Comments (17)
Video showing the problem happen at different times through the index process.
http://www.screencast.com/t/zIvPSvje5bX
from fulltextsearch.
alright, were able to recreate your issue, appears when you create a fresh user. thanks.
from fulltextsearch.
Is there a way to do a CRC32 (or some other method) and store that in the search with each file - that way shared files won't need to be indexed more than once - if, when indexing, the CRC32 (or some other method) matches, then simply update the existing record in solr to account for the shared document.
Does that make sense ?
from fulltextsearch.
In fact, I store the mtime of the file in Solr, so it just compare when extracting. Also there is not extraction on shared file. The sharing part is managed later by updating some field on the Solr document.
Meaning that even if it writes 1000+ files, it won't do anything as long as it is a shared files.
I will write some documentation about it for 0.4.0 that I hope to release later this week.
from fulltextsearch.
Can you give a try to the 0.4.0-rc1 on master ? If it does not resolve your issue, please use with the --debug option:
./occ nextant:index --debug
from fulltextsearch.
`[root@kb public_html]# sudo -u apache /usr/bin/php -f ./occ nextant:check
Ping: ok
Checking Solr schema fields
-
Checking dynamic-field 'nextant_attr_*': ok.
-
Checking field 'nextant_owner': ok.
-
Checking field 'nextant_mtime': ok.
-
Checking field 'nextant_share': ok.
-
Checking field 'nextant_sharegroup': ok.
-
Checking field 'nextant_deleted': ok.
Your solr contains 594 documents
[root@kb public_html]# sudo -u apache /usr/bin/php -f ./occ nextant:indexThis is an alpha release, please report any issue to
https://help.nextcloud.com/t/nextant-navigate-through-your-cloud-using-solr/2954/ -
Extracting new files to Solr:
PHP Fatal error: Call to a member function getStorage() on null in /var/www/public_html/lib/private/Files/View.php on line 1689
[root@kb public_html]# sudo -u apache /usr/bin/php -f ./occ nextant[Symfony\Component\Console\Exception\CommandNotFoundException]
Command "nextant" is not defined.
Did you mean one of these?
nextant:optimize
nextant:index
nextant:clear
nextant:check
nextant:scan
[root@kb public_html]# sudo -u apache /usr/bin/php -f ./occ nextant:clear
This will remove all indexes on your Solr. Continue with this action? (y/N) y
Your Solr core is now empty
[root@kb public_html]# sudo -u apache /usr/bin/php -f ./occ nextant:index
This is an alpha release, please report any issue to
https://help.nextcloud.com/t/nextant-navigate-through-your-cloud-using-solr/2954/
Your solr is locked by a running script like an index command or background jobs (cron)
You can still use the --force
[root@kb public_html]# sudo -u apache /usr/bin/php -f ./occ nextant:optimize
Your core will not be accessible while optimize is running. Continue with this action? (y/N) Y
Your index has been optimized (1ms)
[root@kb public_html]# sudo -u apache /usr/bin/php -f ./occ nextant:scan
deprecated use nextant:index instead
[root@kb public_html]# sudo -u apache /usr/bin/php -f ./occ nextant:index
This is an alpha release, please report any issue to
https://help.nextcloud.com/t/nextant-navigate-through-your-cloud-using-solr/2954/
Your solr is locked by a running script like an index command or background jobs (cron)
You can still use the --force
[root@kb public_html]# sudo -u apache /usr/bin/php -f ./occ nextant:index --force
This is an alpha release, please report any issue to
https://help.nextcloud.com/t/nextant-navigate-through-your-cloud-using-solr/2954/
-
Extracting new files to Solr:
PHP Fatal error: Call to a member function getStorage() on null in /var/www/public_html/lib/private/Files/View.php on line 1689
[root@kb public_html]# sudo -u apache /usr/bin/php -f ./occ nextant:index --force --debugThis is an alpha release, please report any issue to
https://help.nextcloud.com/t/nextant-navigate-through-your-cloud-using-solr/2954/00FAC915-A2AE-48E5-8341-E4C99A578B37: 0/ 0 [>---------------------------] 0%
[InvalidArgumentException]
$absolutePath must be relative to "files"
nextant:index [--output [OUTPUT]] [-d|--debug] [-f|--force] [-bg|--background]
[root@kb public_html]#
`
from fulltextsearch.
Also, when accessing the admin settings, I'm shown "Number of documents : Solr Servlet is down" when the server is definitely up and running and the 'Test and Save' completes successfully.
from fulltextsearch.
Seems worst that before ...
You might have you more in your cloud's log after doing the ./occ nextant:index --debug
Please try using logreader and paste the last few lines here
from fulltextsearch.
{"reqId":"SQJdkQ62bQLzwnRlzHJH","remoteAddr":"","app":"no app in context","message":"Exception: {"Exception":"OCP\Files\NotFoundException","Message":"File with id \"3169\" has not been found.","Code":0,"Trace":"#0 \/var\/www\/public_html\/apps\/activity\/lib\/FilesHooks.php(222): OC\Files\View->getPath(3169)\n#1 \/var\/www\/public_html\/apps\/activity\/lib\/FilesHooks.php(151): OCA\Activity\FilesHooks->getSourcePathAndOwner('\/0265F9B0-30D2-...')\n#2 \/var\/www\/public_html\/apps\/activity\/lib\/FilesHooks.php(107): OCA\Activity\FilesHooks->addNotificationsForFileAction('\/0265F9B0-30D2-...', 'file_created', 'created_self', 'created_by')\n#3 \/var\/www\/public_html\/apps\/activity\/lib\/FilesHooksStatic.php(46): OCA\Activity\FilesHooks->fileCreate('\/0265F9B0-30D2-...')\n#4 [internal function]: OCA\Activity\FilesHooksStatic::fileCreate(Array)\n#5 \/var\/www\/public_html\/lib\/private\/legacy\/hook.php(106): call_user_func(Array, Array)\n#6 \/var\/www\/public_html\/lib\/private\/Files\/View.php(1233): OC_Hook::emit('OC_Filesystem', 'post_create', Array)\n#7 \/var\/www\/public_html\/lib\/private\/Files\/View.php(1159): OC\Files\View->runHooks(Array, '\/0265F9B0-30D2-...', true)\n#8 \/var\/www\/public_html\/lib\/private\/Files\/View.php(263): OC\Files\View->basicOperation('mkdir', '\/0265F9B0-30D2-...', Array)\n#9 \/var\/www\/public_html\/lib\/private\/Files\/Node\/Folder.php(145): OC\Files\View->mkdir('\/0265F9B0-30D2-...')\n#10 \/var\/www\/public_html\/lib\/private\/Files\/Node\/Root.php(345): OC\Files\Node\Folder->newFolder('\/0265F9B0-30D2-...')\n#11 \/var\/www\/public_html\/apps\/nextant\/lib\/Command\/Index.php(185): OC\Files\Node\Root->getUserFolder('0265F9B0-30D2-4...')\n#12 \/var\/www\/public_html\/apps\/nextant\/lib\/Command\/Index.php(131): OCA\Nextant\Command\Index->browseUserDirectory('0265F9B0-30D2-4...', Object(Symfony\Component\Console\Output\ConsoleOutput))\n#13 \/var\/www\/public_html\/3rdparty\/symfony\/console\/Command\/Command.php(259): OCA\Nextant\Command\Index->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))\n#14 \/var\/www\/public_html\/core\/Command\/Base.php(159): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))\n#15 \/var\/www\/public_html\/3rdparty\/symfony\/console\/Application.php(844): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))\n#16 \/var\/www\/public_html\/3rdparty\/symfony\/console\/Application.php(192): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Nextant\Command\Index), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))\n#17 \/var\/www\/public_html\/3rdparty\/symfony\/console\/Application.php(123): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))\n#18 \/var\/www\/public_html\/lib\/private\/Console\/Application.php(146): Symfony\Component\Console\Application->run(NULL, NULL)\n#19 \/var\/www\/public_html\/console.php(90): OC\Console\Application->run()\n#20 \/var\/www\/public_html\/occ(11): require_once('\/var\/www\/public...')\n#21 {main}","File":"\/var\/www\/public_html\/lib\/private\/Files\/View.php","Line":1700}","level":3,"time":"2016-09-23T13:56:34+00:00","method":"--","url":"--","user":"--"}
{"reqId":"SQJdkQ62bQLzwnRlzHJH","remoteAddr":"","app":"PHP","message":"Call to a member function getStorage() on null at /var/www/public_html/lib/private/Files/View.php#1689","level":3,"time":"2016-09-23T13:56:34+00:00","method":"--","url":"--","user":"--"}
from fulltextsearch.
Are you saying that it writes this in your log when you are doing the ./occ nextant:index --debug ?
Is this a local folder for sure ? not some dropbox stuff ?
from fulltextsearch.
Can you try the last release on master ?
from fulltextsearch.
Downloading the master.zip, unzipping it in the apps directory, running 'make', then running 'occ upgrade' yields:
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Set log level to debug
Checking whether the database schema can be updated (this can take a long time depending on the database size)
Done
27/27 [============================] 100%
Checked database schema update
Checking updates of apps
Checked database schema update for apps
Updating database schema
Updated database
Updating ...
PHP Fatal error: Call to a member function checkSchema() on null in /var/www/public_html/apps/nextant/lib/Migration/NextantUpgrade.php on line 63
from fulltextsearch.
Sorry for 2day late reply. I now see that you committed a lot of other changes. Should I only use the revision you mentioned or should I use the master.zip file when checking this issue ?
from fulltextsearch.
Please use the last revision, also the upgrade script should be fixed, sorry about that
from fulltextsearch.
When running the latest version, I'm able to start the index after clean, but in the Solr log file I'm seeing a lot of:
Error opening new searcher.
Exceeded limit of maxWarmingSearchers=2, try again later
errors. It is still working its way through but the number of failures seems to be quite high:
FD017632-B932-4972-B3CA-22D77DE07DBE: 0/ 0 [>---------------------------] 0%
FFBD67D3-669B-4CDD-9D89-543626770D4C: 0/ 0 [>---------------------------] 0%
admin: 0/ 0 [>---------------------------] 0%
kb-sync: 1257/ 1257 [============================] 100%
1 file(s) processed ; 563 extracted documents
0AA217C5-5367-4D36-8393-B0E60270962A: 1/ 1 [============================] 100%
kb-sync: 71/ 562 [===>------------------------] 12%
[updating] - 489.7 MB (%99.8) failure(s): 37
from fulltextsearch.
I believe the final issue I was experiencing was due to Apache Solr using all of its memory - so I simply edited the solr.in.sh file (centos) to give more memory to Apache Solr - from 512MB limit to 8GB limit - and reran the index after clean:
FFBD67D3-669B-4CDD-9D89-543626770D4C: 0/ 0 [>---------------------------] 0%
admin: 0/ 0 [>---------------------------] 0%
kb-sync: 1257/ 1257 [============================] 100%
594 file(s) processed ; 594 extracted documents
0AA217C5-5367-4D36-8393-B0E60270962A: 1/ 1 [============================] 100%
kb-sync: 593/ 593 [============================] 100%
[updating] - 411.8 MB (%5.2) failure(s): 0
[root@kb public_html]#
After giving Solr more memory, the failures dropped to 0 and the whole index process took only 2 minutes versus 30-45 minutes (with a high failure rate).
So, at least as far as my issue is concerning, this could be closed out.
from fulltextsearch.
in fact, the second index process was just fixing the failures, so yes it is faster :)
from fulltextsearch.
Related Issues (20)
- Question: When ES is working for files and deck provider how to switch off DB operation for those providers?
- 0 results, test returns "Unexpected SearchResult" error HOT 6
- fulltextsearch:index decrypts and copies all files to /tmp HOT 7
- indexing OpenOffice documents - first line after headline missing
- Links to results are wrong when searching in the desktop client HOT 2
- Search returns no results HOT 3
- Constantly getting warning message Exception while improving searchresult HOT 2
- Fulltextsearch upgrade fails 27.0.1 -> 27.1.4 HOT 13
- List of Platform Apps: Add Xapian
- Search return only results from filenames HOT 1
- Full Text Search Enable App fails "An error occurred during the request. Unable to proceed." HOT 3
- High, permantent CPU load on two MySQL connections
- Usability is a Joke HOT 2
- nc 28 compatibility HOT 1
- elasticsearch connecting string gets saved in db in cleartext HOT 1
- Open search result directly
- NC 29 Beta 3: Cron job execution fails because `OC\BackgroundJob\TimedJob` was not found HOT 19
- `No alive nodes. All the 1 nodes seem to be down.` in `fulltextsearch:reset` but `fulltextsearch:stop` is working HOT 4
- [Bug]: scrollto is not executed from fulltextsearch HOT 4
- Hard Error/Force Quit on files with parentheses - ( ) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fulltextsearch.