cbeer / solr_wrapper

Wrap your tests with Solr 5+

solrwrapper

Wrap any task with a Solr instance:

SolrWrapper.wrap do |solr|
  # Something that requires Solr
end

Or with Solr and a solr collection:

SolrWrapper.wrap do |solr|
  solr.with_collection(dir: File.join(FIXTURES_DIR, "basic_configs")) do |collection_name|
  end
end

Basic Options

SolrWrapper.wrap port: 8983,
                 verbose: true,
                 managed: true,
                 instance_dir: '/opt/solr'

Valid ruby and YAML options

Option
instance_dir	Directory to store the solr index files
url	URL of the TGZ artifact to download
mirror_url	Mirror to download the solr artifacts from (e.g. http://lib-solr-mirror.princeton.edu/dist/)
version	Solr version to download and install
port	port to run Solr on
version_file	Local path to store the currently installed version
download_dir	Local path for storing the downloaded Solr tgz file
artifact_path	Local path to the Solr tgz file
checksum	Path/URL to checksum
solr_xml	Path to Solr configuration
verbose	(Boolean)
managed	(Boolean)
ignore_checksum	(Boolean)
solr_options	(Hash)
env	(Hash)
persist	(Boolean) Preserves the data in you collection between startups
contrib	(Array)

solr.with_collection(name: 'collection_name', dir: 'path_to_solr_configs')

From the command line

$ solr_wrapper -p 8983

To see a list of valid options when using solr_wrapper to launch a Solr instance from the command line:

$ solr_wrapper -h

Configuration file

SolrWrapper can read configuration options from a YAML configuration file. By default, it looks for configuration files at .solr_wrapper and ~/.solr_wrapper.

You can also specify a configuration file when launching from the command line as follows:

$ solr_wrapper --config <path_to_config_file>

Cleaning your repository from the command line

By defualt SorlWrapper will clean out your data when it shuts down. If you utilize the preserve option your data will remain between runs.

To clean out data that is being preserved explicitly run:

$ solr_wrapper <configuration options> clean

Note You must use the same configuration options on the clean command as you do on the run command to clean the correct instance.

Rake tasks

SolrWrapper provides rake tasks for installing, starting and stopping solr. To include the tasks in your Rake environment, add this to your Rakefile

  require 'solr_wrapper/rake_task'

You can configure the tasks by setting SolrWrapper.default_instance_options. For example:

SolrWrapper.default_instance_options = {
    verbose: true,
    cloud: true,
    port: '8888',
    version: '5.3.1',
    instance_dir: 'solr',
    download_dir: 'tmp'
}
require 'solr_wrapper/rake_task'

solr_wrapper's Issues

Export rake tasks to downstream users

So that we can, e.g., rake solr:start.

We did this in jettywrapper: https://github.com/projecthydra/jettywrapper/blob/master/Rakefile#L7

should download solr.zip from a mirror

Solr product/apache does not want us downloading from the main apache site, they want people to use mirrors. I think it could possibly work fine hard-coded to a single mirror that does not have rate limits we know about (such as princetons), or more complicatedly an auto-discovery and random selection of a mirror, which is also possible.

Screen-scraping apache.org is not a polite solution to the rate limiting.

Rake solr:start startup message raises NoMethodError on @solr_instance.options

Seeing #42 closed, I pulled solr_wrapper from master.

When executing rake solr:start, the printed startup message attempts to call @solr_instance.options(), raising a NoMethodError

https://github.com/cbeer/solr_wrapper/blob/master/lib/solr_wrapper/tasks/solr_wrapper.rake#L20

$ rake solr:start
rake aborted!
Failed to start solr. NoMethodError: undefined method `options' for #<SolrWrapper::Instance:0x007f0ed7745e18>

NoMethodError: undefined method `options' for #<SolrWrapper::Instance:0x007f0ed7745e18>

Tasks: TOP => solr:start

solr_wrapper is not starting

Hi,
I am trying to run the command in my project:

solr_wrapper --config .solr_wrapper

However, i got the error below :

Starting Solr 6.0.0 on port 8983 ... Traceback (most recent call last):
	10: from /Users/users/.rbenv/versions/2.6.3/bin/solr_wrapper:23:in `<main>'
	 9: from /Users/users/.rbenv/versions/2.6.3/bin/solr_wrapper:23:in `load'
	 8: from /Users/users/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/solr_wrapper-2.1.0/exe/solr_wrapper:113:in `<top (required)>'
	 7: from /Users/users/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/solr_wrapper-2.1.0/lib/solr_wrapper/instance.rb:64:in `wrap'
	 6: from /Users/users/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/solr_wrapper-2.1.0/exe/solr_wrapper:114:in `block in <top (required)>'
	 5: from /Users/users/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/solr_wrapper-2.1.0/lib/solr_wrapper/instance.rb:215:in `with_collection'
	 4: from /Users/users/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/solr_wrapper-2.1.0/lib/solr_wrapper/instance.rb:151:in `create'
	 3: from /Users/users/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/retriable-3.1.2/lib/retriable.rb:56:in `retriable'
	 2: from /Users/users/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/retriable-3.1.2/lib/retriable.rb:56:in `times'
	 1: from /Users/users/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/retriable-3.1.2/lib/retriable.rb:61:in `block in retriable'
/Users/users/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/solr_wrapper-2.1.0/lib/solr_wrapper/instance.rb:152:in `block in create': Not started yet (RuntimeError)

My version information :

Ruby -v : ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-darwin18]
Rails -v : 5.1.7
solr_wrapper version : v2.1.0
which solr_wrapper : /Users/users/.rbenv/shims/solr_wrapper
Device: MacOS Mojave - 10.14.6 (18G103)
Java -v : openjdk 12.0.2 2019-07-16

I had no clue about what is going on on my laptop. It ran perfectly fine on one of my co-worker machine.
However, my co-worker installed those package several months ago. We were trying to compare the version and all things.
We are not sure if is there anything wrong with our environment variable, version incompatible.

Appreciate for any suggestion or clue.
Thanks,

Stash the downloads?

Stash the downloads into a tmp folder for safe keeping, check the tmp folder for the downloads before downloading them again? (Perhaps just download the md5 and use it to verify the stash again if a double check is required.) This could save some time when using solr_wrapper during transit-time development activities (commuting or flights or just anywhere with lousy network connections). e.g.

./tmp/solr_wrapper/solr-6.5.1.zip
./tmp/solr_wrapper/solr-6.5.1.zip.md5

Automatically getting latest solr version is failing

Apparently, Apache decided to no longer provide latest version info at
https://lucene.apache.org/latestversion.html

This breaks solr_wrapper's fetch_latest_version method.

A workaround is to provide a specific version in config.

Error on clean when bin is missing says solr still running

Run solr_wrapper clean twice. The first was successful. The error message on the second run was:

Please stop solr before cleaning

Of course, by that point no solr is running (or even installed) so that is unhelpful.

Travis-CI is reporting that solr_wrapper certificate has expired

I'm getting some test failures this week because of a certificate error when Travis-CI is setting up solr_wrapper:

rake aborted!
1522OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (certificate has expired)
1523/home/travis/build/joelakes/TCD-Hyrax-Web-App/vendor/bundle/ruby/2.5.0/gems/solr_wrapper-2.2.0/lib/solr_wrapper/configuration.rb:105:in `mirror_url'

All was working fine last week.

I tried updating my solr_wrapper gem to version 3.1.2 but got the same error.

I eventually got a workaround, by inserting a new mirror_url: https://archive.apache.org/dist/

Has the certificate expired? Anybody else having this issue?

Thanks,
Joe

travis log.txt

Check `bin/solr start` exit code

If bin/solr start fails, we should raise an exception and provide the user the $stderr messages.

consider adding other committer(s)?

solr_wrapper is code that many blacklight and samvera developers really rely on.

I think @cbeer is very busy, and sometimes takes some time to get to solr_wrapper issues, which can really increase development time and inconvenience for developer users.

One example is the "mirror_url" feature, which is pretty crucial because of the way apache.org sometimes rate-limits travis IPs.

A mirror_url feature was first added by cbeer in March 2018. https://github.com/cbeer/solr_wrapper/pull/113/files

It was not included in a solr_wrapper release until 2.0.0 released Jun 27.

But it was released with a bug that makes it unusable for any solr version that isn't latest release on apache.org (as opposed to archive.apache.org).

One fix to this problem was provided by @jcoyne on July 25th, and is still unmerged/unreleased. #119

I think perhaps @dkinzer's #114 from Jan 26 actually may fix this same problem. When I asked about a release incuding the mirror_url feature in #116 in May, @dkinzer actually noted that the mirror_url feature wasn't working without #114. But it was released with the bug anyway, and #114 still remains unmerged/released.

Here at my place of work, we spent several hours re-debugging the same bug that jcoyne already had an unmerged PR for, and dkinzer may have also 8 months ago.

It would be a huge help if those existing PR's could be addressed, so we can keep using the mirror_url feature, to avoid getting rate-limited by apache.org. The only alternatives we can think of is a fork of solr_wrapper, or monkey-patching some of the bugfixes in -- neither of which are attractive options.

But I understand @cbeer may have other priorities on his time. @cbeer, would you consider adding other commiter(s) (with rubygems release rights) to help resolve this issue that's been outstanding for ~8 months, and others as they come up?

Looking to help

I have no idea what you're doing here, and yet instantly want to help. :-)

I did some of this sort of admin work with SimpleSolrClient, but already your stuff looks betterererer. Lemme know if you want an extra set of eyes.

can combine solr_wrapper and solr cloud together??

Hi
Can I combine solr_wrapper and solr cloud that are on different servers?
And does anyone have a tutorial regarding my question earlier?

Thanks before :)

Use `bin/solr status` to determine whether solr is running (instead of storing local state)

Setting the download directory so you only get one instance downloaded

This was reported in samvera/active_fedora#1080 but was decided in samvera/active_fedora#1081 that the fix should be here in solr_wrapper.

Out of box configuration have solr 6.1

And it raised an error message during solr:start:
SolrWrapper::SolrWrapperError: Unable to download solr from http://www.us.apache.org/dist/lucene/solr/6.1.0/solr-6.1.0.zip.md5

Adding version:6.2.0 or ignore_md5sum=true in the configuration solve the issue. But it would be helpful to fix the default version.

release 1.3.0 with mirror_url?

Last week, a number of people are having trouble with tests on travis, timing out at the point solr_wrapper is trying to download solr. An example stack trace on travis: https://travis-ci.org/sciencehistory/chf-sufia/builds/377350433

Several reported that using solr_wrapper from git, with the mirror_url feature, and pointing mirror_url in config to princeton's mirror at http://lib-solr-mirror.princeton.edu/dist/ resolved the issue and let travis pass again. It was reported by several to be a fix. (I didn't try it myself while the problem was still going on).

This week, it no longer seems to be a problem. That does kind of suggest it's the issue with apache.org rate-limiting travis IPs, which seems to crop up now and then. So it will probably be an issue again.

As far as I can tell, the mirror_url feature is not in a released gem. Would it be possible to get a 1.3.0 of solr_wrapper released including the mirror_url, so we can use it, and avoid the apparent apache.org rate-limiting problem in the future?

I really prefer using actual releases on rubygems instead of dependencies to a git commit from github. Our dependency tree is confusing enough already. I know I'm not alone here. Would really appreciate a release.

Thank you!

poll push

When I run the solr server via solr_wrapper it writes to solr.log and solr-port-concole.log every second..

2016-03-09 23:15:27.822 INFO (qtp1450821318-18) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system params={wt=json} status=0 QTime=4

Is this by design. It seems to chew quite a bit of cycles w/out any requests.

Solr_wrapper keeps wanting to download a new solr instance, how can I configure the Solr_wrapper to use an existing Solr instance>

mirror_url should use https?

I was getting an EOR error using this gem until I changed the mirror_url defined in Configuration.rb to use https instead of http. I am not entirely sure under what exactly conditions this error occurs, but I thought I would report it.

Crash on start up

Hello,

I'm getting this error on start up and totally no idea why.

gem version 2.20

solr_wrapper -v
Loading configuration from /home/paul/app/.solr_wrapper
Starting Solr 8.7.0 on port 8983 ... /home/paul/.rvm/gems/ruby-2.7.2@app/gems/solr_wrapper-2.2.0/lib/solr_wrapper/configuration.rb:105: warning: calling URI.open via Kernel#open is deprecated, call URI.open directly or use URI#open
/home/paul/.rvm/gems/ruby-2.7.2@app/gems/json-2.1.0/lib/json/common.rb:156: warning: Using the last argument as keyword parameters is deprecated
Traceback (most recent call last):
	9: from /home/paul/.rvm/gems/ruby-2.7.2@app/bin/ruby_executable_hooks:24:in `<main>'
	8: from /home/paul/.rvm/gems/ruby-2.7.2@app/bin/ruby_executable_hooks:24:in `eval'
	7: from /home/paul/.rvm/gems/ruby-2.7.2@app/bin/solr_wrapper:23:in `<main>'
	6: from /home/paul/.rvm/gems/ruby-2.7.2@app/bin/solr_wrapper:23:in `load'
	5: from /home/paul/.rvm/gems/ruby-2.7.2@app/gems/solr_wrapper-2.2.0/exe/solr_wrapper:113:in `<top (required)>'
	4: from /home/paul/.rvm/gems/ruby-2.7.2@app/gems/solr_wrapper-2.2.0/lib/solr_wrapper/instance.rb:64:in `wrap'
	3: from /home/paul/.rvm/gems/ruby-2.7.2@app/gems/solr_wrapper-2.2.0/exe/solr_wrapper:114:in `block in <top (required)>'
	2: from /home/paul/.rvm/gems/ruby-2.7.2@app/gems/solr_wrapper-2.2.0/lib/solr_wrapper/instance.rb:215:in `with_collection'
	1: from /home/paul/.rvm/gems/ruby-2.7.2@app/gems/solr_wrapper-2.2.0/lib/solr_wrapper/instance.rb:158:in `create'
/home/paul/.rvm/gems/ruby-2.7.2@app/gems/solr_wrapper-2.2.0/lib/solr_wrapper/instance.rb:321:in `exec': Failed to execute solr create:  (RuntimeError)
ERROR: Failed to create collection 'hydra-development' due to: Underlying core creation failed while creating collection: hydra-development

The related log entries are:

2020-12-17 11:35:25.878 ERROR (OverseerThreadFactory-18-thread-1-processing-n:127.0.1.1:8983_solr) [   ] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: hydra-development operation: create failed:org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: hydra-development
        at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:310)
        at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264)
        at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:517)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

java -version
openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-Ubuntu-0ubuntu1.20.10)
OpenJDK 64-Bit Server VM (build 11.0.9.1+1-Ubuntu-0ubuntu1.20.10, mixed mode, sharing)

During the boot process I ran ps aux | grep [s]olr in another window and got this. Then the crash happens and no Solr could be found.

paul       17384 13.2  0.5 181288 44956 pts/1    Sl+  11:45   0:00 ruby /home/paul/.rvm/gems/ruby-2.7.2/bin/solr_wrapper -v
paul       17405  1.0  0.0  19924  3916 pts/1    S+   11:45   0:00 bash /tmp/solr-8.7.0/bin/solr start -p 8983 -c
paul       17453  283  3.6 5618012 285208 pts/1  Sl+  11:45   0:05 java -server -Xms512m -Xmx512m -XX:+UseG1GC -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=250 -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:+ExplicitGCInvokesConcurrent -Xlog:gc*:file=/tmp/solr-8.7.0/server/logs/solr_gc.log:time,uptime:filecount=9,filesize=20M -Dsolr.jetty.inetaccess.includes= -Dsolr.jetty.inetaccess.excludes= -DzkClientTimeout=30000 -DzkRun -Dsolr.log.dir=/tmp/solr-8.7.0/server/logs -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -XX:-OmitStackTraceInFastThrow -Djetty.home=/tmp/solr-8.7.0/server -Dsolr.solr.home=/tmp/solr-8.7.0/server/solr -Dsolr.data.home= -Dsolr.install.dir=/tmp/solr-8.7.0 -Dsolr.default.confdir=/tmp/solr-8.7.0/server/solr/configsets/_default/conf -Xss256k -Dsolr.jetty.https.port=8983 -Dsolr.log.muteconsole -XX:OnOutOfMemoryError=/tmp/solr-8.7.0/bin/oom_solr.sh 8983 /tmp/solr-8.7.0/server/logs -jar start.jar --module=http
paul       17457  0.0  0.0  19792  2500 pts/1    S+   11:45   0:00 bash /tmp/solr-8.7.0/bin/solr start -p 8983 -c

Any help woud be much appreciated.

Thank you.

solr_wrapper --version=... is heavily anti-pattern

solr_wrapper --version should output version info for solr_wrapper. Another option should be used to set Solr version to be used, possibly --solr-version=.

old md5s should be retrieved from the archive

SolrWrapper::SolrWrapperError: Unable to download solr from http://www.us.apache.org/dist/lucene/solr/6.1.0/solr-6.1.0.zip.md5

Failure to create core shouldn't raise an exception.

Should we trap for this error in with_collection and just log it?

Failed to execute solr create:
ERROR:
Core 'blacklight-core' already exists!
Checked core existence using Core API command:
http://localhost:8983/solr/admin/cores?action=STATUS&core=blacklight-core

Orphaned java process after starting with Foreman

When starting solr_wrapper using Foreman, sending an interrupt (Ctl+C) to Foreman leaves an orphaned java process still running Solr. The solr_wrapper binary exits, but it leaves the java process running. Observed on OS X 10.11.5.

To reproduce:

1. Add solr_wrapper and foreman to the Gemfile
2. bundle install
3. Create a Procfile containing solr: solr_wrapper -p 8983
4. Run bundle exec foreman start
5. Allow Solr to start up
6. Press Ctl+C to kill Foreman
7. Run ps aux | grep "[j]ava" and observe Solr still running.

The issue does not seem to happen on every run, but happens with some regularity.

A repo demonstrating the issue can be found here: https://github.com/electrickite/solr_wrapper_foreman

Idea: Version as X.Y.Z.A, where X.Y.Z is the solr version targeted

I don't know if this is a good idea or not, but it sure would make it easy to know which version of the wrapper will be downloading which version of solr.

Just a thought from an impressed user.

Might find something useful in simple solr client repo

Your stuff is....well, let's just say "more elegant" than mine, but you might get some ideas and/or code from similar work I did at https://github.com/billdueber/simple_solr

Just a head's up, in case you hadn't seen it.

Capture Psych::SyntaxError and display a helpful message

$ solr_wrapper
/home/vagrant/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/psych.rb:377:in `parse': (<unknown>): could not find expected ':' while scanning a simple key at line 4 column 1 (Psych::SyntaxError)
    from /home/vagrant/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/psych.rb:377:in `parse_stream'
    from /home/vagrant/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/psych.rb:325:in `parse'
    from /home/vagrant/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/psych.rb:252:in `load'
    from /home/vagrant/.rvm/gems/ruby-2.3.1/gems/solr_wrapper-0.18.0/lib/solr_wrapper/configuration.rb:133:in `read_config'

Like "Your config file is invalid"

rake tasks do not respect .solr_wrapper file

I have a .solr_wrapper file that specifies solr version 6.3.0.

If I run bundle exec solr_wrapper, as expected I get:

Starting Solr 6.3.0 on port 8983 ..

However, if I use the solr_wrapper rake tasks, (require 'solr_wrapper/rake_task'), and run bundle exec solr:start, it seems to ignore the .solr_wrapper file, I get:

Starting solr at http://127.0.0.1:8983/solr/
solr-8.0.0.zip: |================                               | 70% ( ETA: 00:00:06 )

Is this a bug, should the rake task be using the .solr_wrapper config file too?

If so, I am happy to prepare and submit a PR, feedback welcome.

Expose -force parameter for running as root

Blacklight their docker files run on alpine linux, which only has a root user. The docker-compose now fails because the solr_wrapper refuses to continue as its being started as root.

The -force paremeter is not exposed in solr_wrapper meaning I am now at a loss.

I am in the progress of starting a small museum and really want to test different collection management systems with some ontological values.

solr_wrapper attempts to create a collection when one already exists.

``
Starting Solr 6.2.0 on port 8983 ... /Users/jcoyne/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/solr_wrapper-0.18.1/lib/solr_wrapper/instance.rb:356:in`exec': Failed to execute solr create: (RuntimeError)
Connecting to ZooKeeper at localhost:9983 ...
Re-using existing configuration directory hydra-development

ERROR:
Collection 'hydra-development' already exists!
Checked collection existence using Collections API command:
http://localhost:8983/solr/admin/collections?action=list

from /Users/jcoyne/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/solr_wrapper-0.18.1/lib/solr_wrapper/instance.rb:154:in `create'
from /Users/jcoyne/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/solr_wrapper-0.18.1/lib/solr_wrapper/instance.rb:211:in `with_collection'
from /Users/jcoyne/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/solr_wrapper-0.18.1/exe/solr_wrapper:109:in `block in <top (required)>'
from /Users/jcoyne/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/solr_wrapper-0.18.1/lib/solr_wrapper/instance.rb:64:in `wrap'
from /Users/jcoyne/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/solr_wrapper-0.18.1/exe/solr_wrapper:108:in `<top (required)>'
from /Users/jcoyne/.rbenv/versions/2.3.0/bin/solr_wrapper:23:in `load'
from /Users/jcoyne/.rbenv/versions/2.3.0/bin/solr_wrapper:23:in `<main>'

Wrapper is not shutting down all processes

the stop method is looking at pid 52139

but after solr_wrapper completes, I see this:

$ ps aux|grep solr
justin          51982   0.1  1.5  4478372 246632 s009  S     8:12AM   0:07.72 /usr/bin/java -server -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/folders/qs/vbqrmjh91kv6q_w0zj977_sc0000gn/T/solr-5.4.1/server/logs/solr_gc.log -Djetty.port=50307 -DSTOP.PORT=49307 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/var/folders/qs/vbqrmjh91kv6q_w0zj977_sc0000gn/T/solr-5.4.1/server -Dsolr.solr.home=/var/folders/qs/vbqrmjh91kv6q_w0zj977_sc0000gn/T/solr-5.4.1/server/solr -Dsolr.install.dir=/var/folders/qs/vbqrmjh91kv6q_w0zj977_sc0000gn/T/solr-5.4.1 -Xss256k -jar start.jar -XX:OnOutOfMemoryError=/var/folders/qs/vbqrmjh91kv6q_w0zj977_sc0000gn/T/solr-5.4.1/bin/oom_solr.sh 50307 /var/folders/qs/vbqrmjh91kv6q_w0zj977_sc0000gn/T/solr-5.4.1/server/logs --module=http

checksum_path doesn't work with mirror_url

"body has already been consumed"

Today solr_wrapper started failing for me trying to download solr.

(Not sure when the last time I downloaded solr was, since I normally cache it, it's possible my caches were just busted today. So not sure when last known good working was).

I am curious if anyone else is running into this. It happens on my local machine, not just CI, and when pointing at princeton mirror, so it's not a rate-limiting issue.

          SolrWrapper::SolrWrapperError:
            Unable to download solr from http://lib-solr-mirror.princeton.edu/dist/lucene/solr/8.6.3/solr-8.6.3.zip
            body has already been consumed

solr_wrapper: 3.1.2
http gem: reproduces with http 5.0.1 or 4.4.1. At first I thought it might be caused by upgrading to http 5, but does not seem to be.

Anyone else running into this? I am trying to debug, but having trouble figuring out what's going on.

Downloading Solr fails after new version is released

After a new version of Solr is released, the old version is quickly removed from the Apache mirrors, so downloading fails.

Can we fallback on using the Apache archive for older releases, and log a prominent warning message to let people know they should update?

Is Rubocop a dependency?

Rubocop configs exist in the repo, but it is not in the gemspec.

Why does managed? => false imply started?

when I type
solr_wrapper clean

I get the message:
"Please stop solr before cleaning"

This is happening because
instance.started? is checking status, which returns true when config.managed? is false.
managed? returns false because the instance_dir doesn't exist.

So the message is misleading.

Describe the order of precedence for configuration in README

It would be nice for the readme to clearly describe the order of precedence for configuration options. Particularly since it now looks like solr_wrapper will read multiple config files simultaneously ;)

I just ran a recent version of solr wrapper and it says it's loading configuration from two files

$ bin/rails ci
Loading configuration from /home/vagrant/laevigata/.solr_wrapper.yml
Loading configuration from /home/vagrant/laevigata/.solr_wrapper
solr-6.5.0.zip: ...

I didn't create either file so I'm assuming both are created by generators called by the hyrax template application following the instructions here https://github.com/projecthydra-labs/hyrax#creating-a-hyrax-based-app. Based on the contents, it looks like .solr_wrapper.yml is put there by a blacklight generator and .solr_wrapper by some hydra/hyrax generator.

$ bundle list solr_wrapper
/home/vagrant/.rvm/gems/ruby-2.3.4@laevigata/gems/solr_wrapper-0.23.0
``

hydra 10 depending on solr_wrapper 0.12.1 which appears to have a non existant mirror reference

When I pull hydra 10.0.0 I get in hydra.gemspec..
gem.add_dependency 'solr_wrapper', '~> 0.12.1'

which I mention because of..
samvera-deprecated/hydra#108

and with that setup my initial run of solr_wrapper attmpts to download..
http://www.trieuvan.com/apache/lucene/solr/6.0.1/solr-6.0.1.zip

which does not exist there.

Is solr_wrapper 0.12.1 the proper version for hydra 10?
Where is the download url set and is the version of solr correct. I do find..
...solr/6.1.0/solr-6.1.0.zip at the trieuvan site above. I can't seem to find where, in the solr_wrapper-0.12.1 source where config.download_url is set.

Running solr_wrapper a 2nd time with "persist" option set does not find the existing core

Versions used:

• Using solr_wrapper 0.12.0
• Using Solr 6.4.1 (the version selected by solr_wrapper)

Expected Result:

I should be able to:

1. run solr_wrapper for a 1st time, with a configuration that has the persist flag set to true.
2. stop solr_wrapper
3. run solr_wrapper a 2nd time, with the same config, and not get an error.

Actual Results:

Running solr_wrapper a 2nd time, with the same config, gives an error (see below).

To Replicate:

1. create a config file named .solr_wrapper.yml containing the following:

   collection:
     dir: solr/config/
     name: my-core
     persist: true

2. Run solr_wrapper for the first time (without the core having been created yet) with the specified config, like this...

   solr_wrapper --config .solr_wrapper.yml
   

3. At this point, Solr starts.

4. Stop Solr using Ctrl+C

5. Run solr_wrapper again, with the same command and config file...

   solr_wrapper --config .solr_wrapper.yml
   

6. At this point, I get the following error...

   ERROR: 
   Core 'my-core' already exists!
   Checked core existence using Core API command:
   http://localhost:8983/solr/admin/cores?action=STATUS&core=my-core
   

Background:

• I've tracked this down to the fact that SolrWrapper::Client#core? is returning false when checking for the core named my-core.
• In my case, the core was created (during the first start of solr_wrapper) in /tmp/solr-6.4.1/server/solr/my-core, and after getting the error, I can still see the directory corresponding to my-core on the filesystem.
• The Solr api call being made to check for the core inside of SolrWrapper::Client#core? looks like this: admin/cores?action=STATUS&wt=json&core=my-core'
• The response returned by Solr is:

   {"responseHeader":{"status":0,"QTime":1},"initFailures":{},"status":{"my-core":{}}}

• This would indicate that SolrWrapper::Client#core? is actually behaving correctly and returning false, given the response from Solr. However, Solr itself is not finding the core that it created previously.
• This causes SolrWrapper::Client#exists? to return false, which in turn causes this conditional to also return false, which in turn runs the line exec('create', create_options).
• And it is the call to exec('create', create_options) where Solr is somehow able to find the core it couldn't find before, and throws the error for trying to create a core that already exists.

Explicit config silently ignored

From .internal_test_app I run:

solr_wrapper -v -d solr/config/ -n hydra-test -p 8985

I expect the running solr to use the configurations supplied at solr/config.

But the live solrconfig at http://127.0.0.1:8985/solr/#/hydra-test/files?file=solrconfig.xml doesn't match solr/conf/solrconfig.xml, i.e. the configuration I specified directly.

Maybe this is from invoking solr_wrapper previously? If so, the mode of silently ignoring the command is bogus. I would expect to use serial execution to facilitate testing different configurations, like:

solr_wrapper -v -d solr/config/ -n hydra-test -p 8985
solr_wrapper -v -d solr/config2/ -n hydra-test -p 8985
solr_wrapper -v -d solr/config3/ -n hydra-test -p 8985

If it cannot, then I expect it to complain loudly, at a minimum. Preferably it should fail loudly, since it cannot do the one thing solr_wrapper is good for.

Rake solr:environment calls SolrWrapper.instance without options arg

SolrWrapper.instance was recently introduced with a non-optional options argument
baa610c

The solr:environment task attempts to call SolrWrapper.instance without any arguments, causing an ArgumentError in any dependent task.

https://github.com/cbeer/solr_wrapper/blob/master/lib/solr_wrapper/tasks/solr_wrapper.rake#L6-L8

download_path gives error, download_dir does not

I'd like to be able to persist my downloaded source solr file (so I can distribute it in classes & workshops as part of a Vagrant box). When I use the download_path option described in the README, I get and error; when I use download_dir, things run as I'd expect.

In the two examples below, I have previously downloaded the zip and md5 files to /var/tmp

using download_dir

$ rm -rf /tmp/solr-5.4.1

$ ls /var/tmp
solr-5.4.1.zip  solr-5.4.1.zip.md5

$ cat .solr_wrapper 
port: 8983
download_dir: /var/tmp
version_file: /vagrant/heliotrope/tmp/solr-5.4.1
collection:
  dir: /vagrant/heliotrope/solr/config
  name: hydra-development

$ solr_wrapper
Loading configuration from /vagrant/heliotrope/.solr_wrapper
Starting Solr 5.4.1 on port 8983 ... http://127.0.0.1:8983/solr/ 

using download_path

$ rm -rf /tmp/solr-5.4.1

$ ls /var/tmp
solr-5.4.1.zip  solr-5.4.1.zip.md5

$ cat .solr_wrapper 
port: 8983
download_path: /var/tmp
version_file: /vagrant/heliotrope/tmp/solr-5.4.1
collection:
  dir: /vagrant/heliotrope/solr/config
  name: hydra-development

$ solr_wrapper
Loading configuration from /vagrant/heliotrope/.solr_wrapper
Starting Solr 5.4.1 on port 8983 ... /home/vagrant/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/digest.rb:52:in `read': Is a directory @ io_fread - /var/tmp (Errno::EISDIR)
    from /home/vagrant/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/digest.rb:52:in `block in file'
    from /home/vagrant/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/digest.rb:50:in `open'
    from /home/vagrant/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/digest.rb:50:in `file'
    from /home/vagrant/.rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/digest.rb:35:in `file'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/gems/solr_wrapper-0.7.3/lib/solr_wrapper/md5.rb:15:in `validate?'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/gems/solr_wrapper-0.7.3/lib/solr_wrapper/instance.rb:233:in `download'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/gems/solr_wrapper-0.7.3/lib/solr_wrapper/instance.rb:195:in `extract'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/gems/solr_wrapper-0.7.3/lib/solr_wrapper/instance.rb:184:in `extract_and_configure'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/gems/solr_wrapper-0.7.3/lib/solr_wrapper/instance.rb:59:in `wrap'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/gems/solr_wrapper-0.7.3/exe/solr_wrapper:65:in `<top (required)>'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/bin/solr_wrapper:23:in `load'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/bin/solr_wrapper:23:in `<main>'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/bin/ruby_executable_hooks:15:in `eval'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/bin/ruby_executable_hooks:15:in `<main>'

Documentation for configuration

README is missing collection level YML configuration and with_collection options. At least the dir option should be explained what the user should expect to happen with it, what it should contain, etc.

How to add the -force argument?

I am trying to run this in a container but it does not allow running as root user so I need to set the -force flag. How would I do that?

Unexpected error not caught

When I ran solr_wrapper -p 8983 -d solr/config/ --collection_name hydra-development

I got the error

Starting Solr 5.4.1 on port 8983 ...                                                                                   /home/vagrant/.rvm/gems/ruby-2.3.0/gems/ruby-progressbar-1.7.5/lib/ruby-progressbar/progress.rb:58:in `progress=': You can't set the item's current value to be greater than the total. (ProgressBar::InvalidProgressError)
    from /home/vagrant/.rvm/gems/ruby-2.3.0/gems/ruby-progressbar-1.7.5/lib/ruby-progressbar/base.rb:156:in `block in update_progress'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/gems/ruby-progressbar-1.7.5/lib/ruby-progressbar/output.rb:38:in `with_refresh'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/gems/ruby-progressbar-1.7.5/lib/ruby-progressbar/base.rb:155:in `update_progress'
    from /home/vagrant/.rvm/gems/ruby-2.3.0/gems/ruby-progressbar-1.7.5/lib/ruby-progressbar/base.rb:99:in `progress='
    from /home/vagrant/.rvm/gems/ruby-2.3.0/gems/solr_wrapper-0.5.1/lib/solr_wrapper/instance.rb:403:in `block in fetch_with_progressbar'

Duplicate options

In the OptionParser it looks like you are using the -d option twice.

Here:
https://github.com/cbeer/solr_wrapper/blob/master/exe/solr_wrapper#L35

and here:
https://github.com/cbeer/solr_wrapper/blob/master/exe/solr_wrapper#L51

Have a way to call with_collection from the CLI

Add binscripts to this gem for starting and stopping a solr instance

Getting MD5 mismatch

solr-6.6.0.zip.md5: |==================================================================================================| 100% (Time: 00:00:00 )
MD5 missmatch,

I'm using postgres DB in rails 5

Checksum mismatch on solr 9.0.0

See https://github.com/projectblacklight/demo.projectblacklight.org/runs/6500680593?check_suite_focus=true

solr-9.0.0.zip: |==============================================================|



solr-9.0.0.zip.sha512: |=======================================================|


rake aborted!
Checksum mismatch
/home/runner/work/demo.projectblacklight.org/demo.projectblacklight.org/Rakefile:18:in `block in <top (required)>'
/opt/hostedtoolcache/Ruby/2.7.6/x64/bin/bundle:23:in `load'
/opt/hostedtoolcache/Ruby/2.7.6/x64/bin/bundle:23:in `<main>'