napoler / solr-php-client Goto Github PK

It should be possible to specify the timeout for a connection, rather than
relying on the default_socket_timeout setting.

When I create the index in phpsolrclient,calling setboost method.
why is not the effect of the settings ？The code as follows：

while($row=mysql_fetch_array($result,MYSQL_ASSOC)){
        $part = new Apache_Solr_Document();
        $part->setFieldBoost("msg_sale_title",2);
            $part->title=$row['title'];
        $part->article=$row['article'];
                $documents[] = $part;
        $count++;
        if($count ==100)
        {
            $solr->addDocuments( $documents );
            $solr->commit();
            $documents = array();
            $count=0;
        }
}

in Service.php

require_once('Apache/Solr/Document.php');
require_once('Apache/Solr/Response.php');


It's much easier to use the library if it's something like:

require_once(dirname(__FILE__) . '/Document.php');
require_once(dirname(__FILE__) . '/Response.php');

What steps will reproduce the problem?
1.Get an account on a Mittwald server
2.Try to do a rawPost to a Solr instance

What is the expected output?
It should just do the post

What do you see instead?
HTTP ERROR: 400 missing content stream

What version of the product are you using? On what operating system?
Linux
php -v
PHP 5.2.6 (cgi) (built: Sep 12 2008 17:23:59)
Copyright (c) 1997-2008 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2008 Zend Technologies
    with the ionCube PHP Loader v3.1.32, Copyright (c) 2002-2007, by ionCube Ltd., and
    with Zend Optimizer v3.3.3, Copyright (c) 1998-2007, by Zend Technologies

Please provide any additional information below.
There is a comment in the php manual:
http://nl.php.net/manual/en/function.stream-context-create.php#90927

And the changelog for the http context options:
http://nl.php.net/manual/en/context.http.php
Notes in the changelog: 5.2.10   The header can now be an numerically indexed 
array .

But the provided patch solves a communication issue between two servers running 
PHP 5.2.6, one Mittwald (client) and a stock Debian server (Solr).

What steps will reproduce the problem?
1. Attempt to verify if faceting information is present, e.g.
isset($response->facet_counts)
2. This returns false always, as facet_counts is generated via that __get()
magic method


What is the expected output? What do you see instead?

I expect that isset($response->facet_counts) would return true if facet
counts are present; this would imply that checking isset parses the data
and returns an appropriate isset() on the parsed data.


What version of the product are you using? On what operating system?

Current head.

Please provide any additional information below.

Note that this affects things like highlighting as well.  The same solution
(implementing __isset(), in which the data is parsed and then checked)
would fix this too.

in Service.php, line 982, the content-type must be : 'application/x-www-
form-urlencoded; charset=UTF-8'

otherwise, POST query with UTF-8 characters return no row.

It would be very useful to be able to query the TermsComponent from
solr-php-client.

A sample patch is included with this report.

I'm currently trying to use SolrPhpClient with the Drupal apachesolr
module. Because of security concerns, we'd like keep this instance's admin
interfaces require authentication, and we're using HTTP Basic for the time
being. We've set the instance and the servlet container up appropriately
and confirmed that it works.

However, there's no place to pass SolrPhpClient the username and password.
I've tried including it as part of the $host variable that gets passed to
__construct() (e.g. 'user:pass@host') as that should work with the
file_get_contents() calls in _sendRawGet() and _sendRawPost. However, this
doesn't work for ping(), as it uses fsockopen() instead.

Stream_context_create in 'ping' function uses method 'HEAD' which is not
allowed (only GET and POST are).

                $context = stream_context_create(
                        array(
                                'http' => array(
                                        'method' => 'HEAD',
                                        'timeout' => $timeout
                                )
                        )
                );

http://nl.php.net/manual/en/context.http.php

Apache Solr has an option to query the index for spelling suggestions. The
suggested patch builds out functionality in solr-php-client to make use of
this feature.

Notable changes are at lines 121, 299, and 979-1028

The public deleteByMultipleIds() function should be added also to the
Apache_Solr_Service_Balancer class. Attached is a patch against Balancer.php

Code:
/**
 * Create and post a delete document based on multiple document IDs.
 *
 * @param array $ids Expected to be utf-8 encoded strings
 * @param boolean $fromPending
 * @param boolean $fromCommitted
 * @param float $timeout Maximum expected duration of the delete operation
on the server (otherwise, will throw a communication exception)
 * @return Apache_Solr_Response
 *
 * @throws Exception If an error occurs during the service call
 */
public function deleteByMultipleIds($ids, $fromPending = true,
$fromCommitted = true, $timeout = 3600)
{
    $service = $this->_selectWriteService();

    do
    {
        try
        {
            return $service->deleteByMultipleIds($ids, $fromPending, $fromCommitted,
$timeout);
        }
        catch (Exception $e)
        {
            if ($e->getCode() != 0) //IF NOT COMMUNICATION ERROR
            {
                throw $e;
            }
        }

        $service = $this->_selectWriteService(true);
    } while ($service);

    return false;
}

Error (when enable error_reporting = E_ALL | E_STRICT in php.ini): 

Warning: fsockopen() [function.fsockopen]: unable to connect to
192.168.1.5:8983 (Connection timed out) in Apache/Solr/Service.php on line 552

Solr PHP Client (Build 03/11/09)

Since there are various changes that make it useful to identify one
specific rev as the desired version to use, and to insure that the correct
one is installed, it would be useful to add svn keywords and make some of
the svn values available as class methods.  This is what Solr itself does
in the admin interface and code to allow ready tracking of the code being run.

e.g. from Solr trunk (1.4)
src/java/org/apache/solr/handler/FieldAnalysisRequestHandler.java

  @Override
  public String getVersion() {
    return "$Revision: 771099 $";
  }

  @Override
  public String getSourceId() {
    return "$Id: FieldAnalysisRequestHandler.java 771099 2009-05-03
18:47:50Z shalin $";
  }

note:

svn propset svn:keywords "Revision Id" Apache/Solr/*.php Apache/Solr/*/*.php

How to deal with multi core with solr-php-client?
I can checkout data with url
"http://192.168.2.58:8080/solr-multicore/core0/select/?q=solr&version=2.2&start=
0&rows=10&indent=on",but
when I use function ping(),it fetch url
"http://192.168.2.58:8080/solr-multicore/core0/admin/ping",does it mean I
have to copy ping.jsp in /solr-multicore/admin/ to
/solr-multicore/core0/admin/?
thanks

At line 534 of Apache/Solr/Service.php, the function 'setNamedListTreatmet'
is named incorrectly.

It should be 'setNamedListTreatment'.

The library  make entities safe when constructing the XML document like:

$multivalue = htmlspecialchars($multivalue, ENT_NOQUOTES, 'UTF-8');

however - this means that I can't search for one of of the entities.  I'm
not sure if this patch will break anything else, but it at least fixes my
keyword searches (using the dismax handler).  

however, this might break full lucene syntax queries that use The symbol &&
in place of "AND".  Maybe that's a reasonable tradeoff?

Apache_Solr_Service::setCreateDocuments(new Custom_Solr_Document)

I think, it is easy to implement this feature

What steps will reproduce the problem?
1. Stop the solr server
2. Call Apache_Solr_Service#ping for that server

What is the expected output? What do you see instead?
Expected: Just returns false without any output.
Actual: Outputs warning
<b>Warning</b>:  fsockopen() [<a
href='function.fsockopen'>function.fsockopen</a>]: unable to connect
 to localhost:8983 in Apache/Solr/Service.php on line
 <b>552</b>

What version of the product are you using? On what operating system?
2009-03-11

Please provide any additional information below.
To fix the problem, just add @ for fsockopen at line 552.
$fp = @fsockopen($this->_host, $this->_port, $errno, $errstr, $timeout);

Thanks for the library. It's wonderful.

See http://issues.apache.org/jira/browse/SOLR-1548

Hi,

Regarding commits, there is an "expungeDeletes" option (which is false by 
default) thanks to which documents deleted using for instance 
Apache_Solr_Service::deleteById() are actually deleted from the index. But 
currently, Apache_Solr_Service::commit() doesn't provide a method for using 
this "expungeDeletes" option.
The doc about the "expungeDeletes" option I'm talking about can be found here: 
http://wiki.apache.org/solr/UpdateXmlMessages#A.22commit.22_and_.22optimize.22

Here is a patch which adds a $expungeDeletes parameter to 
Apache_Solr_Service::commit() ...

Cheers, and thanks for your work...

Olivier

Depending on the query used, deleteByQuery operations can take several minutes 
on a very large 
index. Currently there is no way to pass an expected duration (a timeout) to 
this operation so we're 
at the mercy of the system defaults.

When combined with the Apache_Solr_Service_Balancer this can create a problem 
since delete 
operations that timeout will seem like communication errors - and the class 
will resubmit the 
operation. Causeing undue stress on the Solr service as it tries to satisfy 
them all.

What steps will reproduce the problem?
1. Index a document containing at least one control character

What is the expected output? What do you see instead?
Document fails to index at solr and a XML parsing exception will occur.

I attached a patch to solve this issue.

Hello,

Response.php can throw an Apache_Solr_ParserException at some point but will 
fail because the class has not been included.
Fatal error: Class 'Apache_Solr_ParserException' not found in 
.../solr-php-client/Response.php on line 272

Call Stack:
    0.0008      85492   1. {main}() .../consolidate2.php:0
   96.2110    3901432   2. Apache_Solr_Response->__get() .../solr-php-client/Response.php:0
   96.2110    3901496   3. Apache_Solr_Response->_parseData() .../solr-php-client/Response.php:230

Fix proposal:
- add a require_once(dirname(__FILE__).'ParserException.php'); statement in 
Service.php along the other require_onces that import *Exception.php files.

Take care !

What steps will reproduce the problem?
1. Attempt to reference a field which doesn't exist in an Apache_Solr_Document
2. No checks that the key is present are made; a warning about an invalid index 
is raised

We develop in strict standards and throw ErrorExceptions whenever a warning is 
picked up, so this actually results in an uncaught exception for us.  

What is the expected output? What do you see instead?

I expect a null value.  I get a null + warning (which then throws an exception).

What version of the product are you using? On what operating system?

Current trunk.

Please provide any additional information below.

I've attached a patch if you're interested; the current behaviour may be 
desirable.  The patch simply checks if the array key is there and then returns 
null if it's not.

in Apache/Solr/Service.php

There are a couple places where we experienced problems because this
$fieldBoost variable: 

                if ($fieldBoost !== false)

is NULL, so it passed the test and then when sent to Solr causes failure. 
Either there is a problem when setting the values, or possibly this check
should be something more robust like:

if (is_numeric($fieldBoost))

Hi,

Thanks for the nice Solr PHP library.
I would just change split() calls by explode() in Apache/Solr/Response.php
to avoid warnings.

Solr PHP Client uses file_get_contents(), which requires the allow_url_fopen to 
be enabled; while the use here seems to be quite safe, it is often recommended 
that the feature be entirely disabled so that there is no way the system can be 
tricked into using include() or require() to load content from a foreign URL.

If you would consider including a patch that changes the system to use curl if 
the extension is enabled, and fall back to the current behaviour if it is not, 
then I am happy to write the patch for you; I just want to know that you're 
happy to include the patch in principle (subject to whatever changes you 
require).

Currently there is no method in the client that supports deleting a list of
documents by ID.  One would have to construct a large OR query and pass it
to the method to delete by query. Solr supports delete documents with
multiple ID elements, which shoudl have a little better performance than
the mass query.

I am adding such a method here for our subclassed Solr client for the
Drupal module http://drupal.org/node/561082#comment-2147374

relevant code:
<code>
+  /**
+   * Create and post a delete document based on multiple document IDs.
+   *
+   * @param array $ids Expected to be utf-8 encoded strings
+   * @return Apache_Solr_Response
+   *
+   * @throws Exception If an error occurs during the service call
+   */
+  public function deleteMultipleById($ids) {
+    $rawPost = "<delete>\n";
+    foreach ($ids as $id) {
+      // Escape special xml characters
+      $id = htmlspecialchars($id, ENT_NOQUOTES, 'UTF-8');
+      $rawPost .= '<id>' . $id . "</id>\n";
+    }
+    $rawPost .= '</delete>';
+    return $this->delete($rawPost);
+  }
</code>

Currently the code throws instances of the Exception class. This makes it
hard to distinguish the exception source and cause in from the caller side.

With Exception subclasses you can do things like this:

try
{
  MyAppCall1();
  SomeOtherLibCall4('sfdsdf');
}
catch(MyApp_HttpException $ex)
{
  ...
}
catch(MyApp_Exception $ex)
{
  ...
}
catch(SomeOtherLib_Exception $ex)
{
  ...
}

I've added a diff which introduces the following exception hierarchy:

Exception
  - Apache_Solr_Exception
    - Apache_Solr_HttpTransportException (with HTTP 4xx or 5xx status code)
    - Apache_Solr_InvalidArgumentException
    - Apache_Solr_NoServiceAvailableException
    - Apache_Solr_ParserException

All of these exception classes are subclasses of Exception so it should not
break any existing code.

They should have been subclasses of PEAR_Exception if you'd want to follow
the PEAR guidelines:
http://pear.php.net/manual/en/standards.errors.php
But this project doesn't seem to aim to be a PEAR package so using it might
just introduce an unnecessary dependency on PEAR.

How can i get exact words matches in solr search ?

for e.g When i enter "what is my name" which is already present in database,
I should be getting only "what is my name" but not the others like "what is
...." or "what i am doing rt now ?"

Please help here.

the Solr return invalid json with empty non string value

Index: SolrPhpClient/Apache/Solr/Response.php
===================================================================
--- SolrPhpClient/Apache/Solr/Response.php  (revision 24)
+++ SolrPhpClient/Apache/Solr/Response.php  (working copy)
@@ -247,6 +247,11 @@
        //An alternative would be to use Zend_Json::decode(...)
        $data = json_decode($this->_rawResponse);

+    if($data === null)
+    {
+      // the Solr return invalid json with empty non string value
+      throw new Exception('JSON is not valid.');
+    }
        //if we're configured to collapse single valued arrays or to convert
them to Apache_Solr_Document objects
        //and we have response documents, then try to collapse the values and /
or convert them now
        if (($this->_createDocuments || $this->_collapseSingleValueArrays) &&
isset($data->response) && is_array($data->response->docs))

What steps will reproduce the problem?
1. Index a document using the Apache_Solr_Service::extract() function, passing 
a file, parameters and an Apache_Solr_Document.
2. On line 1027 a call is made to getFieldBoost without a key being specified

What is the expected output? What do you see instead?

The field name should be provided to the getFieldBoost call.

What version of the product are you using? On what operating system?

Current SVN trunk.

Please provide any additional information below.

In using arrays of arrays in parameters, such as facet.field note the
foo[#]=bar

the preg_replace of preg_replace('/%5B(?:[0-9]|[1-9][0-9]+)%5D=/', '=',
$queryString); 

should be

preg_replace('/\[(?:[0-9]|[1-9][0-9]+)\]=/', '=', $queryString);

as the '[' and ']' are not encoded by http_build_query

Solr Cell does not appear to be supported.  I have attached a patch which
provides basic support, but it could do with some improvements
(specifically, it should allow an Apache_Solr_Document to be passed in, and
the URL should be built from that).

This patch is against the source from SolrPhpClient.r22.2009-11-09.tgz

how to set boost with solr-php-client?

I'd like to work on making the constructor more flexible so that it doesn't
assume things like that the servlet will be 'select'.

    /**
     * Constructor. All parameters are optional and will take on default values
     * if not specified.
     *
     * @param string $host
     * @param string $port
     * @param string $path
     */
    public function __construct($host = 'localhost', $port = 8180, $path =
'/solr/')
    {
        $this->setHost($host);
        $this->setPort($port);
        $this->setPath($path);

        $this->_initUrls();
    }

    /**
     * Return a valid http URL given this server's host, port and path and a
provided servlet name
     *
     * @param string $servlet
     * @return string
     */
    protected function _constructUrl($servlet, $params = array())
    {
        ...
        return 'http://' . $this->_host . ':' . $this->_port . $this->_path .
$servlet . $queryString;
    }

    /**
     * Construct the Full URLs for the three servlets we reference
     */


// Note that this doesn't take any params...
    protected function _initUrls()
    {
        //Initialize our full servlet URLs now that we have server information
        ...
// This is more or less hardcoded and immutable. 
        $this->_searchUrl = $this->_constructUrl(self::SEARCH_SERVLET);


        $this->_urlsInited = true;
    }

What steps will reproduce the problem?
1. search() with a URL over a certain size

What is the expected output? What do you see instead?
Instead of a result, Solr serves a comms. error (0) 

What version of the product are you using? On what operating system?
1.2 on RHEL5

Please provide any additional information below.
the search() method in Apache_Solr_Service was presumably exceeding a
maximum size value. A quick goole pointed me at
http://www.nabble.com/limit-on-query-size--td23301642.html which inturn
lead me to look into the sourcecode and I noticed that the client was using
GET. I modified the return call to use POST like so:

return $this->_sendRawPost($this->_searchUrl . $this->_queryDelimiter .
$queryString);

And it seemed to improve. Is there any reason not to use POST by default in
future or make it configurable? 

There is a comment before this line that says http_response_header is set
by file_get_contents but there is no assignment to $http_response_header

If this value is needed it should be assigned. If this value is not needed
null should be used as parameter for the call.

hi,
i am using solr1.4 .i want to  search box with autocomplete in the client 
side.

plz help me

Since Solr (1.4) now offers php output, I decided to get the class to use
it.  In addition, I tried to make it a little flexible so that you can
switch which wt to use.

See the attached path.  It should not affect current behavior in anyway. 
The constant in the class is now SOLR_DEFAULT_WRITER and is set to php
(serialized php didn't work for me, PHP was also a royal PITA due to the
array to object conversion stuff).

If you want to change back to using json, simply call setQueryWriter('json').

I needed to have support for highlighting so I produced a quick patch to do 
what I needed to do.  All I did was have it add the highlighting to the 
document if highlighting was turned on using the extra options parameter in the 
search function.

Don't know if you want to use this or not (since it really is just a quickfix), 
but I figured it would be polite to offer it.

All of the tutorials I have read show that this code should work for me:

$response = $solr->search($query, $start, $limit, $params);

if ($response->getHttpStatus() == 200) {
    if ($response->response->numFound > 0) {
        foreach ($response->response->docs as $doc) {
            echo $doc->title;
        }
    } else {
        echo 'Solr Search Found No Results<br>';
    }
}

However, the $response->response object is null.  If I var_dump($response) 
I can see the rawResponse and all the results it contains with no 
problems.  Trying $response->docs or $response->numFound returns those as 
both null as well.  What's the proper way to access the response from my 
queries?

If not, how should I approach the method of that in phpsolrclient?

Thanks;

I noticed that you have a section on your home page that links to related 
projects.

I've recently completed a PHP library that provides a query builder on top of 
Solr PHP Client. Originally, it was for Drupal (which uses your library), but I 
think I have made it sufficiently generic now that it can be used by other PHP 
projects as well.

If you don't mind adding a link, I'd appreciate it:

http://github.com/technosophos/SolrAPI

Thanks,

Matt Butcher

When using your classes with PHP 5.3, a bunch of E_DEPRECATED errors appears 
due to the use of 
the split() function. 

This is minor, but in Service.php the version is always added to the query
string in public function search():

        $params['version'] = self::SOLR_VERSION;

There are 2 problems with this.  Frist, as best I can tell from the Solr
documentation and the source code of the JSON response writer, the version
parameter is not used by the JSON writer at all.  It only seems to be used
by the XML writer:

http://wiki.apache.org/solr/XMLResponseFormat#version

That page also list valid versions as 2.0, 2.1, and 2.2 (though the source
code for that writer actually just treats anything < 2.1 the same).

but Service.php defines:  


    /**
     * Response version we support
     */
    const SOLR_VERSION = '1.2';

It seems like this 'version' param can be simply removed, or at least
omitted if the request writer type is not 'xml'.

Solr 1.3 has been out for a while 
(http://lucene.apache.org/solr/#15+September+2008+-
+Solr+1.3.0+Available) so I was wondering if you are sticking with JSON?

That message just makes the project look a bit unmaintained/out of date, like 
some other Solr 
clients, when actually this seems to be the most active PHP client for Solr....?

Add support for indexing documents using the Extracting request Handler: 
http://wiki.apache.org/solr/ExtractingRequestHandler

When adding a lot of documents (100.000+) to Solr in one go, in this case a
CLI script used for an initial import, the memory usage gets very high
(over 500M)

I could trace the memory usage to the function _sendRawPost in Service.php.
For every request a new stream context is created, using about as much
memory as the data in the request. With a big data set this starts to add
up very quickly.

I managed to solve the issue by reusing the same context for each request,
and modifing the options to suit the next request. This way the memory
usage remained very stable even for a 600.000 document run.

In \SolrPhpClient\Apache\Solr\Service.php, I found that there is absolutely
NO support on setting up filter (fq=).

Please add support of single/multiple filters support.

Thank you