massiveart / massivesearchbundle Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
Add dispatching of a PRE_DEINDEX event in SearchManager::deindex
to giv a possibility to react on deindexing an object.
When executing the index rebuild command on ~ 10.000 Contacts:
app/console massive:search:index:rebuild
a OutOfMemoryException accurs:
[Symfony\Component\Debug\Exception\OutOfMemoryException]
Error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 65552 bytes)
Hi there. i'm trying to create new adapter, in this case to make it work with Algolia. i've created the services but problem is when i set the new adapter in the config like
# MassiveSearch Configuration
massive_search:
adapter: algolia
When i do this i get an error like
In EnumNode.php line 46:
The value "algolia" is not allowed for path "massive_search.adapter". Permissible values: "zend_lucene", "elastic", "test"
so, is there any way to add another adapter or to modify that list of possible values?
Thanks in advance.
When i want to extend the e.g. SuluContactBundle:Account Entity with SuluContactExtensionBundle:Account by overriding the massive-search/Account.xml file, it does not have any impact.
The override attempt:
sulu/SuluContactExtensionBundle#18
Resulting from the following discussion:
sulu/sulu#1349
If you have 2 projects on the same server and reindex both at the same time the first project will lock the second project because the same directory (sys temp dir) is used:
I scratched my head around why my search isn't filtered by locale. I'm using DoctrineExtensions Translatable extension and I made a custom reindex provider because I see this is used in ReindexCommand
to extract translations for each object.
Well, I didn't expect that calling $this->searchManager->index($object, $locale);
ignores $locale
because index()
method takes only one argument. So, this entire block of code is in fact useless:
$locales = [null];
if ($provider instanceof LocalizedReindexProviderInterface) {
$locales = $provider->getLocalesForObject($object);
}
try {
foreach ($locales as $locale) {
if (null !== $locale) {
$object = $provider->translateObject($object, $locale);
}
$this->searchManager->index($object, $locale);
}
This all has no effect on entitie's indexed locale because real locale is always extracted from the entity in ObjectToDocumentConverter
.
I actually ended up using deprecated call to entity repository that returns all objects that I need to index even though it's probably supposed to throw an error with trigger_error()
which is silenced with @
so I doesn't.
I think the entire thing with creating custom providers is useful but now it's too tightly coupled with the entity structure. It has to contain its locale somewhere which is exactly how DoctrineExtensions Translatable extension doesn't work. The LocalizedReindexProviderInterface
could work for me when reindexing even though I'd appreciate if providers could check whether they support particular entity class.
Maybe If I could use the same LocalizedReindexProviderInterface
provider in IndexListener
to grab all translations for particular entity and index them instead of relying on locale
property? That would help me a lot.
Instead of unStored
, stored
, text
, etc. Use store
and index
.
Appplies to the Index::INDEX_*
constants.
Warning: Invalid argument supplied for foreach() in /home/daniel/www/sulu-cmf/sulu-standard/vendor/massive/search-bundle/Search/Adapter/ZendLuceneAdapter.php line 62
Hi!
I am trying to add a new field to the Document object by a custom factory:
My factory class:
namespace SearchBundle;
use Massive\Bundle\SearchBundle\Search\Factory as BaseFactory;
class Factory extends BaseFactory
{
public function createDocument()
{
$document = new SearchBundle\Document();
return $document;
}
}
My Document class:
namespace SearchBundle;
use Massive\Bundle\SearchBundle\Search\Document as BaseDocument;
class Document extends BaseDocument {
protected $editUrl;
public function getEditUrl()
{
return $this->editUrl;
}
public function setEditUrl($editUrl)
{
$this->editUrl = $editUrl;
}
public function jsonSerialize()
{
return [
'id' => $this->id,
'title' => $this->title,
'description' => $this->description,
'class' => $this->class,
'url' => $this->url,
'image_url' => $this->imageUrl,
'locale' => $this->locale,
'edit_url' => $this->editUrl,
];
}
My config:
massive_search:
persistence:
doctrine_orm:
enabled: true
adapter: elastic
metadata:
prefix: %database_name%
services:
factory: factory_service
services:
factory_service:
class: SearchBundle\Factory
In the onPreIndex event I call this
$document->setEditUrl('test-edit-url')
But as I see the index still does not include the edit_url field.
Did i miss something?
Thanks!
_default
context).The latest Elastic Search release supported by elastic/[email protected] [1] is Elastic Search 2.4.x to which all support ended on 28.02.2018 [2].
It looks like the elastic feature set used by MassiveSearchBundle is fully supported by Elastic Search 6 and therefore elastic/elasticsearch-php should be upgraded to the 6.0 branch.
[1] https://github.com/elastic/elasticsearch-php/tree/master#version-matrix
[2] https://www.elastic.co/support/eol
Before making the next release we should review the documentation and ensure that it is accurate.
If you define adapter
as elastic, and do not define adaptres: elastic
then you get an undefined index error,
I'm playing around with this nice bundle and noticed that the image-fields of my search-index were empty even if I defined them. After digging through the code, it seems to me, that the mapping is missing.
With the following additions it works as expected:
diff --git a/Search/Metadata/Driver/XmlDriver.php b/Search/Metadata/Driver/XmlDriver.php
index bfc3f76..0e7f044 100644
--- a/Search/Metadata/Driver/XmlDriver.php
+++ b/Search/Metadata/Driver/XmlDriver.php
@@ -118,6 +118,7 @@ class XmlDriver extends AbstractFileDriver implements DriverInterface
$indexMetadata->setTitleField($mapping['title']);
$indexMetadata->setUrlField($mapping['url']);
$indexMetadata->setDescriptionField($mapping['description']);
+ $indexMetadata->setImageUrlField($mapping['image']);
foreach ($mapping['fields'] as $fieldName => $fieldData) {
$indexMetadata->addFieldMapping($fieldName, $fieldData);
@@ -148,6 +149,9 @@ class XmlDriver extends AbstractFileDriver implements DriverInterface
$urlField = $this->getMapping($mapping, 'url');
$indexMapping['url'] = $urlField;
+ $imageField = $this->getMapping($mapping, 'image');
+ $indexMapping['image'] = $imageField;
+
$descriptionField = $this->getMapping($mapping, 'description');
$indexMapping['description'] = $descriptionField;
Am I missing something or is this feature really missing in current master?
At the moment the queries are created as shown in the images below.
By dispatching an event that will create the queries, the bundle would become more flexible and it would for example allow to implement fuzzy queries, or anything more advanced.
Another way to allow more flexibility is to extract the search method of the adapter into two stages, creating the query prepareSearch
and running it executeQuery
. Extending the massive_search.adapter
would be much easier that way.
In order to display search results and link to documents we need to map these properties.
I just tried to install your promising package on a symfony 3.0.4 installation and sadly failed because of the dependency on "symfony/property-access ~2.4". Are there any plans to support sf3 in the near future?
Thanks for your awesome work!
The metadata is currently not cached.
The sorting of the fields doesn't work as expected. Seems like the mapping of the indexed fields should look like followed:
"type" : "string",
"index": "not_analyzed"
Further information can be found on the elasticsearch documentation:
https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-intro.html
https://www.elastic.co/guide/en/elasticsearch/guide/current/_sorting.html
In version 8 of elasticsearch/elasticsearch
the namespace changed from Elasticsearch
to Elastic\Elasticsearch
(https://github.com/elastic/elasticsearch-php/blob/main/BREAKING_CHANGES.md). Integrating this bundle into a Symfony 6 project I've noticed that there is no way to change massive_search.search.adapter.elastic.client.class
(https://github.com/massiveart/MassiveSearchBundle/blob/2.8/Resources/config/adapter_elastic.xml#L7).
The issue is that in loads the adapter_elastic.xml
before checking the existence of that class and loading the xml will always reset it:
https://github.com/massiveart/MassiveSearchBundle/blob/2.8/DependencyInjection/MassiveSearchExtension.php#L79-L81
Hello,
I use https://github.com/sulu/sulu witch implement the massive-search with ZendLucene adapter with load-balancing.
The indexes directory was shared between servers.
Thanks for help
Here the stack trace :
ZendSearch\Lucene\Exception\InvalidFileFormatException:
Wrong TermInfoIndexFile file format
at vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Index/DictionaryLoader.php:52
at ZendSearch\Lucene\Index\DictionaryLoader::load('�' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '��_8s.fdx' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '�_8s.fdt' . "\0" . '' . "\0" . '')
(vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Index/SegmentInfo.php:774)
at ZendSearch\Lucene\Index\SegmentInfo->_loadDictionaryIndex()
(vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Index/SegmentInfo.php:803)
at ZendSearch\Lucene\Index\SegmentInfo->getTermInfo(object(Term))
(vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Index.php:898)
at ZendSearch\Lucene\Index->hasTerm(object(Term))
(vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Search/Query/Preprocessing/Term.php:120)
at ZendSearch\Lucene\Search\Query\Preprocessing\Term->rewrite(object(Index))
(vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Search/Query/Boolean.php:132)
at ZendSearch\Lucene\Search\Query\Boolean->rewrite(object(Index))
(vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Index.php:669)
at ZendSearch\Lucene\Index->find(object(Boolean))
(vendor/massive/search-bundle/Search/Adapter/ZendLuceneAdapter.php:390)
at Massive\Bundle\SearchBundle\Search\Adapter\ZendLuceneAdapter->removeExisting(object(Index), object(Document))
(vendor/massive/search-bundle/Search/Adapter/ZendLuceneAdapter.php:113)
at Massive\Bundle\SearchBundle\Search\Adapter\ZendLuceneAdapter->index(object(Document), 'massive_media-fr-i18n')
(vendor/massive/search-bundle/Search/SearchManager.php:178)
at Massive\Bundle\SearchBundle\Search\SearchManager->index(object(FileVersionMeta))
(vendor/massive/search-bundle/Search/EventListener/IndexListener.php:39)
at Massive\Bundle\SearchBundle\Search\EventListener\IndexListener->onIndex(object(IndexEvent), 'massive_search.index', object(TraceableEventDispatcher))
(vendor/symfony/event-dispatcher/Debug/WrappedListener.php:117)
at Symfony\Component\EventDispatcher\Debug\WrappedListener->__invoke(object(IndexEvent), 'massive_search.index', object(TraceableEventDispatcher))
(vendor/symfony/event-dispatcher/EventDispatcher.php:230)
at Symfony\Component\EventDispatcher\EventDispatcher->callListeners(array(object(WrappedListener)), 'massive_search.index', object(IndexEvent))
(vendor/symfony/event-dispatcher/EventDispatcher.php:59)
at Symfony\Component\EventDispatcher\EventDispatcher->dispatch(object(IndexEvent), 'massive_search.index')
(vendor/symfony/event-dispatcher/Debug/TraceableEventDispatcher.php:154)
at Symfony\Component\EventDispatcher\Debug\TraceableEventDispatcher->dispatch(object(IndexEvent), 'massive_search.index')
(vendor/massive/search-bundle/Search/EventSubscriber/DoctrineOrmSubscriber.php:77)
at Massive\Bundle\SearchBundle\Search\EventSubscriber\DoctrineOrmSubscriber->indexEntity(object(FileVersionMeta))
(vendor/massive/search-bundle/Search/EventSubscriber/DoctrineOrmSubscriber.php:51)
at Massive\Bundle\SearchBundle\Search\EventSubscriber\DoctrineOrmSubscriber->postPersist(object(LifecycleEventArgs))
(vendor/symfony/doctrine-bridge/ContainerAwareEventManager.php:68)
at Symfony\Bridge\Doctrine\ContainerAwareEventManager->dispatchEvent('postPersist', object(LifecycleEventArgs))
(vendor/doctrine/orm/lib/Doctrine/ORM/Event/ListenersInvoker.php:99)
at Doctrine\ORM\Event\ListenersInvoker->invoke(object(ClassMetadata), 'postPersist', object(FileVersionMeta), object(LifecycleEventArgs), 4)
(vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:1150)
at Doctrine\ORM\UnitOfWork->executeInserts(object(ClassMetadata))
(vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:415)
at Doctrine\ORM\UnitOfWork->commit(null)
(vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:388)
at Doctrine\ORM\EntityManager->flush(null)
(var/cache/admin/dev/ContainerYLhkVOa/EntityManager_9a5be93.php:136)
at ContainerYLhkVOa\EntityManager_9a5be93->flush()
(vendor/sulu/sulu/src/Sulu/Bundle/MediaBundle/Media/Manager/MediaManager.php:576)
at Sulu\Bundle\MediaBundle\Media\Manager\MediaManager->createMedia(array('locale' => 'fr', 'collection' => '20', 'contentLanguages' => array(), 'publishLanguages' => array(), 'title' => 'transition énergétique', 'formats' => array(), 'id' => null, 'storageOptions' => array('segment' => '05', 'fileName' => 'transition-energetique.svg'), 'name' => 'transition énergétique.svg', 'size' => 2700, 'mimeType' => 'image/svg', 'properties' => array('width' => 72, 'height' => 73), 'type' => array('id' => 2)), object(User))
(vendor/sulu/sulu/src/Sulu/Bundle/MediaBundle/Media/Manager/MediaManager.php:527)
at Sulu\Bundle\MediaBundle\Media\Manager\MediaManager->buildData(object(UploadedFile), array('locale' => 'fr', 'collection' => '20', 'contentLanguages' => array(), 'publishLanguages' => array(), 'title' => 'transition énergétique', 'formats' => array(), 'id' => null, 'storageOptions' => array('segment' => '05', 'fileName' => 'transition-energetique.svg'), 'name' => 'transition énergétique.svg', 'size' => 2700, 'mimeType' => 'image/svg', 'properties' => array('width' => 72, 'height' => 73), 'type' => array('id' => 2)), object(User))
(vendor/sulu/sulu/src/Sulu/Bundle/MediaBundle/Media/Manager/MediaManager.php:322)
at Sulu\Bundle\MediaBundle\Media\Manager\MediaManager->save(object(UploadedFile), array('locale' => 'fr', 'collection' => '20', 'contentLanguages' => array(), 'publishLanguages' => array(), 'title' => 'transition énergétique', 'formats' => array(), 'id' => null), 1)
(vendor/sulu/sulu/src/Sulu/Bundle/MediaBundle/Controller/MediaController.php:533)
at Sulu\Bundle\MediaBundle\Controller\MediaController->saveEntity(null, object(Request))
(vendor/sulu/sulu/src/Sulu/Bundle/MediaBundle/Controller/MediaController.php:414)
at Sulu\Bundle\MediaBundle\Controller\MediaController->postAction(object(Request))
(vendor/symfony/http-kernel/HttpKernel.php:152)
at Symfony\Component\HttpKernel\HttpKernel->handleRaw(object(Request), 1)
(vendor/symfony/http-kernel/HttpKernel.php:74)
at Symfony\Component\HttpKernel\HttpKernel->handle(object(Request), 1, true)
(vendor/symfony/http-kernel/Kernel.php:202)
at Symfony\Component\HttpKernel\Kernel->handle(object(Request))
(public/index.php:66)
Hey,
i want to use this bundle outside of Symfony - in that case for ZendFramework.
Is there an easy way to work without the whole symfony mvc in use?
regards thx
Currently localized index names are inferred from existing indexes, however if the indexes do not exist then no indexes are inferreed.
Adapters tend to perform global searches when no indexes are given.
This needs to be fixed.
massive:search:reindex
would be a better command name than massive:search:index:rebuild
.
Also it makes for a better namespace in Search\ReIndex\..
Note this is a BC breaking change.
When you clone the repo and try to run only the Zend Lucene tests (vendor/bin/behat --suite=zend_lucene
) you end up with many same errors:
There is no search mapping for object with class "Massive\Bundle\SearchBundle\Tests\Resources\TestBundle\Entity\Product" (Massive\Bundle\SearchBundle\Search\Exception\MetadataNotFoundException)
This happens because MassiveSearchExtension on line 123 checks whether the massive-search
exists, which doesn't. Also, directory Entity
doesn't exist.
The traviscl tests run fine because they're executed in order where vendor/bin/phpunit --coverage-text
is run first. As a side effect it creates also massive-search
and Entity
directories and doesn't remove them when it finishes (which it probably should).
To see how it fails run:
$ git clone https://github.com/massiveart/MassiveSearchBundle.git
$ cd MassiveSearchBundle
$ composer install
$ vendor/bin/behat --suite=zend_lucene
To execute the tests properly run:
$ git clone https://github.com/massiveart/MassiveSearchBundle.git
$ cd MassiveSearchBundle
$ composer install
$ mkdir Tests/Resources/TestBundle/Resources/config/massive-search
$ mkdir Tests/Resources/TestBundle/Entity
$ vendor/bin/behat --suite=zend_lucene
If other indexes exist in Elastic search and a global search is performed, then these external items will also be found.
I use Sulu for a couple of weeks now and I tried to index an entity of my project in App\Entity (App\Entity\Commune).
I put the Commune.xml in src/Resources/config/massive-search/Commune.xml
But when I execute php bin/console massive:search:reindex, nothing happens.
It seems that the explored paths for metadatas are only from bundle :
$container->getParameter('kernel.bundles')
And so when I dump the paths form getBundleMappingPaths
(DependencyInjection/MassiveSearchExtension.php), I only get this :
array:3 [
"Sulu\Bundle\ContactBundle\Entity" => "/Users/me/Documents/WebProjets/myproject/vendor/sulu/sulu/src/Sulu/Bundle/ContactBundle/Resources/config/massive-search"
"Sulu\Bundle\MediaBundle\Entity" => "/Users/me/Documents/WebProjets/myproject/vendor/sulu/sulu/src/Sulu/Bundle/MediaBundle/Resources/config/massive-search"
"Sulu\Bundle\CategoryBundle\Entity" => "/Users/me/Documents/WebProjets/myproject/vendor/sulu/sulu/src/Sulu/Bundle/CategoryBundle/Resources/config/massive-search"
]
Is there a way to add AppBundle or is it a limitation which obliges to use a bundle to index entities ?
Thank you very much :)
Add command to remove unused indexes.
It should remove indexes which are managed by massive search including localized indexes.
Support document load event.
Document URL generators listen to a massive_search.document_post_load
event (for example) and can do things such as setting the URL for a document after it has been loaded from the search index.
There would be two:
RouteReferrersInterface
for example..The Document and the QueryHit are two differnet classes. One is about storing the data, the other is, maybe, about the search results.
Currently the document has the methods getTitle
, getDescription
and getUrl
. Which I think would better belong to QueryHit
.
But having hard-methods in QueryHit
is quite limiting to extensiblility, so they should instead be registered dynamic fields.
There is no XSD file for the search mapping defined in XML. It should definitely be there, because the IDE can support the developer better, and there is an additional validation.
I noticed a difference between the docs source and the rendered docs. For example, directories for the mapping:
https://github.com/massiveart/MassiveSearchBundle/blob/2.8/Resources/docs/mapping.rst#full-example
https://massivesearchbundle.readthedocs.io/en/latest/mapping.html#full-example
Source: "This file MUST be located in config/massive-search
/YourBundle/Resources/config/massive-search
"
Docs: "This file MUST be located in YourBundle/Resources/config/massive-search
"
Clicking at "edit in Github" results in a 404 too. Looks like an error to me.
In order to support multiple languages, one option would be to have a listener which modifies the name of the index name depending on the locale.
Introduce support for offest and max results in order to support pagination.
Apply the CS fixer and fix all the headers (should not read part of the Sulu CMS
)
Currently an event is fired instructing listeners to reindex, all of the responsiblity is on the "client" side, including logging progress, filtering etc.
It would be better to have the "clients" simply provide the entities which need to be reindexed. This however presents some issues -
Perhaps the best option would be to have a ReindexProvider
class:
class ReindexProviderInterface
{
public function provide($offset, $limit, $classFqn = null)
{
}
}
Please update the site to the latest GIT version https://massivesearchbundle.readthedocs.io/en/latest/
I had a hard time to figure how the Elastic Search configuration is because the latest github Docpage is not used for your doc site linked from the readme. Lost some hours because of you :P Glad I asked Alex about it!
In use with Sulu v2.3.3, i tried to add an additional, non-searchable field to an index by listening to the PreIndexEvent and adding the field manually using Massive\Bundle\SearchBundle\Search\Factory\Factory
like so:
$document->addField($this->factory->createField(
'my_additional_field',
$value,
Field::TYPE_STRING,
true,
false
));
Although the $indexed
-param of $factory::createField()
is set to false
, the additionally created field is still searchable within the index.
According to feedback on this topic in the Sulu Slack Channel, the root of the problem is to be found in ElasticSearchAdapter::prepareSearch(SearchQuery $searchQuery), as the query_string
-array should contain an additional fields array containing the names of all fields where $indexed
is true
.
The unit tests for the adapters are quite basic and should be extended with pagination and sorting.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.