querqy / querqy-unplugged

⚠️ IMPORTANT: Querqy 5.5 for Solr introduces some breaking changes that will affect you if you are upgrading from an older version and if

• you are using Info Logging, or
• rely on the debug output format, or
• you are using a custom rewriter implementation

See here for the release notes: https://querqy.org/docs/querqy/release-notes.html#major-changes-in-querqy-for-solr-5-5-1

Querqy

Querqy is a framework for query preprocessing in Java-based search engines.

This is the repository for querqy-core, querqy-lucene and querqy-solr. Repositories for further Querqy integrations can be found at:

• https://github.com/querqy/querqy-elasticsearch/ (Querqy for Elasticsearch)
• https://github.com/querqy/querqy-opensearch/ (Querqy for OpenSearch)
• https://github.com/querqy/querqy-unplugged/ (Querqy for rewriting queries outside the search engine)

Documentation and 'Getting started'

Visit docs.querqy.org/querqy/ for detailed documentation.

Please make sure you read the release notes!

Check out Querqy.org for related projects that help you speed up search software development.

Developer channel: Join #querqy on the Relevance & Matching Tech Slack space

License

Querqy is licensed under the Apache License, Version 2.

Contributing

Please read our developer guidelines before contributing.

querqy-unplugged's Issues

Proposition to a solution for boosting score DOWN

Hello,

In the Elasticsearch negative values are not allowed in function weight. Thus, if I understand your solution, you want to use must_not in function filter for boosting DOWN score. However, as you mentioned in your document the DOWN -instructions are not yet supported. Here are your current code:

@Override
public Query convertBoostDown(final BoostQueryDefinition<Query> boostQueryDefinition) {
    return new Query(
            new BoolQuery.Builder()
                    .should(createMatchAllQuery())
                    .mustNot(createBoostQuery(boostQueryDefinition))
                    .build()
    );
}

The above code does not work correctly because it generates a query like this:

 Query : {
    "bool": {
      "filter": [],
      "must": [
   ...
      ],
      "should": [
        {
          "bool": {
            "**must_not**": [
              {
                "function_score": {
                  "boost_mode": "sum",
                  "functions": [
                    {
                      "filter": {
                        "match_all": {}
                      },
                      "weight": 100
                    }
                  ],
                  "query": {
                    "bool": {
                      "boost": 1,
                      "minimum_should_match": "0%",
                      "**must**": [
  ...
                            ],
                            "tie_breaker": 0
                          }
                        }
                      ]
                    }
                  }
                }
              }
            ],
            **"should": [
              {
                "match_all": {}
              }
            ]**
          }
        }
      ]
    }
  }

After investing into this case, I would like to propose a solution for DOWN -instructions with ADDITIVE as following:

@Override
public Query convertBoostDown(final BoostQueryDefinition<Query> boostQueryDefinition) {
    return new Query(
            new BoolQuery.Builder()
                    .must(createBoostDownQuery(boostQueryDefinition))
                    .build()
    );
}

private Query createAdditiveDownScoreQuery(final BoostQueryDefinition<Query> boostQueryDefinition) {
    Query query = boostQueryDefinition.getQuery();
    Query mustQuery = query.bool().must().get(0);

    BoolQuery downQuery = QueryBuilders.bool()
            .mustNot(mustQuery)
            .build();

    return new Query(
            new FunctionScoreQuery.Builder()
                    .query(downQuery._toQuery())
                    .functions(createFunctionScoreByWeight(boostQueryDefinition.getBoost()))
                    .boostMode(FunctionBoostMode.Sum)
                    .build()
    );
}

The new code generates a query like below:

 Query : {
    "bool": {
      "filter": [],
      "must": [
   ...
      ],
      "should": [
        {
          "bool": {
            "**must**": [
              {
                "function_score": {
                  "boost_mode": "sum",
                  "functions": [
                    {
                      "filter": {
                        "match_all": {}
                      },
                      "weight": 100
                    }
                  ],
                  "query": {
                    "bool": {
                      "boost": 1,
                      "minimum_should_match": "0%",
                      "**must_not**": [
  ...
                            ],
                            "tie_breaker": 0
                          }
                        }
                      ]
                    }
                  }
                }
              }
            ]
          }
        }
      ]
    }
  }

Here is an unit test:
Given some documents indexed:

private final List<Product> products = List.of(
        product("1", "iphone", "smartphone", "apple"),
        product("2", "apple", "smartphone", "apple"),
        product("3", "apple smartphone", "smartphone", "apple"),
        product("4", "apple", "case", "apple"),
        product("5", "iphone", "case", "apple"),
        product("6", "samsung", "case", "samsung"),
        product("7", "samsung", "smartphone", "samsung"),
        product("8", "the more you learn; plus you forget", "somewhere", "")
);
final Map<String, Float> fieldScores = Map.of(
        "name", 40.0f,
        "category", 20.0f
);

When search:

@Test
public void testScoreBoostAdditiveWithDOWN() throws IOException {
    final String queryInput = "apple";
    final String filterRule = "apple => \n DOWN(100): smartphone"; 

    List<FnacProduct> results = querqyService.search(
            getIndexName(), filterRule, "", queryInput, fieldScores);

    assertThat(EsPocTools.toIdAndScoreMaps(results)).containsExactlyInAnyOrder(
            idAndScoreMap("2", 40.0), //  40 "apple", "smartphone"
            idAndScoreMap("3", 40.0), // 40 "apple smartphone", "smartphone"
            idAndScoreMap("4", 140.0) // 40 + 100 "apple", "case"
    );
}

Hope that is enough clear and I am looking forward your reply as soon as possible.

Best regards,
Quoc-Anh

Doubt Regarding QuerqyConfig

I'm trying to find a better approach when there are multiple rules. Lets say I have two rules for iphone and jeans, is it better to configure querqy config like below or to have a map with search term as key and rule as value.

cons of below approach is if there are hundreds of rules, it is difficult to manage in a string.
cons of map based approaches are if there is a typo in the search term we can't get the rule from map, also if we need to append multiple rules.

val querqyConfig = QuerqyConfig.builder()
          .commonRules(
              CommonRulesDefinition.builder()
                  .rewriterId("id1")
                  .rules("iphone => \n SYNONYM: apple \n DOWN(50): case \n  \n jeans => \n UP(5): levis")
                  .build(),
          )
          .build()

Dev Question: Integration with Amazon Opensearch Service

Is there any documentation/examples on how querqy-unplugged could work with Amazon OpenSearch - managed search engine? Is it supported?
This statement does not make it very clear:
Querqy-Unplugged currently is able to interact with Elasticsearch via its [Java API Client] and with Solr via its [JSON Query DSL]. However, the library is implemented in a way to minimize efforts to add support for additional approaches (e.g. Elasticsearch Query DSL) and search engines (e.g. OpenSearch).