robinbetts / toposearch

Search in decentralised systems. Search federation, result aggregation and hypermedia in API to round it all of.

TopoSearch

A federated search service with result aggregation, moderation and feedback mechanism to potentially update the source data. What it means I will try to explain in a moment, but first, a brief disclaimer before I will tell you more about the TopoSearch idea.

The code and ideas developed in this repository are part of my dissertation. I expect it will be very active until Summary 2022. What will happen next will depend on how useful the output will be. If you are interested in similar topics to those used in the project, I'd be happy to chat. If you think my approach makes no sense, I'd love to hear from you - create an issue and let me know where I went wrong.

Keywords or topics explored in this project

Federated Search, Search Results Aggregation, OpenSearch, Decentralised Data Solutions, Hypermedia in RESTful API, Hypermedia as the Engine of Application State (HATEOAS)

---

Why?

• To improve data discoverability in systems where centralisation is not an option.
• To improve the search results based on the relative location of the searcher and the information in the network topology.
• To find a mechanism to send feedback and suggest data improvements from the results back to the data sources.
• To get ready for a more distributed data landscape.

What?

• An API
• A Service
• A Demo (or two)

How?

• The search query will be federated (distributed) in a mesh search network.
• The results will be filtered by the source systems
• The results will be moderated by the network
• The results will be aggregated before presenting them to the user
• The feedback will be pushed back through the network to the source systems to suggest updates.

The Problem

We have increasing amounts of data all around us. Discoverability is a challenge. It is especially true in systems where fine-grained access controls are necessary across organisational boundaries.

Public crawling and indexing are not possible. Centralisation of data in data warehouses and lakes are currently the go-to solution. But that means the data governance has to be centralised. While it works in some organisations, it poses significant challenges in heterogeneous systems and searches between multiple organisations.

The increasing amount of data results in a rising number of possible search results.

Possible Use Cases

(A list of possible use cases will have to be developed to better explain the problem and possible applications of the solution)

Technology

• API: OpenSearch, OpenAPI, REST, HAL, HATEOAS, JSON-LD,
• Back-end: Python, Django
• Front-end: JavaScript, Design System, React

TBC