Feature Discussion: Full Text Search about ponzu

Question was:
Do you think we could extend Item to implement a default mapping interface so the Ponzu developer wouldn't need to do this themselves in their content file?

Answer:
Yes I think you can use a default mapping on the types that users generate. That's exactly what I was also thinking. But you need to provide a few more specific types that bleve needs as first class types in the core of ponzu that users specify in their generated types.

• numeric range
• datatime Range
• maybe one or two other. Dont have time to look right now

These are needed for bleve in more complex search cases which you will hit.
The common GUI widgets can then be fully aware of these first class types
as well as bleve. Numeric range and datetiem range always need specific widgets !
I think it closes off lots of potential bugs too.

Here is another example of similar thinking in terms of first class types to provide data sync. Might be good idea.
I have raised it as a new FeatureIssue here: #53

[nilslice]

@gedw99 -

Although it's been some time since we discussed the full text search integration via Bleve, I have an initial version in the ponzu-dev branch. If you'd like to check it out and let me know if:

1. I have any glaring mistakes
2. You have improvements or suggestions

I would greatly appreciate it! I'm hoping to merge a cleaned up version of that branch early next week.

All of the search-related code is in this compare: https://github.com/ponzu-cms/ponzu/compare/ponzu-dev

@nilslice thankyou. your really diligent. I figured it would be forgotten.

So i had a look at the code. It works, but is missing a ton of things needed IMHO.

If you want a nice clean example i recommend papernet:
https://github.com/bobinette/papernet/blob/master/bleve/paper.go

• see the way boltdb and bleve are cleanly seperated

https://github.com/bobinette/papernet/blob/master/bleve/paper.go#L57

• encapsulation of the search terms and search results. nice :)

https://github.com/bobinette/papernet/blob/master/bleve/mapping.json

• Nice and easy way to add mapping.

He has wrapped all dependencies like boltDB and bleve nicely. Its elegantly done and keep suse out of trouble.
I also work with it a bit.
You could take the wrappers and then your good !!

We should be clear about goals and scoping too. My scope would be:

• BoltDB storage only for now. We can use Alias's later to allow multiple servers.
• Clear delineation between search storage and business data storage because they have different HA patterns later. Also in the workflows you want an emergent messaging style archi so that you can setup as many stores as you need for different things. We have 2 now but i bet you might add minio later. Not suggesting you need to add NATS or that complexity.
• Facets support with a data driven GUI. So when new terms are added you see it in the global search page when you type any search term.
• golang structure to support bleve tagging ? Would be neat i think. Seen it before.
• 18n from the start of course

i saw the changes with regards to the main architectural points in the issues.
Really looks great, and more useful to a broader audience i feel now. I have to give it a whirl again.
I see the new examples too :)
I still feel that having examples in a folder outside the ponzu repo using a .sh or Makefile is needed because:

• Show how to make your own apps to drive the thing where you want your app to be outside the ponzu repo.
• Is a good integration test layer for contributors. It tests the restAPI

[nilslice]

Thank you for the thoughtful response -- I agree there is much more to do to take full advantage of Bleve's features. From the Papernet example, I see how I should wrap Bleve's API a bit more - thank you for sharing that example.

My main goal with this integration at the moment is to enable a user to get efficient search from an API endpoint without requiring much (or any) configuration by the Ponzu user to enable phrase matching against indexed content. The use case for Ponzu is outside client conducting a search via HTTP like /api/search?type=Song&q=Us+and+Them

In a later release, I think doing more of a Admin-focused feature set would be cool, plus adding better support to the API for bleve's numeric and date range queries (which I'm a bit confused by to be honest). When you refer to a Data driven GUI using Facets, I don't really understand what that would look like - I apologize for my rather naive abilities and familiarity with Bleve (and search in general)!

From a Ponzu-user's perspective, I want to know if search indexing should be done by default (ALL your content types are indexed) with the option to disable indexing, OR if it should be enabled on a type-by-type basis. Already the search results via http api endpoint will respect the other rules (item.Hideable & item.Omittable) enforced by a Ponzu user, but I could see performance reasons to not index every content type.. so I'm not sure if auto-enabling is the best way to go..

I agree with your reasoning for external folders for examples to run tests against and demonstrate features -- and I would like that to be a community effort if possible. I will add a repo within the Ponzu organization to host these projects.

I really appreciate you asking me more...
i just wanted to point you to a lib that shows how to use it better. Hope this is ok.

About bleve and the numeric and datatime range queries, they are pretty simple and just more terms. With range queries like tis you need to precalcuate your min and max and then pick the bracket in the middle. Liek "low, medium and high" in their examples. You can find the bottom and top by dong a quick bolt search on each index mapping using range terms. I guess thats a little weird but its how it works and is the only place i needed to add to bleve to get it working.

The word "terms" and "facets" can be confusing for sure. they are the same thing in a way.
Terms are what you use to index data against with different term types.
Facets is a generic industry word for using those terms in the GUI.
At least thats how i have arrived at this understanding on my own. I am probably not absolutely correct :)
BTW marty on the bleve google groups is super supportive. I asked him lots of newbie questios and as long as i put in the effort to get half way he was really helpful.

I think a good example is Amazon shopping to talk around this.
The user starts at the free text search field on the home page.
The user types "tv", and get a result set of TVs in a list view.
Once the user picks one you are at that TV's item page (which you got from bolt), and on the left the facets show up (which came from bleve), showing the different facets like manufacturer, screen type, screen size, vesa plate, etc, etc.
The facets are pre "checked" (as check boxes and sliders) to match the TV your current looking at BTW:)
Then you can use the facets to widen or narrow your search.
Its very powerful and intuitive. I guess i can say its intuitive since its amazon and so designed for every man and his dog to be able to use :)

So maybe you can see that this is all data driven. The bleve terms mapping that occurred when you indexed a Record (or document has they call it in Bleve) determines the facets. Bleve actually returns back to you the facets ( it calls them facets in the JSON data and golang structs if i recall ) and all you have to do is display it like Amazon does as a tree.
Then you check box or range slide facets and send the query back in.
there is no logic from the programmer to implement. Its very cool and i hope you agree.
The bleve search name space is where all the action is and it does the request response cycle fully for you, and all you need to do is parse it. I tend to encapsulate it it and not use the JSON.
Mostly because i find separating my binding and transport to me better architecturally, but thats not appropriate for this context.

Hopefully what i wrote above helps ??

You asked if i think search indexing should be done by default.
I think YES :) Just because your CMS is designed to show you all your data types, and so why not index all that data ? Then you can have a generic search page just like Amazon. But Materials Design of course :) I saw one done with material design in Dart and it was the sexy.
Its also very easy thanks to Bleve being so well designed so why not .
I always set things up so that any mutation to the DB hits the DB and hits bleve. Just that simple.

You also ask if once a user gets to a type should we show the search. Well once you get to a Type, the search facets can be found that its is a member of, just like the Amazon example. SO yes i would say :)

Also with bleve you can scale it horizontally on many servers. They call this Alias's, and its more work but generic. Just so you know its there. I am using mini allot to make any data storage system HA in this way at the moment as a bit of an experiment. Mini has a dsync repo that does one thing and one thing only - Its does record locking across the cluster. This allows minio to scale up to 16 servers, and have full linearizability. https://github.com/minio/dsync
Just mentioning this in case your curious. Its damn useful because you can use it in a myriad of ways. We can talk about that in another discussion if you want.

Example folder. Cool. Its a hack that way i did it but works well enough because its a full tear down and rebuilt so no side effects. I used sh scripts, but i am guessing you will make it more golang ish.

BTW i checked out buffalo that you wrote about in another issue.
https://github.com/gobuffalo
It seems to be a good match with ponzu. They use different DB storgae. Ponzu uses boltdb and buffalo uses the sqlx ones (sqlite, etc etc). But there feels to be some synergy there in that they both are frameworks.

I typically use bleve with any DB, and use bleve for search and when i get down to a record type fall back to using basic sqlish queries (like " select x from y where firstname = z " type of thing). The two work well i find for almost anything.

[nilslice]

@gedw99 -

Thank you so much for all your input and assistance getting this feature to a usable point! Although I didn't implement it to your full recommendation, I think it is a place where the majority of Ponzu users will see the benefits and once more interest is present, we can revisit the topics discussed to further the functionality.

I am going to close this issue now, since #116 is merged. However, please keep ideas logged and let me know if you think of other improvements to the existing implementation.

Steve

[nilslice]

@gedw99

Example folder. Cool. Its a hack that way i did it but works well enough because its a full tear down and rebuilt so no side effects. I used sh scripts, but i am guessing you will make it more golang ish.

I added an examples repository, if you'd like to share any of your examples.
See it here:
https://github.com/ponzu-cms/examples