Features suggestions about ponzu

I wil now open a new issue just for the bleve stuff over at #51 . This issue has too many things in it.

Regarding Searchkit, there is the same functionality in golang by using bleve, which can use boltdb. All Content in the CMS can then be searchable, with full facet searching.
I think this is much better than using elastic with Java dependencies.

https://github.com/blevesearch/bleve

[roscopecoltran]

I was thinking to bridge elasticsearch with a docker container through a docker-compose file for larger dataset to index and query; bleve won't work on a dataset with 10k+ records.

interesting projects using bleve:

• https://github.com/bobinette/papernet (bleve + boltdb)
• https://github.com/dorsha/lennon ( bleve and elastic)

Actually, I am giving a try to qor-admin and ponzu, as i d like to create an easy and quick web admin for a project like limo and maybe re-use this boilerplate for another project called dim; both are using bleve as a full-text search engine.

I think your mistaken about how large a dataset bleve can handle. i have used bleve on huge datasets. It support alias and so can be run on clusters of servers.
Couchbase wrote it and integrated bleve into their full text search.
https://blog.couchbase.com/2016/february/couchbase-4.5-developer-preview-couchbase-fts?facet=blog%2Fblogs.facets%2FAuthor%2FLaurent+Doguin%2FYear%2F2016%2FTags%2Fbleve

i have done 3 projects with bleve and its been solid as a rock. We were indexing 4 Petabytes through it a year ago with alias shards.

Its very easy to make full text searching & faceted searching with bleve on any content.
i think doing projects like limo and dim would be a good match for bleve.

Also the developer behind bleve (Marty) is awesome and very approachable.

[nilslice]

Hi @roscopecoltran -

Thank you for your feedback and suggestions - it's great to see other people enjoying Ponzu and sharing ideas about how to make it better.

Regarding your suggestions:

Searchkit, for some advanced search features

I would be interested in making the CMS admin search feature more complete, moving beyond the current word/phrase exact match implementation. Is the enhancement you're suggesting for search inside the CMS as an admin, or outside as consumer (via some new /search endpoint)?

I think @gedw99's suggestion to consider bleve is probably the right move, since the project is already entirely Go, and I'd like to keep it as such.

vue-admin for more backend oriented widgets

I'm planning to write more documentation on the addons system, which would allow anyone to extend ponzu with virtually any feature and UI they'd like (given some constraints). I don't know if adding Vue.js as a dependency is the best route, but if you try this and find that it works well, please let me know!

admin-on-rest a rest admin boilerplate for content browsing, editing and filtering

Would you clarify what the intended use-case is here? I'm not familiar with this project, but it seems like it would be replacing the UI already available at /admin

An html ui composition interface, eg lib-compose

Interesting project, but I think adding another dependency to simply manage HTML views may be a bit much. The first step to changing any UI code would be to attempt a port of the editor and manager code to use more of the html/template package as discussed in #34.

[nilslice]

@gedw99 - first, nice profile picture. PF is the best. I've played guitar for the majority of my life and David Gilmour is just incredible.

If you have suggestions as to how we could implement Bleve into our search system (or replace it), I would be grateful to hear them. Currently there is only a search handler inside the admin UI to do exact word/phrase matching on the JSON documents in Bolt. The purpose is solely to help admins find content they have added.

I think that Bleve could support a really nice API endpoint for external search requests by outside users.. but I haven't used it and would need to make sure we can prevent indexing of the Bolt buckets that hold sensitive information, and the ones which are hidden as per a Ponzu developers implementation of the item.Hideable interface which prevents content types from being exposed through the content API.

Another consideration would be how Bleve would work with other DB backends. There has been some talk about providing support for other common databases like Postgres & MySQL. Although adding swappable DB backends is probably fairly distant, its good to think about it now when possibly adding more DB-related features.

Am on mobile.. It's maybe 50 lines of code. Each content interface needs to only implement a bleve mapper. It can handle numeric and time ranges too. Then on search there is a standard bleve interface. You probably want a global search that can display a GUI with a few etc for each content item. You can use aliases to ensure it does not clash with other boltdb buckets. Also have a look at storm boltdb in github. Might help everything else That would do it. I can write more when not on mobile with some links !!

…

On Fri, 27 Jan 2017, 18:25 Steve, ***@***.***> wrote: @gedw99 <https://github.com/gedw99> - first, nice profile picture. PF is the best. I've played guitar for the majority of my life and David Gilmour is just incredible. If you have suggestions as to how we could implement Bleve into our search system (or replace it), I would be grateful to hear them. Currently there is only a search handler inside the admin UI to do exact word/phrase matching on the JSON documents in Bolt. The purpose is solely to help admins find content they have added. I think that Bleve could support a really nice API endpoint for external search requests by outside users.. but I haven't used it and would need to make sure we can prevent indexing of the Bolt buckets that hold sensitive information, and the ones which are hidden as per a Ponzu developers implementation of the item.Hideable interface which prevents content types from being exposed through the content API. Another consideration would be how Bleve would work with other DB backends. There has been some talk about providing support for other common databases like Postgres & MySQL. Although adding swappable DB backends is probably fairly distant, its good to think about it now when possibly adding more DB-related features. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATuCwgB_A1kbctcWOf2F35e7FYAUf5JXks5rWhpngaJpZM4LvEDa> .

[roscopecoltran]

@gedw99 I looked at bleve, 1/2 year ago and asked the author about the index scalability of bleve, he did not told me such number (much more 200k).

In one word, that would be awesome :-) I know that couchbase is doing some great things, for eg they did a couple of cool things for distributed deep learning like elastic-thoughts.

I will take a couple of hours to read the article and make some tests.

I saw the following interesting project:

• https://github.com/couchbase/cbft

Cheers,
Richard

Checked the new issue I created. Should help slot.

…

On Sat, 28 Jan 2017, 15:15 roscopecoltran, ***@***.***> wrote: @gedw99 <https://github.com/gedw99> I looked at bleve, 1/2 year ago and asked the author about the index scalability of bleve, he did not told me such number (much more 200k). In one word, that would be awesome :-) I know that *couchbase* is doing some great things, for eg they did a couple of cool things for distributed deep learning like elastic-thoughts <https://github.com/tleyden/elastic-thought>. I will take a couple of hours to read the article and make some tests. I saw the following interesting project: - https://github.com/couchbase/cbft Cheers, Richard — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATuCwnzrrsqGrN0O_ushKgDtfsMaSlRGks5rWz9agaJpZM4LvEDa> .

[nilslice]

I need to familiarize myself with Bleve and check out storm. Will report back on #51. Thank you!

I am curious myself because the design of Ponzi looks pretty awesome. It's really similar to something I started to build, but now I am getting on board with adding some of my ideas to ponzu. I will raise them as feature discussion / issues if that's ok. See what you think... The technical aspects are pretty easy thanks to the work and maturity that Marty has put into bleve

…

On Sat, 28 Jan 2017, 21:54 Steve, ***@***.***> wrote: I need to familiarize myself with Bleve and check out storm. Will report back on #51 <#51>. Thank you! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATuCwjfDRGNanl9vT5oLDWjH2CgO8Z7tks5rW50PgaJpZM4LvEDa> .

[nilslice]

Thank you, I would be very happy to have contributions and get more people involved. I think the design already lends itself well to the approach you mentioned where Content items implement a mapper interface.

If you look at the 'system/item' package, you'll see a number of other interfaces that are exported and that Item implements by default (most as no-ops, which the intention of them being overrriden).

Do you think we could extend Item to implement a default mapping interface so the Ponzu developer wouldn't need to do this themselves in their content file?

yes i saw that way the interface works. I really liked it. Very extendable design and innovative. I answered on the proper page so we keep it all together: #51

…

On Sat, 28 Jan 2017, 22:26 Steve, ***@***.***> wrote: Thank you, I would be very happy to have contributions and get more people involved. I think the design already lends itself well to the approach you mentioned where Content items implement a mapper interface. If you look at the 'system/item' package, you'll see a number of other interfaces that are exported and that Item implements by default (most as no-ops, which the intention of them being overrriden). Do you think we could extend Item to implement a default mapping interface so the Ponzu developer wouldn't need to do this themselves in their content file? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATuCwvxHtcHzBNX2qPx78bE5tbqrfidhks5rW6RhgaJpZM4LvEDa> .

[nilslice]

@roscopecoltran - since it seems like the Bleve / full text search is the most interesting to us on this thread, I am going to close out this issue and we can carry out the conversation on #51.

If you would like to discuss the other ideas you suggested, please read my responses and we can chat in Gopher Slack on the ponzu channel (join here: https://invite.slack.golangbridge.org/)

Thanks!