Data Source filtering syntax about symphonycms

[brendo]

I still don't see this as Datasource Editor, it's more Field filtering, but anyway.

But in this case we should make sure that all core fields allow negations (e. g. not: {$title}).

Agreed, consistency is best. We have implemented regexp and not-regexp on all core fields, so I see this as a slight extension.

Furthermore it would be nice if we could improve variable replacements.

I like the ? idea, but I think it could be extended more so that the syntax is identical to PHP5.3. From the manual

Since PHP 5.3, it is possible to leave out the middle part of the ternary operator. Expression expr1 ?: expr3 returns expr1 if expr1 evaluates to TRUE, and expr3 otherwise.

So your above example would be {$title?:my-default-value}. It's subtle, but if we're going to change it, we may as we align it to something :D

[allen]

The concept of enumeration and ternary are quite different. Enumeration "hops" in turn from one value to another in a series until true. Ternary on the other hand returns value one or value two depending on a particular boolean condition.

In your specific case, the enumeration syntax can produce the same result as your ternary syntax. But doing the reverse, ternary syntax does not allow the ability to enumerate beyond two values, such as: {$param-1:$param-2:$param-3:0}

[nilshoerrmann]

Would it be a problem to provide both, emumeration and ternary? It would be a huge improvement.

[brendo]

Very true, so ?: would not replace :, it could be a shortcut, but it doesn't seem worth it at all now that I think about it.

For the record, Enumeration "hops" in turn from one value to another in a series until true is probably the clearest explanation of : that exists anywhere. This needs to be worked into the docs somewhere!

[allen]

I've been a defector of the DS filtering syntaxes soon after we conceived it.

Here are some of the concepts we already have in our system:

• Curly brace for parameters
• Enumeration
• Regexp
• Not-regexp
• Union operand
• Intersection operand
• Date ranges

The ideal was for Symphony's backend to have as minimal amount of custom syntaxes as possible.

So to my eyes, why is custom syntax so bad? The main reason is that programming syntaxes is something that you cannot deduce, but rather something you have to learn. For a user who's got an idea of what they want to do, they (ideally) should be able to approach the DS editor and be able to follow the user interface and achieve what they want without having to refer to an external resource. At the very least, the UI should prompt the user to what selection of syntaxes are available for filtering.

I'm sure many of the regular users still get confused by the differences between the comma operand versus the plus operand and what they represent.

I think we've managed to solve this problem part-way in v3 by providing a logic gate select box per filter field. For example, the number field would have a selection of greater than, less than, equal to, not equal, etc. Furthermore the duplicator allows multiple filters on a field. For example, the date field can have two filter items one for greater than 15 and another for less than 30.

Going back to v2, I understand that unless we rethink the user interface, the simplest solution is to introduce new syntaxes to cover different filtering scenarios, but every time we do this, we're moving one step further away from our ideal.

At the very least, do we think we'd be able to combine the concept of enumeration and ternary into one syntax and not two?

[ahwayakchih]

Sorry to interrupt, but i lately i have noticed that some of the fields i used, have a use for the same filters. Wouldn't it be nice to have class.field.php to have some default filters available as functions? Similar to the way it provides functions for generating parts of settings panel. For example, filterDate(), filterInteger() (or filterNumeric()), filterBoolean() and filterString() could be very useful and would prevent code duplication between many fields. Each of those functions could take the same arguments as buildDSRetrievalSQL() plus $columnName, so they knew what they should compare to. They would return false if data did not contain correct syntax, or true if they accepted data and created filtering SQL.

[nilshoerrmann]

As a normal user I would actually like to have more custom syntaxes that allow me to keep away from the PHP source instead of having to edit my data sources which make the data source editor useless because I have to disallow editing in these cases. I think we need to have pragmatic solution that make things work without trying to be the ideal approach.

I think we've managed to solve this problem part-way in v3 by providing a logic gate select box per filter field. For example, the number field would have a selection of greater than, less than, equal to, not equal, etc. Furthermore the duplicator allows multiple filters on a field. For example, the date field can have two filter items one for greater than 15 and another for less than 30.

My main concern is that I still have to live without Symphony 3 for quite while: I need a system that is production ready. I love the improved interface in version 3 because it solves core issues that are visible since the first release of version 2. But I don't want to dream about possible future improvements: when will we see a stable version 3? No one knows.

That said I'd give a lot to have S3 filtering available in the S2 branch. So much about dreaming ...

[ahwayakchih]

That said I'd give a lot to have S3 filtering available in the S2 branch. So much about dreaming ...

That is another issue. I can't understand why there is so much separation between S2 and S3 (or any future versions). Why not test new ideas for next version, and then implement them in current version? Incremental development would not only bring improvements to the current version tree, but also help with upgrades later (because less things would get broken between versions).

As a normal user I would actually like to have more custom syntaxes that allow me to keep away from the PHP source

My suggestion about default syntaxes does not prevent any new syntaxes. Field authors woould be able to call default filter syntax, and then add their own syntax on top, just like with any other part of Field functionality, i.e., options displayed in field's settings panel.

[nilshoerrmann]

Just as a note: My comment regarding custom syntaxes was in reply to @allen's comment. Yours was not visible when I was replying – I like the idea of default filters by the way.

[allen]

That is another issue. I can't understand why there is so much separation between S2 and S3 (or any future versions).

v3 aims to solve fundamental issues that persist in v2. The symphony core is showing its age and we are hitting some limitations both in terms of the core architecture and the user interface framework. v3's architecture sheds the baggage of v2's existing underpinnings which opens up new possibilities such as a more modular DS filtering mechanism, possibility of introducing XPath type filtering or new concepts like dynamic delegates (something that Brendan, Rowan and Nick have discussed recently).

The things that can be shared between v2 and v3 would be broader feature concepts, but its method of implementation into its core would be different enough that the work involved would be twofold.

@nilshoerrmann can you provide a couple of examples that the ternary operator can do that other existing syntaxes cannot solve for you? Perhaps others might be able to offer a combined syntax that could solve this particular problem.

[nilshoerrmann]

@allen: It's more or less a way to allow two things:

1. Make a data source return nothing (or a default), if a certain parameter is set.
2. Make a data source return nothing (or a default), if multiple parameters are set.

Both can only be achieved with custom data source additions.

We just worked on a site with multilingual content which has a completely dynamic navigation structure and we needed to customize 90% or our data sources because this kind of logic was not available.

But the initial idea to have a ? or ! syntax came from Subsection Manager captions where we are searching for a way to handle empty fields, e. g. to only display a comma when needed in a caption, you could write something like this: {$title?, :}{$title}

[ahwayakchih]

allen,

possibility of introducing XPath type filtering

That's one of the things i was thinking about some time ago. I have code that can generate SQL from very basic XPath (no XPath functions for now, but i think they could be implmented too), as long as it knows about connections between database tables. I was planning to turn that into either custom data source, or another "fake" field used for filtering (just like my Filter field :).

The things that can be shared between v2 and v3 would be broader feature concepts, but its method of implementation into its core would be different enough that the work involved would be twofold.

I understand that, but i noticed at least few times that Symphony developers push even small changes to the v3, even though they could be implemented in v2 without much trouble. I don't know about others, but for me it sometimes is a bit troubling to see that. It's like developers kept saying "This is v2, you can use it or not, but we will ditch it soon, or not so soon, you never know, and you will be left all alone. That thing there, over horizon, is v3 and it will be awesome CMS, full of magic and unicorns, it will do everything you dreamed of and more, but you cannot use it anytime soon. v3 will be so much better and different from v2, that most probably you will not be able to upgrade v2 sites to it. At least not without huge amount of additional work, which practically will mean that it would be easier to just start fresh with v3.". I don't know if you mean that or not, but that is how i see that. My past experience with lack of upgrade path from v1 to v2 does not help here either.

I know it is open source (and i love it :), and that i can always add own code. But i still would like to be able to keep up with official branch, instead of turning into "re-apply all patches with every new release" hell hole. The magic cloud you guys create around v3, and the "dead meat" stink that is automatically created around v2, are not really encouraging for anyone that would like to build sites on Symphony (which is sad, because Symphony is best CMS i have every worked with).

nilshoerrmann, i also have a use for a lot of conditional data-source execution, that is why i made Filter field. Those two cases you mentioned are perfect example of what that field is used for :). It could use easier to follow syntax, and that is what i plan to implement later, but it is already used a lot in my projects.

[nilshoerrmann]

@ahwayakchih: Thanks for the link. I didn't know your Filter field. Good stuff. But I still think this is something Symphony should be able to do without an extension and a pseudo field. We already have this half-backed Required URL Parameter ...

[ahwayakchih]

nilshoerrmann, i agree that Symphony should be able to do that. But i am not sure if $param syntax should be changed. Right now it is very close to the one used in XSLT and very simple. I think that conditional execution of data-sources should be possible to configure through UI (we're talking about future versions, where we can implement new stuff :), without a need for complicating $param syntax. I think that if we start with ? and !, we will soon need more special markup, and end up with re-implementation of REGEXP :).
Required URL Parameter is nice for simplest situations. When i need to handle more conditions, i use filter field. Sometimes even with Required URL Parameter too (though mostly because of filter field syntax being unreadable ;).

[brendo]

That said I'd give a lot to have S3 filtering available in the S2 branch. So much about dreaming ...

Why dream? I believe we can make it happen. While there is some fundamental differences that S3 solves, in my opinion the Datasource Editor is not one of them, it's a nicer interface to essentially create the same file we have now. Going with what @allen mentioned earlier, introducing new syntaxes takes us away from the ideal, so why bother? Lets do it right, or not do it at all.

Why not test new ideas for next version, and then implement them in current version? Incremental development would not only bring improvements to the current version tree, but also help with upgrades later (because less things would get broken between versions).

I agree with this statement wholeheartedly. I know previously I was a pretty big advocate of 'why bother with S2, S3 is coming'. My opinion has changed after thinking about our Convore discussions in recent times. It's pretty clear that the system we want is not what S3 is, it's something entirely different. We have done some great things in S3 (we've also done some horrible things), but instead of letting it go to waste, lets try to bring some improvements back into the real world, for everyone to use.

v3's architecture sheds the baggage of v2's existing underpinnings which opens up new possibilities such as a more modular DS filtering mechanism, possibility of introducing XPath type filtering or new concepts like dynamic delegates (something that Brendan, Rowan and Nick have discussed recently).

To clarify that though @allen, that's not S3, that's Crane. I see S3 as a technical prototype, released a little too enthusiastically as a beta. It's almost a step backwards from S2 in some respects due to our recent documentation and working group efforts.

So to get back on topic, I don't think we should introduce any new filters, I think if we're going to put any effort into this at all, we do it right and implement the UI to build filters and add logic settings, otherwise lets spend time on other features. This said, our 2.3 TODO is getting rather long, which is dangerous because it's then seen as overwhelming, and it doesn't get done, people get unhappy, and eventually we've got a giant release with a 1000 things to test. If this doesn't make in 2.3, lets not despair, there can always be a 2.4.

Release early, and often.

[nilshoerrmann]

I agree with this statement wholeheartedly. I know previously I was a pretty big advocate of 'why bother with S2, S3 is coming'. My opinion has changed after thinking about our Convore discussions in recent times. It's pretty clear that the system we want is not what S3 is, it's something entirely different. We have done some great things in S3 (we've also done some horrible things), but instead of letting it go to waste, lets try to bring some improvements back into the real world, for everyone to use.

That's music in my ears!

[ahwayakchih]

Lets do it right, or not do it at all. [...]

Hip

[...] lets try to bring some improvements back into the real world, for everyone to use.

Hip

Release early, and often.

Hooray :)!

[nilshoerrmann]

Continued in #757.