Idea: node pattern similarity detector tool about rubocop-ast

[pirj]

It will be able to detect identical definitions, that's for sure.
As we go further, the same similarity analysis algorithms will be required. I like the idea of making it a cop, as it would be standalone and non-intrusive. It applies restrictions, though. It won't be able to point out that a part of the node pattern is available already in a given class' inheritance chain. Neither it will be able to detect that a and b are identical:

class_or_module = '{:Class :Module}'
module_or_class = '{:Module :Class}'
def_node_matcher :a, "(const nil? #{class_or_module})"
def_node_matcher :b, "(const nil? #{module_or_class})"

I would be more in favour of a runtime tool, profiler-alike, that would load the code up, collect a tree considering inheritance, and would come up with refactor recommendations.

As @marcandre points out, it's quite a challenge, since NP is not being parsed to an AST right now, and it's a lot of work for just a single usage. Also, a weighting algorithm should be present, taking the length of the matching sequence, and their relationship in the inheritance chain. Not to say that some may have distinct parts that can be %'d.

So basically the idea was to reduce NPs into smaller parts, be able to extract common parts out of them while avoiding too much generality (end up with all variants of (%), (% %) etc in a single includable).

[marcandre]

I have a working proof of concept of a NodePattern compiler, working basically exactly as parser works (it generates ::Parser::AST::Nodes with locations and all), which would make this not too difficult. I'm planning on specializing the case of {series of literals} to a set-like node, so {:Class :Module} could be detected as similar to {:Module :Class}, but if we want more complex things like {nil? const} === {const nil?}, this could be achieved easily by specializing the eql? for the Union node to disregard the order of its children.

We could even envision applying NodePattern to... node patterns 😆

[marcandre]

Example:

> rake parse '(send {nil? const} :example <#func 42...> int+)'
(sequence
  (node-type :send)
  (union
    (predicate :nil?)
    (node-type :const))
  (symbol :example)
  (any-order
    (function-call :func)
    (number 42)
    (rest :"..."))
  (repetition
    (node-type :int) :+))

[pirj]

predicted rubocop-nodepattern will appear in the future

A RuboCop extension that would enforce writing good node patterns. And a node pattern style guide indeed :D

[marcandre]

Seems quite challenging 😮

Not related, but we can now write (block (send #global_const(:MyClass) :goodbye)) which I think is much better and can only be done by humans.

[pirj]

Sure, it's understandable that :myclass matcher can be done in a better way.
The point is similarities.

No doubt challenging, maybe even to a point where it's not really worth it :(

One option is to make it a separate binary. In this case, it would have to extract node patterns, but it will fail to find similarities in node patterns that are inherited from base classes and included from concerns.

For runtime detector, it's hard to tell when it's the right moment to start this analysis and to keep the original definitions of node patterns somewhere.

Either way, a parser for Node Pattern or an algorithm operating on expanded node pattern strings will have to be written.

[jonatas]

Not focused in detect similarity but I’m working on something like this. Basically my fast gem is going to make it possible through shortcuts. That would work out of the box. My dream would be that libraries include their on patterns to make it easy so preview code or select Tuples with the node patterns. I’m also working on another project with @jodosha to extrapolate details about any code base traversing code mixing AST and runtime. This is quite challenging and we’re stuck in the dependencies using only AST and we need to find where the proper module dependencies come from. I hope we open source it when it’s ready. The system consists on collectors of metadata from AST and it allows to capture information from dependencies recursively to easily explore all code dependencies in one place. Enviado do meu iPhone

…

Em 24 de jul de 2020, à(s) 14:56, Phil Pirozhkov ***@***.***> escreveu:  Sure, it's understandable that :myclass matcher can be done in a better way. The point is similarities. No doubt challenging, maybe even to a point where it's not really worth it :( One option is to make it a separate binary. In this case, it would have to extract node patterns, but it will fail to find similarities in node patterns that are inherited from base classes and included from concerns. For runtime detector, it's hard to tell when it's the right moment to start this analysis and to keep the original definitions of node patterns somewhere. Either way, a parser for Node Pattern or an algorithm operating on expanded node pattern strings will have to be written. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[jonatas]

Maybe we can start with a simple warning as a cop to detect the same node pattern already defined in another file?

 fast "^^(send nil? {:def_node_matcher :def_node_search} ...)" lib/rubocop 
# lib/rubocop/ast/node.rb:325
        {(class (const $_ $_) ...)
# lib/rubocop/ast/node.rb:326
         (module (const $_ $_) ...)
# lib/rubocop/ast/node.rb:327
         (casgn $_ $_        (send (const nil? {:Class :Module}) :new ...))
# lib/rubocop/ast/node.rb:328
         (casgn $_ $_ (block (send (const nil? {:Class :Module}) :new ...) ...))}
# lib/rubocop/ast/node.rb:479
        {(block (send nil? :proc) ...)

[pirj]

I recall someone has predicted rubocop-nodepattern will appear in the future :D

[bbatsov]

@marcandre Pretty cool!

I recall someone has predicted rubocop-nodepattern will appear in the future :D

Well, technically that's rubocop-ast. 😉

[pirj]

The idea may be interesting, but probably not worth the effort. I'll close for now.