Coder Social home page Coder Social logo

getgrit / gritql Goto Github PK

View Code? Open in Web Editor NEW
2.4K 6.0 41.0 42.69 MB

GritQL is a query language for searching, linting, and modifying code.

Home Page: https://docs.grit.io/

License: MIT License

Rust 95.63% Shell 0.25% JavaScript 1.35% TypeScript 2.36% Python 0.03% Vue 0.08% HCL 0.04% PLSQL 0.02% RenderScript 0.21% HTML 0.03%
ast codemod javascript linter refactoring rust search tree-sitter

gritql's Introduction

Grit logo


GritQL is a declarative query language for searching and modifying source code.

  • 📖 Start simply without learning AST details: any code snippet is a valid GritQL query
  • ⚡️ Use Rust and query optimization to scale up to 10M+ line repositories
  • 📦 Use Grit's built-in module system to reuse 200+ standard patterns or share your own
  • ♻️ Once you learn GritQL, you can use it to rewrite any target language: JavaScript/TypeScript, Python, JSON, Java, Terraform, Solidity, CSS, Markdown, YAML, Rust, Go, or SQL
  • 🔧 GritQL makes it easy to include auto-fix rules for faster remediation

Getting started

Read the documentation, interactive tutorial, or run grit --help.

Installation

Install the Grit CLI:

curl -fsSL https://docs.grit.io/install | bash

Usage

Search for all your console.log calls by putting the desired pattern in backticks:

grit apply '`console.log($_)`'

Replace console.log with winston.log, using => to create rewrites:

grit apply '`console.log($msg)` => `winston.log($msg)`'

Save the pattern to a grit.yaml file and exclude test cases in a where clause:

cat << 'EOF' > .grit/grit.yaml
patterns:
  - name: use_winston
    level: error
    body: |
      `console.log($msg)` => `winston.log($msg)` where {
        $msg <: not within or { `it($_, $_)`, `test($_, $_)`, `describe($_, $_)` }
      }
EOF
grit apply use_winston

Run grit check to enforce your patterns as custom lints.

grit check

Examples

Remove all console.log calls, unless they are inside a try-catch block

`console.log($log)` => . where {
  $log <: not within `try { $_ } catch { $_ }`
}

Replace a method call with a new method call

`$instance.oldMethod($args)` => `$instance.newMethod($args)` where {
  $program <: contains `$instance = new TargetClass($_)`
}

More examples

Many more examples can be found in the GritQL standard library.

Patterns can be combined to create complex queries, including large refactors.

Why GritQL?

GritQL comes from our experiences with conducting large scale refactors and migrations.

Usually, migrations start with exploratory work to figure out the scope of the problem—often using simple grep searches. These are easy to start with, but most migrations end up accumulating additional requirements like ensuring the right packages are imported and excluding cases which don’t have a viable migration path.

Eventually, any complex migration ends up being a full codemod program written with a tool like jscodeshift. This comes with its own problems:

  • Most of the exploratory work has to be abandoned as you figure out how to represent your original regex search as an AST.
  • Reading/writing a codemod requires mentally translating from AST names back to what source code actually looks like.
  • Most frameworks are not composable, so you’re stuck copying patterns back and forth.
  • Performance is often an afterthought, so iterating on a large codemod can be painfully slow.
  • Codemod frameworks are language-specific, so if you’re hopping between multiple languages—or trying to migrate a shared API—you have to learn different frameworks.

GritQL is our attempt to develop a powerful middle ground:

  • Exploratory analysis is easy: just put a code snippet in backticks and use $metavariables for holes you want to represent.
  • Incrementally add complexity by introducing side conditions with where clauses.
  • Reuse named patterns to avoid rebuilding queries, and use shared patterns from our standard library for common tasks like ensuring modules are imported.
  • Written in Rust for maximum performance: rewrite millions of lines of code in seconds.

Acknowledgements

GritQL uses tree-sitter for all language parsers and benefits greatly from the Rust ecosystem.

GritQL is released under the MIT license.

Contributing

Contributions are welcome. To get started, check out the contributing guidelines.

You can also join us on Discord.

gritql's People

Contributors

arendjr avatar ayewo avatar gankra avatar ilevyor avatar lmb avatar mistydemeo avatar morgante avatar mparker3 avatar rjected avatar seren5240 avatar shylock-hg avatar urbit-pilled avatar webpro avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

gritql's Issues

`--language` flag on the CLI

For simple inline patterns, needing to specify the language in the declaration is a hindrance.

For example:

$ grit apply 'language css
r"--red-(.+)"($var)'

This would be more ergonomic:

$ grit apply 'r"--red-(.+)"($var)'' --language css

HTML support

We would like to support HTML as a language for matching against.

Review this guide and join us on Discord to discuss.

Acceptance criteria

  1. Working grammar + language support
  2. At least 10 test cases, including rewrites and metavariables.

Allow patterns test to filter out patterns which use ai.

The standard library has patterns that use ai, but ai functionality is not open source, so running grit patterns test using an open source build fails to run, as the patterns which make use of ai fail to compile. easiest fix (although a bit hacky) would be to filter out tests that don't compile, and list them as skipped due to compilation failures at the end of the report.

Cannot rewrite empty field nodes

Currently, if a field is null we can't rewrite/insert into it (because we don't have an associated range for a field that wasn't found).

Workaround

Rewrite the entire parent node that contained the null field.

Example

engine marzano(0.1)
language python

pattern test_func($name, $parameters, $return_type, $body) {
    function_definition($name, $parameters, $return_type, $body) as $function where {
        $name <: r"test_.*",
    }
}

pattern test_class($name, $superclasses, $body) {
    class_definition($name, $superclasses, $body) as $cls where {
        $superclasses <: contains `TestCase`,
    }
}

test_func($return_type) as $function where {
    $function <: within test_class(),
    $return_type => `None`
}

Improve error messages for syntactically invalid queries

Currently, if you have a syntax error in a query the error you receive is not very helpful - just a location.

Error Pattern syntax error at 13:28. If you hit this error while running grit apply on a pattern from the Grit standard library

We would like to have much better error messages for syntax errors.

For example:

Syntax error at 13:28, received [ but expected one of {, {.

There's some prior discussion here:

Add the standard library to CI

We've had pull requests that if merged would have broken patterns used in the standard library, we should add a CI check that runs grit patterns test in CI.

blocked by: #126

Empty field matching is unintuitive

In JSON, empty strings in snippets become wildcard matches:

engine marzano(0.1)
language json

`"x": ""`

This will match "x": "" but also "x": "foo". When we find an empty node in a snippet we probably want to restrict it to only match other empty nodes.

Add some wasm tests to CI

CI currently only ensures that we build wasm. We should have a few javascript tests that run a simple patterns for each language that is supported by wasm.

Watch mode for `grit patterns test`

We would like to have a watch mode for grit patterns test --watch.

Watch mode should be similar to other testing/CLI tools that offer a watch mode: when files change, re-run tests.

Acceptance criteria

  • Watch mode should watch for file changes to any patterns in the .grit directory and automatically re-run those tests
  • If a file changes, only the affected patterns should be re-run. Ex. if I edit this file then only that pattern test should re-run.
  • You will also need to account for pattern dependencies. Ex. if https://github.com/getgrit/stdlib/blob/main/.grit/patterns/js/imports.grit is edited then any patterns which use the ensure_import_from pattern should be re-run. Utilities like this can be leveraged for walking the call graph
  • At least 2 tests should be included, including ideally at least one integration test of the final binary like we do here.

nixos-compatible install process

thanks for putting this together; it looks awesome!

unfortunately, the model of downloading binaries and running them is a moderately incompatible with nixos; it would be really nice if there was an alternative way of getting all the dependencies

some ideas:

  • add a flake.nix that builds the projects and all the deps
  • create a docker image with all the necessary deps so that can be a workaround
  • ???

thanks :)

C++ support

We would like to support C++ as a language for matching against.

Review this guide and join us on Discord to discuss.

Acceptance criteria

  1. Working grammar + language support
  2. At least 10 test cases, including rewrites and metavariables.

Python: Invalid rewrite for `$something => .`

Summary

Invalid rewrites when trying to remove a node.

Repro

https://app.grit.io/studio?key=Mtz14P9InB2-2vdJYtxLk

engine marzano(0.1)
language python


pattern test_func($name, $parameters, $return_type, $body) {
    function_definition($name, $parameters, $return_type, $body) as $function where {
        $name <: r"test_.*",
    }
}

pattern test_class($name, $superclasses, $body) {
    class_definition($name, $superclasses, $body) as $cls where {
        $superclasses <: contains `TestCase`,
    }
}

test_func($return_type) as $function where {
    $function <: within test_class(),
    $return_type => .
}

And target:

from unittest import TestCase

class TestFirst(TestCase):
  def test_typed(self) -> None:
    pass

  def test_another(self):
    pass

Produces the following:

image

The dangling -> : is invalid, the -> should be removed as well.

I also noticed this in some cases while removing some kwarg from a list of kwargs -- unfortunately I don't have a repro on that though. Removing a kwarg in some cases left a dangling comma, i.e. Something(a=1, b=2, c=3) -> "remove kwarg a" -> Something(, b=2, c=3) invalid.

Error message when pattern name contains invalid characters is confusing

The error message that comes out when a pattern name contains a - is confusing and not helpful

Reproducer:
grit.yaml

patterns:
  - name: use-winston
    level: error
    body: |
      `console.log($msg)` => `winston.log($msg)` where {
        $msg <: not within or { `it($_, $_)`, `test($_, $_)`, `describe($_, $_)` }
      }

Run:

$ grit apply use-winston
Your working tree currently has untracked changes and Grit will rewrite files in place. Do you want to proceed? yes
ERROR (code: 299) - Pattern syntax error at 1:1. If you hit this error while running grit apply on a pattern from the Grit standard library, try running grit init. If you are running a custom pattern, check out the docs at https://docs.grit.io/ for help with writing patterns.

Re-indentation

I'm looking through the docs and wonder how to handle re-indentation. The use case would be to re-write a python unitttest.TestCase class to using pytest which is plain python functions.

includes match not working

This should find many results in our monorepo:

`$client.$_($_)` as $call where {
  $client <: imported_from($source),
  $source <: includes "client"
}

TypeScript grammar needs update

This valid TypeScript code is interpreted as an error node. Error recovery creates comments where none exist:

type RouteResponse = `${infer Key extends keyof RouteResponses}:${string}`
        ? RouteResponses[Key]
        : never;

Cypress.Commands.add('getRouteVariant', (routeVariant) =>
  cy
    .request<{
      preview: {
        body: RouteResponses[typeof routeVariant extends `${infer T}:${string}`
          ? T
          : never];
      };
    }>('GET', `http://localhost:3110/api/mock/variants/${routeVariant}`)
    .then((res) => res.body.preview.body),
);

Vue module breaks tree-sitter for other grammars.

this test:

    #[test]
    fn print_sexp() {
        let code = r#"<Hello attr="foo"></Hello>"#;
        let mut parser = tree_sitter::Parser::new().unwrap();
        let lang = tree_sitter_html::language().into();
        parser.set_language(&lang).unwrap();
        let tree = parser.parse(code, None).unwrap().unwrap();
        let root = tree.root_node();
        let sexp = root.to_sexp();
        assert!(!sexp.contains("ERROR"));
        println!("{sexp}");
    }

fails even though the snippet is valid html, and parses correctly through the tree-sitter cli.
commenting out the vue module in in language/src/lib.rs fixes this which is odd as the test does not make use of any functionality outside of tree-sitter

Custom .grit path

By default .grit the cli is looking up the .grit in your current folder or in the root of your repo. I think this is a sane default. However I also think it would be nice to define a custom path for your .grit. Perhaps you want your .grit committed in a repo of it's own, or perhaps you would always like to look up patterns from ~/.grit.

Failed to fetch standard library grit module

I'm setting up grit for the first time and trying to test a pattern I've created.
When I run the grit, I get an error

$ grit apply props_use_swagger

Error: Failed to fetch standard library grit module getgrit/stdlib: Failed to clone repo getgrit/stdlib: authentication required but no callback set; class=Ssh (23); code=Auth (-16)

I've also tried logging in through grit auth to see if it fixes it, but it hasn't.

Note that I've installed and am using grit with nushell, but I seem to have the same issue when running through zsh.


This is the only pattern I have

// ~/.grit/patterns/<redacted>/common.grit

language js

// Find cases where a member of a prop is imported from swagger types
pattern props_use_swagger() {
  `interface $interface { $entries }` where { and {
    `interface` <: r`Props$`,
    `entries` <: contains `$name: $type` where {
      $type <: ensure_import_from("@/types/swagger")
    }
  } }
}

Improve contribution guide

The contributing guide covers from Grit-specific things, but doesn't provide a brief introduction to building the repo and making contributions.

We should provide a step by step guide for new contributors.

[Question] It's possible to automate "@jest/globals" import in test files

Context

Since version 27, jest start to switch from using @types/jest where types are globally injected, like jest, expect and others modules, to manually import from @jest/globals.

So, we need to refactor all our tests files and import manually, like this:

 import { jest, expect, describe, it, /* and others */ } from '@jest/globals';`

Question

The challenge is: Find on all *.spec.ts(x) files, what modules were used (jest, expect, etc.) and then add a import {...} from '@jest/globals' for them.

It's possible with gritql to automate this refactoring?

Unexpected stack overflow

engine marzano(0.1)
language js

pattern traverse($accumulated_name) {
    or {
        `describe($name, $describe)` where {
            $next_name=`alf`,
            $describe <: contains traverse($accumulated_name) until `describe($_, $_)`
        },
        `it($final, $_)` where {
            $final => `$accumulated_name`
        }
    }
}

file($body) where {
    $body <: contains bubble traverse(accumulated_name=`d`) until `describe($_, $_)`
}

Input file

describe("My describe", () => {
  const input = "5";

  describe("foo", () => {
    it("stops here")
  });

  describe("nested more", () => {
    describe("something...", () => {
      it('branch 2', () => {});
      it('branch 3', () => {});
    })
  });
});

Yaml frontend

People inexplicably like to write their DSLs in yaml, compile yaml down to grit

Python: blank lines break function rewrite

Summary

Blank lines break function rewrite.

Repro

With gritql query:

engine marzano(0.1)
language python

pattern resolver_func($name, $parameters, $return_type, $body) {
    function_definition($name, $parameters, $return_type, $body) as $function where {
        $name <: "resolve",
        $return_type <: "Out1 | None"
    }
}

pattern change_signature() {
    resolver_func($return_type, $body, $name, $parameters) => 
    `
def $name($parameters) -> Out2 | None:
    # TODO: fix body
    $body
    `
}

change_signature()

And test Python code:

from dataclasses import dataclass


@dataclass(frozen=True)
class Out1:
    res: str


@dataclass(frozen=True)
class Out2:
    res: str


@dataclass(frozen=True)
class A:
    a: str

    def resolve(self, a: str, b: str) -> Out1 | None:
        1 + 1
        print("hi")

        1 + 1

        return Out1("hi")


@dataclass(frozen=True)
class B:
    a: str

    def resolve(self, a: str, b: str) -> Out1 | None:
        return Out1("hi")

I get the following error message:

Error expected line to start with 8 spaces, code is either not indented with spaces, or does not consistently indent code blocks

Rewriting A as:

@dataclass(frozen=True)
class A:
    a: str

    def resolve(self, a: str, b: str) -> Out1 | None:
        1 + 1
        print("hi")
        1 + 1
        return Out1("hi")

Then it works as expected.

Seems to be coming from here ish:

let line = line.strip_prefix(&padding).ok_or_else(|| {
anyhow!(
"expected line \n{}\n to start with {} spaces, code is either not indented with spaces, or does not consistently indent code blocks",
line,
pad_strip_amount
)
})?;

PHP language support

We would like to support PHP as a language for matching against.

Review this guide and join us on Discord to discuss.

Acceptance criteria

  1. Working grammar + language support
  2. At least 10 test cases, including rewrites and metavariables.

Ruby support

We would like to support Ruby as a language for matching against.

Review this guide and join us on Discord to discuss.

Acceptance criteria

  1. Working grammar + language support
  2. At least 10 test cases, including rewrites and metavariables.

Assignment approach

Since this is a large project, here is how we will tackle it:

  • If anyone who has already successfully contributed a grammar wants to take it on, we will assign them first dibs.
  • Others are welcome to contribute. We will assign the bounty to the first major PR submitted that is close to complete.

Votes

  • 3

The `sequential` example in the docs doesn't work

When you try the following pattern described in the docs for sequential it doesn't work in the studio.

You also get the following warning.

Log Warning: sequential matches at the top of the file. If a pattern matched outside of a 
sequential, but no longer matches, it is likely because naked patterns are 
automatically wrapped with `contains bubble <pattern>`

This warning is somewhat unclear, as it doesn't really tell the user how they should proceed to make things work the way they expect it to.

Type system queries

Example query to find functions that have a first arg of string

`function ($arg) { $_ }` where {
  $arg <: type `string`
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.