Coder Social home page Coder Social logo

lintrule's Introduction

Lintrule

Lintrule is a new kind of linter and test framework.

Install

curl -fsSL https://www.lintrule.com/install.sh | bash

Usage

In your codebase, setup a rules folder with the init command.

rules init

Next, login to Lintrule.

rules login

This will create a file a rules/no-bugs.md with your first rule. It's just a markdown file that says "don't approve obvious bugs." Try running it with:

rules check

To save on costs, Lintrule runs on diffs. By default, it runs on the changes since the last commit, effectively git diff HEAD^. If you want it to run on other diffs, you can pass them in as arguments.

# Check against main and the a feature branch
rules check --diff main..my-feature-branch

# Run on the last 3 commits
rules check --diff HEAD~3

In a GitHub Action

Create a new secret and add it as an environment variable (LINTRULE_SECRET) to your GitHub Action.


rules secrets create

Then add the following to a workflow file in .github/workflows/rules.yml.

name: Rules Check

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  rules:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v2
        with:
          fetch-depth: 2 # this part is important!

      - name: Install Lint Rules
        run: |
          curl -fsSL https://www.lintrule.com/install.sh | bash

      - name: Run Lint Rules Check
        run: |
          rules check --secret "${{ secrets.LINTRULE_SECRET }}"

Configuring rules

You can ensure rules only run on certain files by adding them to the frontmatter, like this:

---
include: ["**/**.sql"]
---

We're running postgres 8 and have about 1m rows
in the "users" table, please make sure our
migrations don't cause problems.

FAQ

Does Lintrule run on diffs?

Yes. By default, Lintrule runs only on changes that come from git diff HEAD^.

If you're in a GitHub Action, Lintrule smartly uses the GITHUB_SHA and GITHUB_REF environment variables to determine the diff. For PRs, Lintrule uses the GITHUB_BASE_REF and GITHUB_HEAD_REF.

Does it have false positives?

Yes. Just like a person, the more general the instructions, the more likely it will do something you don't want. To fix false positives, get specific.

On the other hand, Lintrule tends to not be flaky. If a rule produces a false positive, it tends to produce the same false positive. If you fix it, it tends to stay fixed for the same type of code.

That's a lot of money, how do I make it cheaper?

  • The estimator shows you how much it costs if you run Lintrule on every commit. Try running Lintrule only on pull requests.
  • Instead of using lots of rules, try fitting more details into one rule. But be warned, the more competing details you have in a rule, the more likely it is that you'll get false positives.
  • Use include to silo your rules to certain files. That makes it easier to add more rules without increasing your cost.

As LLMs get cheaper to run, we expect the prices to go down significantly.

Is it slow?

Not really. Lintrules runs rules in parallel, so regardless of how many rules or files you have, it will complete in a few seconds.

lintrule's People

Contributors

anmolm96 avatar chitalian avatar flaque avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

lintrule's Issues

Consider using temperature of 0 for greater determinism?

I have a single new commit, adding the file bugs.ts with contents throw new Error("I am a bug!"), and the default rules/no-bugs.md with contents don't approve obvious bugs.

Running rules check twice, I get different responses:

➜  rules check

$ git diff HEAD^

Found 2 changed files...

   ✔️ PASS  rules/no-bugs.md => rules/no-bugs.md (4122ms)
   FAIL  bugs.ts rules/no-bugs.md
The code contains an obvious bug: a thrown error with the message 'I am a bug!' (9684ms)
  1 rules failed. 

and then

➜  rules check

$ git diff HEAD^

Found 2 changed files...

   FAIL  bugs.ts rules/no-bugs.md
Obvious bug found: Error 'I am a bug!' is deliberately thrown. (6723ms)
   FAIL  rules/no-bugs.md rules/no-bugs.md
The code snippet is not provided. Please provide a code snippet to evaluate. (7186ms)
  2 rules failed. 

I notice that the error message for the obvious bug is different between the two, which is the core of this report.

However, note that there appears to be a separate bug causing the "Please provide a code snippet to evaluate" error message. Let me know if you'd like me to file that separately.

GH action is asking for billing to be set, even tho its set already

I have just setup lintrule, selecting the paid plan and entering billing details in stripe.

Screenshot 2023-10-20 at 10 38 13

Despite that, running rules check does not work and tells me to setup my billing details.

Uncaught (in promise) Error: Please setup your billing details! Please run rules login to setup your billing details

Not sure if there is anything corrupt with my configuration, or its more of a bug. Is there any support channel I could take this to? Haven't seen anything anywhere.

Thanks 🙏

Checking a diff throws a missing file exception if HEAD doesn't contain a file in the diff

I tried diffing specific commits, such as:
rules check --diff bb9fb02818fd154c101047a71899deef6c953586...5f3b4b91662b094bf994fc86466b9814a2861dd2

and received this error:
Uncaught (in promise) NotFound: No such file or directory (os error 2): readfile 'tests/resolvers.test.ts

It's checking my current file structure for files, but it shouldn't be, it should just be paying attention to the specified diff.

GH actions - Don't store secrets in code. The document contains a secret in the line 'rules check --secret "${{ secrets.LINTRULE_SECRET }}"'.

Good day,

We integrated lintrule to our repository, following the official documentation. However, the action failed with the following error:

Don't store secrets in code. The document contains a secret in the line 'rules check --secret "${{ secrets.LINTRULE_SECRET }}"'.

It seems that the the documentation is missing something to avoid this error; or that the LLM has a bug for this case.

Please, check.

Thank you,

Problem with install script

It looks like the script originally expected the release to be a binary, so it tries to unzip it into /usr/local/bin. Modifying the script to use mktmp for extraction got it to work for me:

#!/bin/bash

# Define the GitHub repository and the name of the binary.
GITHUB_REPO="Flaque/lintrule"
BINARY_NAME="rules"

# Check the operating system
OS="$(uname)"

# If the operating system is Linux, set the target directory to '/usr/local/bin'
# If the operating system is Darwin (macOS), set the target directory to '${HOME}/.local/bin'
if [[ "$OS" == "Linux" ]]; then
  TARGET_DIR="/usr/local/bin"
elif [[ "$OS" == "Darwin" ]]; then
  TARGET_DIR="${HOME}/.local/bin"
else
  echo "Unsupported operating system: $OS"
  exit 1
fi

command -v unzip >/dev/null ||
    error 'unzip is required to install lintrule'

# Make sure the target dir exists
mkdir -p "${TARGET_DIR}"

# Define the target file path for the 'rules' CLI binary.
TARGET_FILE="${TARGET_DIR}/${BINARY_NAME}"

case $(uname -ms) in
'Darwin x86_64')
    target=x86_64-apple-darwin
    ;;
'Darwin arm64')
    target=aarch64-apple-darwin
    ;;
'Linux x86_64' | *)
    target=x86_64-unknown-linux-gnu
    ;;
esac

# Set up temporary directory for download and extraction
TMPDIR=$(mktemp -d)

GITHUB=${GITHUB-"https://github.com"}

github_repo="$GITHUB/$GITHUB_REPO"

if [[ $# = 0 ]]; then
    RULES_BINARY_URL=$github_repo/releases/latest/download/rules-$target.zip
else
    RULES_BINARY_URL=$github_repo/releases/download/$1/rules-$target.zip
fi

# Check if the download URL was found.
if [ -z "${RULES_BINARY_URL}" ]; then
    echo "Failed to find the download URL for the '${BINARY_NAME}' binary."
    echo "Please check the GitHub repository and release information."
    exit 1
fi

# Download the 'rules' CLI binary from the specified URL.
echo "Downloading '${BINARY_NAME}' CLI binary..."
curl -L -o "${TMPDIR}/${BINARY_NAME}.zip" "${RULES_BINARY_URL}"

# Extract the zip file in the temporary directory.
echo "unzip -o \"${TMPDIR}/${BINARY_NAME}.zip\" -d \"${TMPDIR}/dist\""
unzip -o "${TMPDIR}/${BINARY_NAME}.zip" -d "${TMPDIR}/dist" ||
    error 'Failed to extract rules'

# Move the binary to the target directory.
sudo mv "${TMPDIR}/dist/rules-$target" "${TARGET_DIR}/${BINARY_NAME}"

# Make the downloaded binary executable.
sudo chmod +x "${TARGET_FILE}"

# Clean up the temporary directory.
rm -rf "${TMPDIR}"

# Verify that the 'rules' CLI binary is successfully installed.
if [ -f "${TARGET_FILE}" ]; then
    echo "Successfully installed '${BINARY_NAME}' CLI."
    echo "The binary is located at '${TARGET_FILE}'."

    # Provide instructions for adding the target directory to the PATH.
    echo -e "\033[0;32m"
    echo -e "To use the '${BINARY_NAME}' command, add '${TARGET_DIR}' to your PATH."
    echo -e "You can do this by running one of the following commands, depending on your shell:"
    echo -e "\033[0m"
    echo -e "\033[0;32mFor bash:"
    echo -e "\033[1m  echo 'export PATH=\"${TARGET_DIR}:\$PATH\"' >> ~/.bashrc && source ~/.bashrc\033[0m"
    echo -e "\033[0;32m"
    echo -e "\033[0;32mFor zsh:"
    echo -e "\033[1m  echo 'export PATH=\"${TARGET_DIR}:\$PATH\"' >> ~/.zshrc && source ~/.zshrc\033[0m"
    echo -e "\033[0;32m"
    echo -e "After running the appropriate command, you can use '${BINARY_NAME}'.\033[0m"


else
    echo "Installation failed. '${BINARY_NAME}' CLI could not be installed."
fi

Uncaught error when `.gitignore` isn't present

An uncaught error is thrown when a repo doesn't have a .gitignore file:

error: Uncaught (in promise) NotFound: No such file or directory (os error 2): readfile '.gitignore'
    at async Object.readTextFile (ext:deno_fs/30_fs.js:754:18)
    at async walkTextFiles (file:///Users/evan/dev/src/flaque/lintrule/cli/walkTextFiles.ts:78:30)
    at async checkRulesAgainstDiff (file:///Users/evan/dev/src/flaque/lintrule/cli/cmds/check/cmd.ts:81:22)
    at async Command.execute (https://deno.land/x/[email protected]/command/command.ts:696:13)
    at async Command.parseCommand (https://deno.land/x/[email protected]/command/command.ts:584:20)
    at async file:///Users/evan/dev/src/flaque/lintrule/cli/main.ts:43:1

Can take a stab at fixing this tomorrow, assume it's just wrapping some LOCs in a try/catch block?

Lintrule does not respect instructions to ignore certain things

I wanted to add linting to our github workflows, but lintrule is convinced that actions/checkout@v3 does not exist (this makes sense, it wouldn't have a database of package versions). I get warnings like this:

   x FAIL  .rules/github.md => .github/workflows/lintRule.yml (6863ms)

The 'uses' field of the 'actions/checkout' step should have a valid version, such as 'v2' instead of 'v3'.

However, more concerning is that I tried quite a few ways to tell lintrule to ignore this class of errors:

Do not validate the uses field of any workflow step. Allow any package versions, even if they are not the latest or do not exist.

here's my full config:

---
include: [".github/**/*"]
---

YML files related to github workflows should be correct, clear, and secure.
Do not validate the `uses` field of any workflow step. Allow any package versions, even if they are not the latest or do not exist.

1. Disallow typos and copy-paste errors.
2. Disallow configuration errors or bugs.
3. Names for each step should be accurate descriptions of the step

Any guidance on how to tell lintrule to ignore something like this? I would imagine this specific issue (package versions) to affect a lot of folks, as many configuration files end up including dependency versions, which lintrule likely would not have a complete database for.

Lint Rule result in PR comment

Is there a way to have github actions post the lint rule results and comments into the conversation of the PR that ran the workflow?

Rules still executing on files not listed in `include`?

What our rule file looks like:

---
include: ["**/**.rb"]
---

Make sure our code follows these best practices, UNLESS there's a comment explaining why it's okay to break the rule.

1. Avoid typos.
2. Don't have like, really obvious bugs.
3. Follow reasonable conventions of the language we're programming in. No need to be too strict.

What the github integration mentions:

Screenshot 2023-07-12 at 07 12 59

Unsure if this means the check was a noop, or whether the file was actually analyzed for the rule.

Throws an error on recently renamed files

I tried rules init; rules check on my main branch, and the most recent commit had had some files moved; pretend one of them was foo/README.md

I got this stacktrace:

➜  rules check                                    

$ git diff HEAD^
error: Uncaught (in promise) NotFound: No such file or directory (os error 2): readfile 'foo/README.md'
    at eventLoopTick (ext:core/01_core.js:166:11)
    at async Object.readFile (ext:deno_fs/30_fs.js:684:18)
    at async getChangesAsFiles (file:///Users/evan/dev/src/flaque/lintrule/cli/git.ts:164:19)
    at async checkCmd (file:///Users/evan/dev/src/flaque/lintrule/cli/cmds/check.ts:51:26)
    at async Command.execute (https://deno.land/x/[email protected]/command/command.ts:696:13)
    at async Command.parseCommand (https://deno.land/x/[email protected]/command/command.ts:584:20)
    at async file:///Users/evan/dev/src/flaque/lintrule/cli/main.ts:22:1

I'll note it's a bit odd to find your name in the stacktrace 😄 it was also a bit of a bummer to not see local filepaths in the stacktrace that I could use to try to debug.

My workaround was to add a dummy commit on top of that.

Use 32k Context Window Model

Looking to run lintrule one-off on a whole project before we open source it to make sure there aren't any secrets or other footguns but the 8k context window really restricts the number of files it can be run on.

Would love the option to use the 32k context window model (or the 100k Anthropic model) or ability to chunk files.

npm package?

Rather than installing something globally to my system (and only mine), I'd love to be able to add a dep to my project's package.json so that I can count on being able to run, eg, npm run rules or yarn rules or similar on my or my colleague's devices.

It just feels weird that I'd commit a ./rules/foo.md to my repo but not be able to do anything with it by default from others' machines.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.