Coder Social home page Coder Social logo

todogroup / repolinter Goto Github PK

View Code? Open in Web Editor NEW
413.0 61.0 71.0 3.46 MB

Repolinter, The Open Source Repository Linter

Home Page: https://todogroup.github.io/repolinter/

License: Apache License 2.0

JavaScript 98.96% Ruby 0.43% Batchfile 0.01% Dockerfile 0.60%
javascript linter github

repolinter's Introduction

Repo Linter Build Status

Lint open source repositories for common issues.

Installation

Repolinter requires Node.JS >= v12 to function properly. Once Node.JS is installed, you can install Repolinter using npm:

npm install -g repolinter

Linting a Local Repository

Once installed, run the following to lint a directory:

repolinter lint <directory>

The above command will lint <directory> with the local repolinter.json ruleset or the default ruleset if none is found:

repolinter % repolinter lint .
Target directory: <directory>
Lint:
✔ license-file-exists: Found file (LICENSE)
✔ readme-file-exists: Found file (README.md)
✔ contributing-file-exists: Found file (CONTRIBUTING)
✔ code-of-conduct-file-exists: Found file (CODE-OF-CONDUCT)
✔ changelog-file-exists: Found file (CHANGELOG)
...
repolinter % echo $?
0

Linting a Remote Repository

Repolinter also supports linting a git repository using the --git flag. With this flag enabled, the directory input will be interpreted as a git URL which Repolinter will automatically clone into a temporary directory.

repolinter lint -g https://github.com/todogroup/repolinter.git

Formatting the Output

The Repolinter CLI currently supports three output formatting modes:

  • Default (also referred to as result)
  • JSON
  • Markdown

You can switch formatters using the --format flag. An example of using the JSON formatter:

repolinter % repolinter lint --format json .
{"params":{"targetDir":"/Users/nkoontz/Documents/code/repolinter","filterPaths":[],...

An example of using the Markdown formatter:

repolinter % repolinter lint --format markdown .
# Repolinter Report

This Repolinter run generated the following results:
| ❗  Error | ❌  Fail | ⚠️  Warn | ✅  Pass | Ignored | Total |
|---|---|---|---|---|---|
| 0 | 0 | 0 | 15 | 10 | 25 |
...

Limiting Paths

Repolinter supports an allowed list of paths through the --allowPaths option to prevent the accidental linting of build artifacts. These paths must still be contained in the target directory/repository.

repolinter lint --allowPaths ./a/path --allowPaths /another/path

Disabling Modifications

By default Repolinter will automatically execute fixes as specified by the ruleset. If this is not desired functionality, you can disable this with the --dryRun flag.

Ruleset Configuration

Similar to how eslint uses an eslintrc file to determine what validation processes will occur, Repolinter uses a JSON or YAML configuration file (referred to as a ruleset) to determine what checks should be run against a repository. Inside a ruleset, there are two main behaviors that can be configured:

  • Rules - Checks Repolinter should perform against the repository.
  • Axioms - External libraries Repolinter should use to conditionally run rules.

These combined capabilities give you fine-grained control over the checks Repolinter runs.

Providing a Ruleset

Repolinter will pull its configuration from the following sources in order of priority:

  1. A ruleset specified with --rulesetFile or --rulesetUrl
  2. A repolint.json, repolinter.json, repolint.yaml, or repolinter.yaml file at the root of the project being linted
  3. The default ruleset

Creating a Ruleset

Any ruleset starts with the following base, shown in both JSON and YAML format:

{
  "$schema": "https://raw.githubusercontent.com/todogroup/repolinter/master/rulesets/schema.json",
  "version": 2,
  "axioms": {},
  "rules": {}
}
version: 2
axioms: {}
rules:

Where:

  • $schema- points to the JSON schema for all Repolinter rulesets. This schema both validates the ruleset and makes the ruleset creation process a bit easier.
  • version - specifies the ruleset version Repolinter should expect. Currently there are two versions: omitted for legacy config (example) and 2 for all others. Use 2 unless you know what you're doing.
  • axiom - The axiom functionality, covered in Axioms.
  • rules - The actual ruleset, covered in Rules.

Rules

Rules are objects of the following format:

"<rule-name>": {
  "level": "error" | "warning" | "off",
  "rule": {
    "type": "<rule-type>",
    "options": {
      // <rule-options>
    }
  },
  "where": ["condition=*"],
  "fix": {
    "type": "<fix-type>",
    "options": {
      // <fix-options>
    }
  },
  "policyInfo": "...",
  "policyUrl": "..."
}
<rule-name>:
  level: error | warning | off
  rule:
    type: <rule-type>
    options:
      <rule-options>
  where: [condition=*]
  fix:
    type: <fix-type>
    options:
      <fix-options>
  policyInfo: >-
    ...
  policyUrl: >-
    ...
  • rule - The check to perform. Repolinter can perform any check listed under the rules documentation. Unlike eslint, Repolinter checks are designed to be reused and specialized: for example, the file-existence check can be used in a README-file-exists rule and a LICENSE-file-exists rule in the same ruleset. This allows a user to write a very specific ruleset from configuring generic checks.
  • level - The error level to notify if the check fails. warning will not change the exit code and off will not run the check.
  • where - Conditionally enable or disable this rule based off of axioms. Strings in this array follow the format of <axiom>=<value>, where value is either an axiom output or * to simply test if the axiom is enabled. If this option is present, this rule will only run if all specified axiom outputs are present. The available axioms in Repolinter can be found in the axioms documentation.
  • fix (optional) - The action to perform if the check performed by rule fails. Repolinter can perform any action listed under fixes documentation.
  • policyInfo, policyUrl (optional) - Information used by the formatter to indicate why the check exists from a policy perspective. Note: policyInfo will automatically have a period appended to it for formatting purposes.

A minimal example of a rule that checks for the existence of a README:

"readme-file-exists" : {
  "level": "error",
  "rule": {
    "type": "file-existence",
    "options": {
      "globsAny": ["README*"]
    }
  }
}
readme-file-exists:
  level: error
  rule:
    type: file-existence
    options:
      globsAny:
      - README*

Checking that the README matches a certain hash, and replacing it if not:

"readme-file-up-to-date" : {
  "level": "error",
  "rule": {
    "type": "file-hash",
    "options": {
      "globsAny": ["README*"],
      "algorithm": "sha256",
      "hash": "..."
    }
  },
  "fix": {
    "type": "file-create",
    "options": {
      "file": "README.md",
      "replace": true,
      "text": { "url": "www.example.com/mytext.txt" }
    }
  },
  "policyInfo": "Gotta keep that readme up to date",
  "policyUrl": "www.example.com/mycompany"
}
readme-file-up-to-date:
  level: error
  rule:
    type: file-hash
    options:
      globsAny:
      - README*
      algorithm: sha256
      hash: "..."
  fix:
    type: file-create
    options:
      file: README.md
      replace: true
      text:
        url: www.example.com/mytext.txt
  policyInfo: Gotta keep that readme up to date
  policyUrl: www.example.com/mycompany

Axioms

"axioms": {
  "<axiom-id>": "<axiom-target>"
}
axioms:
  <axiom-id>: axiom-target

Each axiom is configured as a key value pair in the axioms object, where <axiom-id> specifies the program to run and <axiom-target> specifies the target to be used in the where conditional. The available axiom IDs can be found in the axiom documentation. It should be noted that some axioms require external packages to run.

An example configuration using an axiom to detect the packaging system for a project:

{
  "$schema": "https://raw.githubusercontent.com/todogroup/repolinter/master/rulesets/schema.json",
  "version": 2,
  "axioms": {
    "packagers": "package-type"
  },
  "rules": {
    "this-only-runs-if-npm": {
      "level": "error",
      "where": ["package-type=npm"],
      "rule": { /* ... */ }
    }
  }
}
version: 2
axioms:
  packagers: package-type
rules:
  this-only-runs-if-npm:
    level: error
    where: [package-type=npm]
    rule:
      ...

Some axioms (ex. contributor-count) output numerical values instead of strings. For these axioms, numerical comparisons (<, >, <=, >=) can be also be specified in the where conditional. Note that if a numerical comparison is used for a non-numerical axiom, the comparison will always fail.

{
  "axioms": {
    "contributor-count": "contributors"
  },
  "rules": {
    "my-rule": {
      "where": ["contributors>6", "contributors<200"],
      // ...
    }
  }
}
axioms:
  contributor-count: contributors
rules:
  my-rule:
    where:
    - contributors>6
    - contributors<200
    rule:
      ...

Extending Rulesets

A ruleset can extend another ruleset, in which case the two files will be recursively merged. Extended rulesets can themselves extend additional rulesets up to 20 rulesets deep.

Extend a ruleset by including an "extends" top-level key which identifies a URL or file path:

{
  "extends": "https://raw.githubusercontent.com/todogroup/repolinter/master/rulesets/default.json"
  "rules": {
    # disable CI check
    "integrates-with-ci": {
      "level": "off"
    }
  }
}
extends: https://raw.githubusercontent.com/todogroup/repolinter/master/rulesets/default.json
rules:
  # disable CI check
  integrates-with-ci
    level: off
    ...

Relative paths are resolved relative to the location used to access the extending file. For example, if repolinter is invoked as:

repolinter -u http://example.com/custom-rules.yaml

And that ruleset includes extends: "./default.yaml", the path will be resolved relative to the original URL as http://example.com/default.yaml. If instead repolinter is invoked as:

repolinter -r /etc/repolinter/custom-rules.yaml

And that ruleset includes extends: "./default.yaml", the path will be resolved relative to the original file path as /etc/repolinter/default.yaml.

YAML and JSON rulesets can be extended from either format.

API

Repolinter also includes an extensible JavaScript API:

const repolinter = require('repolinter')
const result = await repolinter.lint('.')

This API allows the developer to have complete control over the configuration and formatting Repolinter should use. Documentation for this library can be found under API Documentation.

Going Further

License

This project is licensed under the Apache 2.0 license.

repolinter's People

Contributors

ab-hay avatar adamdmharvey avatar ainsleyh avatar bigbluehat avatar bkeepers avatar brend-smits avatar bufferoverflow avatar caniszczyk avatar cdelahousse avatar craigez avatar dependabot[bot] avatar hoexter avatar hyandell avatar iamwillbar avatar jeroenknoops avatar jmertic avatar jparise avatar jwsloan avatar markbirbeck avatar prototypicalpro avatar robinpokorny avatar semantic-release-bot avatar shemnon avatar sschuberth avatar trevmex avatar trevmex-comcast avatar willnorris avatar yannjor avatar yns88 avatar zhaoyuheng200 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

repolinter's Issues

Proposal: Rename to repolint

Before the name gets too set in stone I'd like to propose renaming to repolint for consistency with jslint, eslint, etc.

use SPDX license identifiers

The awesome new license axiom support appears to use the licensee license names rather than SPDX identifiers. Let's settle on using SPDX identifiers as much as possible.

Have rules that spit out reports

Rather than assuming a report goes to STDOUT, a report could go to a file and STDOUT would get a summary. This would allow for deeper linting.

For some options, verify all versions of the repository

For example, it might be useful to verify certain file types are not present in ANY branch/version of the repository. Similar with file contents, etc...

I expect this would be optional behavior as it could be very expensive for some repositories.

Check NOTICE file exists if Apache 2.0

Proposed code. Would be much simpler with a solution for #59. Alternatively, could have a system similar to the linguist plugins where the core code identifies the license, and then rules are run per license.

// Copyright 2018 TODO Group. All rights reserved.
// Licensed under the Apache License, Version 2.0.

const isWindows = require('is-windows')
const spawnSync = require('child_process').spawnSync
const Result = require('../lib/result')

// TODO: In an ideal world this rule would depend on, and obtain the reuslts of, the licensee rule; and it would then simply be an invocation of the file-exists rule
module.exports = function (fileSystem, rule) {
  const expected = /License: ([^\n]+)/

  const licenseeOutput = spawnSync(isWindows() ? 'licensee.bat' : 'licensee', [fileSystem.targetDir]).stdout

  let result = new Result(rule, '', null, false)
  if (licenseeOutput == null) {
    result.message = 'Licensee is not installed'
    return [result]
  }

  const license = licenseeOutput.toString().match(expected)

  if (license != null && license[1] == 'Apache License 2.0') {

    const options = rule.options
    const fs = options.fs || fileSystem
    noticeFiles = ['NOTICE', 'NOTICE.md', 'NOTICE.txt']
    const file = fs.findFirst(noticeFiles)

    result.passed = !!file
    if(!!file) {
      result.message = `found (${file})`
    } else {
      result.message = `not found: (${noticeFiles.join(', ')})`
    }

  } else {
    result.message = 'Licensee did not identify this project as Apache 2.0 licensed'
  }

  return [result]
}

file-contents should support multiple files

The readme-references-license rule currently looks at only README.md because the file-contents rule doesn't support multiple files which is inconsistent with the readme-file-exists rule which looks for README*.

Inherit a ruleset from another ruleset

Today the only way to customize the rules is to copy the default ruleset and edit it... You should be able to create a ruleset that extends a ruleset and add new rules or change configuration on existing rules.

Rules should be developable in Rule language as well as JavaScript

It would be nice to describe a rule in rule format rather than writing JavaScript every time.

Rather than the ruleset having:

"license=Apache-2.0": {
  "notice-file-exists:file-existence": [
    "error",
    {
      "files": ["NOTICE*"],
      "fail-message": "The NOTICE file is described in section 4.4 of the Apache License version 2.0. Its presence is not mandated by the license itself, but by ASF policy."
    }
}

Instead it would have:

"license=Apache-2.0": {
  "notice-file-exists"
}

and notice-file-exists would be defined as the above larger block of rule code.

Support .github/ and docs/ for magic GitHub files

GitHub supports various magic files (CONTRIBUTING, SUPPORT, CODE_OF_CONDUCT, Templates etc). Some of them can also be in .github and docs. I suggest we support the alternatives.

It is tempting to implement it as a new github-magic rule rather than overloading things on the if-file-exists side of things. That way it could also throw an error if the file appears duplicate times, and deal with other specialness that shows up over time.

Make a logo

I am going to go on fiverr and get a logo commissioned.

Conditional rules? Rule dependencies?

The initial use case is:

If license is Apache 2.0, then check if NOTICE file is there.

It's similar to Linguist/language support in principle. If , then .

I think it means running a series of Axioms, and then having rules that are dependent on the Axiom's output. So 'If language=Java' is the axiom, and "Java rule set" is the output.

Request for Group/Org-level scanning

Thanks so much for this awesome tool! One thing that would make it even more amazing would be to add group or org-level scanning to check multiple repos in the same org.

Scan for PATENTS in files

Issue Report

The term "PATENTS" can trigger bad things for some users.

Expected Behavior

Add a rule to detect the usage of the word "PATENTS" in the project.

Actual Behavior

It does not do this right now.

Steps to Reproduce the Issue

N/A

Allow running against GitHub

Right now it assumes a filesystem. It would be great to allow the checks to run against a repo on GitHub using the API.

Option to exclude passing lines

Issue Report

If there are lots of lines in the report and most of them are passing, they can crowd out the errors.

Expected Behavior

A command line option to exclude passing lines.

Actual Behavior

No such option presently exists

Steps to Reproduce the Issue

N/A

Don't output absolute filenames

Currently the output includes absolute filenames. This looks odd when reports are published. For example:

Target directory: /Users/hyandell/oss/repolinter
Ruleset: /Users/hyandell/oss/repolinter/rulesets/default.json
Languages: ruby, batchfile, javascript

✔ readme-references-license: File /Users/hyandell/oss/repolinter/README.md contains license

Would be better for this to be independent of where I ran it. ie It should say:

Target directory: .
Ruleset: rulesets/default.json
Languages: ruby, batchfile, javascript

✔ readme-references-license: File ./README.md contains license

Site to run repolinter

It would be nice to have a site where we can input a URL and get an HTML report, using repolinter as the backend.

Enhance Licensee (or other engine) Integration

Licensee is a little touchy and sensitive to small changes in licenses, I'd like to be able to use repolinter to verify expected licenses on a project. I have a couple changes in this direction (e.g. specifying a threshold for matching) I need to clean up to submit.

repolinter should only consider files versionned in git if a .git directory is present

When I run repolinter on https://github.com/voyages-sncf-technologies/vboard this is part of the output I get:

× binaries-not-present: Excluded file type exists (vboard-front/node_modules/gifsicle/vendor/gifsicle.exe)
× binaries-not-present: Excluded file type exists (vboard-front/node_modules/jpegtran-bin/vendor/jpegtran.exe)
× binaries-not-present: Excluded file type exists (vboard-front/node_modules/optipng-bin/vendor/optipng.exe)
× binaries-not-present: Excluded file type exists (vboard-front/node_modules/phantomjs/lib/phantom/phantomjs.exe)
× binaries-not-present: Excluded file type exists (vboard-front/node_modules/jpegtran-bin/vendor/libjpeg-62.dll)
× license-detectable-by-licensee: Licensee is not installed
√ test-directory-exists: found (repolinter/tests)
√ integrates-with-ci: found (.travis.yml)
‼ source-license-headers-exist: EMFILE: too many open files

I have a feature request : could only the versionned files be parsed by repolinter when used in a git repository, to avoid those false positives and this final application crash ?

Optional summary report at the end of the run

Issue Report

I want a concise summary of errors and passing conditions at the bottom of the run, if optionally asked for.

Expected Behavior

A command line option allows for a summary to appear at the bottom of the report.

Actual Behavior

No option presently exists.

Steps to Reproduce the Issue

N/A

Ideas for new rules

Hello,

Instead of creating one issue for each, I thought it would be simpler to gather those ideas and ask for feedback in one issue:

  • check for the existence of the following files:

    • CONTRIBUTING.md
    • CHANGELOG.md
  • check that issues are enabled in the repo, or else that there is a Contact section in the README

  • suggest to add Github tags if none exists

  • if the repo contains mainly python files, suggest to add a setup.py.
    Same for NodeJS and a package.json

  • finally, I would love a check for a "Screenshots" or "Demo" section in the README it the repo contains HTML (template) files. The rationale is that it serves the project visibility a LOT to include pictures. Of course it makes a lot less sense for pure code libraries. And checking for images links in the README is not a good solution as it could be a logo.

Add wildcard to axioms

This line of the ruleset leads to errors:

  "license-detectable-by-licensee": ["error"],

Ideally that should be within an axiom, perhaps matching to '*'. Same wildcard feature could exist for linguist.

Release 0.5.0

Can we get a new version published to npm?

Thanks!

New rule: package.json validation for NodeJS

  • license parseable as a SPDX expression
  • description provided
  • keywords provided
  • homepage provided
  • author provided
  • main provided
  • repository provided
  • test script provided (and isn't the default error script)

Rename "rules" files to "modules"

So, the way I see it, we have three levels of functionality that we can define:

  1. Rulesets: sets of rules
  2. Rules: applications of modules to create some type of test. Eg. file-existence applied to the 'readme.md' pattern creates the "Does this repo have a readme" rule (ie. "readme-file-exists:file-existence": ["error", {"files": ["README*"], "nocase": true}],)
  3. Modules: query operations on the repo (eg. file-existence)

Currently, we've structured out code as

  1. Rulesets: a series of rules
  2. Rules and Modules: we use them interchangebly. We call the objects from the 'rules' folder in the result object "modules".

I suggest we do the following:

  • Rename the rules directory to modules
  • Try to make this clearer in code

This will make the distinction between rules and 'modules' more clear.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.