sk- / git-lint

improving source code one step at a time

DEPRECATED

Unfortunately I've decided to deprecate this package.

• git-lint would be more valuable to large companies with large legacy codebases. But for smaller ones, it's quite possible (I've done it in many) to progressively make the linter config stricter.
• Users of this project wanted to have a complete out of the box experience, while my original vision was that companies/users would define their own configuration, tailoring to whatever settings and linters they wanted.
• There was not really much feedback on the RFC issue https://github.com/sk-/git-lint/issues/152
• There are some promising tools, like https://pre-commit.com/ which has an ever growing adoption rate. It's not a full replacement of git-lint, but at least would lint only modified files.

Git-Lint

Git-lint is a tool for improving source code one step at a time.

Motivation

Often times enforcing coding styles to an existing project can be a nightmare. Some reasons may include:

• the codebase is already a mess and the output of the tools is overwhelming.
• developers don't feel confident changing lines they do not own.
• or they just don't know what tool to use.

Features

This tool tackles all the 3 problems mentioned above by providing just a single tool that lints all the modified files. For each filetype it may use even more than one linter or tool. Furthermore by default it only report problems of lines that were added or modified.

Current linters:

• CSS
  • Csslint
• SCSS
  • scss-lint
• Python
  • Pylint
  • pycodestyle
• PHP
  • Php Code Sniffer
• Javascript
  • Jshint
  • Gjslint
• JPEG
  • Custom via Jpegtran
• PNG
  • Custom via Pngcrush
  • Custom via Optipng
• RST
  • Via rst2html.py (docutils)
• JSON
  • Via python json.tool module
• YAML
  • yamllint
• INI
  • Custom via ConfigParser module
• HTML
  • HTML-Linter
  • Tidy with preprocessing from template-remover
• Ruby
  • ruby-lint
  • rubocop
• Java
  • PMD (it requires to put the script run.sh in your PATH)
  • Checkstyle
• Coffeescript
  • coffeelint
• C++
  • cpplint

Example use

Below is the simplest call, for a detailed list, see the help:

$ git lint
Linting file: src/html/main.js
Line 13, E:0110: Line too long (328 characters).
Line 31, E:0001: Extra space at end of line
src/html/main.js: line 75, col 11, ['location'] is better written in dot notation.

Linting file: src/html/main.css
src/html/main.css: line 1, col 135, Warning - Duplicate property 'margin' found.

Linting file: api.py
api.py:6: [C0301(line-too-long), ] Line too long (87/80)
api.py:6: [R0913(too-many-arguments), callMethod] Too many arguments (6/5)
api.py:6: [C0103(invalid-name), callMethod] Invalid function name "callMethod"

By default git lint only reports problems with the modified lines (with the exception of some linters that check that the whole file is sound). To force displaying all the output from the linters use the -f option.

Installation

You can install, upgrade or uninstall git-lint with these commands:

$ pip install git-lint
$ pip install --upgrade git-lint
$ pip uninstall git-lint

Configuration

Git-lint comes with a default configuration that includes all the linters listed above. If you don't like that list you can write your own configuration and put it in a file called .gitlint.yaml in the root of your repository. You can copy the file https://github.com/sk-/git-lint/blob/master/gitlint/configs/config.yaml to your repo and modify it.

If you add a new linter or add a new flag to any of the commands, please share that with us, so we can integrate those changes.

The configuration support two variables for the command, requirements and arguments:

• {REPO_HOME}: the root of your repo.
• {DEFAULT_CONFIGS}: the location of the default config files.

If you need to include strings like {} or {foo} in your command, you need to double the braces as in {{}} or {{foo}}.

Git Configuration

git-lint comes with a pre-commit hook for git. To install it for your repo execute:

$ ln -s `which pre-commit.git-lint.sh` $PATH_TO_YOUR_REPO/.git/hooks/pre-commit

or if you want to install it globally execute instead:

$ ln -s `which pre-commit.git-lint.sh` /usr/share/git-core/templates/hooks/pre-commit

Mercurial Configuration

To make available git-lint with a better name in mercurial you have to add the following to your .hgrc configuration:

[alias]
lint = !git-lint $@

To add a pre-commit hook add the following:

[hooks]
pretxncommit.hglint = pre-commit.hg-lint.sh > `tty`

The hook above has a hack to display the output of the command. Additionally, as mercurial does not provide (AFAIK) any way to skip a hook, if you want to force a commit with linter warnings execute the commit command as follow:

$ NO_VERIFY=1 hg commit ...

Note though that mercurial heavily uses commit to leverage all of their commands/extensions. I've found that setting any sort of precommit hook will get on your way when using common actions as rebase or shelve.

Travis Configuration

To run git lint on Travis, you can add the following step:

git reset --soft ${TRAVIS_COMMIT_RANGE%...*} && git lint

Limitations

In some cases a change will trigger a warning in another line. Those cases are unfortunately not handled by git-lint, as it only reports those lines that were modified. Fully supporting this use case would require running the linters twice and reporting only the new lines. The most common case in which this occurs is with unused imports or variables. Let's say we have the following piece of code:

import foo
foo.bar()

If you remove the second line, git-lint will not complain as the warning is for line 1, which was not modified.

Python Versions

Python 2.7 and Python 3.5 and higher are supported.

Development

Help for this project is more than welcomed, so feel free to create an issue or to send a pull request via http://github.com/sk-/git-lint.

Tests are run using nose, either with:

$ python setup.py nosetests
$ nosetests

This same tool is run for every commit, so errors and style problems are caught early.

Adding a linter

Just need to configure the file gitlint/config.yaml. I hope the syntax is self explanatory. (Note to myself: don't be so lazy and write a proper doc for this.)

TODOS and Possible Features

• Support directories as arguments
• Provide a man page so 'git help lint' and 'git lint --help' work. I already have a script for converting the Usage to a man page, but I still need to figure out how to install it on the system.
• Allow to run a command or function when setting up the linter? These can be achieved now by running a bash script wrapping the linter. The rationale for this is that some linters, like jshint, only allow options to be in a configuration file. This is done at the moment via scripts present in the folder linters.
• Decide what linter to use based on the whole filename or even in the filetype, as returned by the command file.
• Provide better options for colorizing the output, and maybe a way to disable it. Also detect if colors are supported or if it is a tty.
• Add support for more version control systems (svn, perforce). This should be easy, it's just a matter of implementing the functions defined in gitlint/git.py or gitlint/hg.py.
• Support windows.

Contributors

• Rovanion Luckey
• Radek Simko
• Adrien Vergé
• Rob Rodrigues
• Alex LordThorsen

Changelog

v0.1.2 (2018-05-24)

• Fixed race condition when creating cache directories.

v0.1.1 (2018-05-15)

• Fixed futures dependency not working corectly in Python 3.

v0.1.0 (2018-02-26)

• Fixed setup in some systems due to default encodings.
• Changed version to use semantic versioning. If the version would have been semantic, this release would have been just a patch.

v0.0.9 (2018-01-22)

• Fixed versioning to match in both pip install and package
• Added multithreading support

v0.0.8 (2015-10-14)

• Fixed git pre commit hook (thanks to Rovanion Luckey)
• Fixed issues #64, #67

v0.0.7 (2015-06-28)

• Better support in python 3
• Removed support for Python 3.2
• Output is sorted by line and column number
• Bugfixes: issues #49, #50, #54, #62
• Added coffelint support
• Improved defaults

v0.0.6 (2014-09-08)

• Added mercurial support
• Run e2e tests on Travis

v0.0.5 (2014-05-09)

• Added linters: ruby-lint, rubocop, checkstyle, pmd
• Variables %(REPO_HOME)s and %(DEFAULT_CONFIGS)s can be specified in configuration
• Added default pylintrc configuration

v0.0.4 (2014-05-08)

• Added linters: html, tidy, scss
• Added way to override default configuration
• Improvements for Python3

v0.0.3 (2014-02-02)

• Fixes to the filter syntax
• Fixes to the git parser
• Added linters (YAML, Ini, PHP) and improved linter for PNG and JPEG.
• Improved pylint configuration.
• Improved phpcs configuration.
• Check if program is available and if not display info to install it.
• Cache the output of linters, so subsequent calls are much faster.

v0.0.2 (2013-10-20)

• Fixes to the installer

v0.0.1 (2013-10-20)

• Initial commit with the basic functionalities. Released mainly to collect feedback about the features and the planned ideas.

git-lint's Issues

Add --commit option or similar

A way to check a particular commit is required to implement the integration between git lint and gerrit.

One option is to add the --commit option, that would allow to run git lint in the files commited in the specified commit, and as always would only count the lines specified in that particular commit. (This has the problem that we would need to somehow get the files of the commit).

Another option (probably easier to implement) would be to add an option --last-commit that checks the changes made in the last commit.

This would allow the flow:
git checkout <commit_id>
git lint --last-commit

Change ruby-lint configuration to not report undefined methods

Ruby-lint does not understand the code that well, so for rails it generates lots of false positives.

As a default, it would be good to remove this check.

git lint only for updated files

It would be great if you could have the option -u to do the git lint only to the updated files.

YAML validation seems having trouble to pass syntax `- !rel `

this is valid syntax though,

city_10_prop_subs_id:
fields:
name: subscription_id
value: 1
model: tests.jobs.fuel_sdk_proxy_classes:PropertyProxy

legacy_city_10_props:
fields:
properties:
-
- !rel city_10_prop_subs_id
model: tests.jobs.fuel_sdk_proxy_classes:RowResultProxy

Add default pylintrc configuration

Include a default pylintrc configuration

Add a configuration file for jshint

JSHint by default only reports the first 50 isssues, which is a huge problem for unhealthy codebases.

Add the following configuration file:

{
  "maxerr": 100000
}

Integrate HTML5 Tidy

The tools is found at https://w3c.github.io/tidy-html5/ and requires to use the package
template_remover to remove the markup.

$ remove_template.py filename.html | tidy -qe

Precommit hook fails for deleted files

When deleting/renaming a file the precommit hook will fail with the following message:
Error: File test/e2etest/data/ruby/error.rb does not exist

Add a formatted_message field to the JSON response

The JSON response has all the data to build a formatted response, but it would be better to just provide the default. So third party app, could use it directly without need of worrying of what fields are present.

Remove test dependencies

The current setup script requires 'nose' and 'mock', which are not required to run git-lint.

Repos with submodules result in fatal errors

When updating a submodule reference, git lint throws:

fatal: no such path 'path/to/module' in HEAD
Traceback (most recent call last):
  File "/usr/local/bin/git-lint", line 21, in <module>
    sys.exit(gitlint.main(sys.argv))
  File "/Library/Python/2.7/site-packages/gitlint/__init__.py", line 230, in main
    commit=commit)
  File "/Library/Python/2.7/site-packages/gitlint/git.py", line 132, in modified_lines
    ['git', 'blame', '--porcelain', filename]).split(
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['git', 'blame', '--porcelain', u'/absolute/path/to/submodule']' returned non-zero exit status 128

This is because running a git blame submodule-folder does not work.

Add coffelint support

Files that has been modified twice are not linted

In git when a file is modified twice, it gets the status MM which is not being considered.

Run e2e test in travis

For this we need to install all the dependencies.

Integrat java checkstyle with the google style guide

https://code.google.com/p/google-api-java-client/source/browse/checkstyle.xml

cache not invalidated on config changes

If .gitlint.yaml has been updated since the cache was written, the output cache should be invalidated. Otherwise, you could edit the config to eg ignore an error type, but it'll still be present when rerunning.

Add cppcheck to the default config

As requested in issue #43 add cppcheck (http://cppcheck.sourceforge.net/) to the default config.

Remove duplicate messsages

Sometimes different linters report the same message and hence is reported twice by the tool.

One such example is in Java with empty catch block

line 36: Avoid empty catch blocks
line 36: Avoid empty catch blocks

Installation fails from Pypi

The problem is that the README.rst file is not accesible.

$ pip install git-lint
Downloading/unpacking git-lint
  Downloading git-lint-0.0.3.1.tar.gz
  Running setup.py egg_info for package git-lint
    Traceback (most recent call last):
      File "<string>", line 16, in <module>
      File "/tmp/foox/build/git-lint/setup.py", line 21, in <module>
        long_description=open('README.rst').read(),
    IOError: [Errno 2] No such file or directory: 'README.rst'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 16, in <module>

  File "/tmp/foox/build/git-lint/setup.py", line 21, in <module>

    long_description=open('README.rst').read(),

IOError: [Errno 2] No such file or directory: 'README.rst'

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/foox/build/git-lint
Storing complete log in /home/skreft/.pip/pip.log

Provide mapping for severities and default severity

Some linters like pep8 or rubocop print the severity with a code or with just a letter.
For example pep8 has E123 and W456, whereas rubocop has C, W, E among others.

One alternative to normalize this would be to have something like this in the configuration file:

severities:
  - error: "E\\d+"
  - warning: "W\\d+"

Additionally, some linters do not provide a severity. In such cases it would be good to provide a default one.

default_severity: error

Check nix files

The syntax of nix files can be checked with nix-instantiate --parse file.nix. The output in case of failures will be something like:

error: syntax error, unexpected LET, expecting ':' or '@', at /home/skreft/code/issue127/default.nix:12:1

Add way to override configuration

Currently there is no way to modify git-lint's configuration, other than modifying the file that is shipped with the distribution.

Users should be allowed to define another config.yaml file in either their home directory, or in the root of the repo.

Names and exact locations have to be decided.

Add findbugs to the default config

As requested in issue #43 add findbugs (http://findbugs.sourceforge.net/) to the default config.

Add variable to configuration file that points to the root of the repository

This variable, that could be called $REPO_HOME, would be useful for custom configurations.

For example one could imagine to have a custom configuration like the following:

command: lint
args:

• --config=$REPO_HOME/linters/lint.ini

Add extra requirements to the configuration file

Currently tghe tool checks that the command is installed and if not prints how the tool can be installed.

However, there are some tools that are scripts and may have more than one dependency, the same applies for piped commands. In this case, git lint will run the command and will gracefully fail. The problem is that the output will be cached and the same output will be shown unless you touch the file.

Integrate HTML Lint

Use the tool html-linter to lint html files

Add cpplint to the default config

As requested in issue #43 add cpplint (https://google-styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py) to the default config.

Add pyflakes to the default config

As requested in issue #43 add Pyflakes to the default config.

locally clone prior to linting

I wrote an internal tool that provides some of the features of git-lint, but it's not well tested and coupled to our code base. I'm really excited about the possibility of scrapping it and switching over to git-lint!

Anywho, one thing my tool did was locally clone the entire git repo prior to linting. This fixed two problems for me:

1. prevented linting on uncommited changes in tracked files (eg after a git commit -p)
2. prevented linters from using context from uncommitted/untracked files

Filtering for changed lines addresses problem 1, but I'd like to always lint the entire file (so as not to miss eg unused imports).

As far as I can tell, problem 2 isn't addressed. Admittedly, it's a weird edge case: the only place I've seen it is in pylint's relative import checker, which needs the context from the rest of the package.

Are you opposed to adding an option that locally clones prior to linting? If not, do you have recommendations for where to implement it? My instinct is to pull the functions in the git module into a class that handles the local clone state.

No new line when skipping one of the linters

In html when tidy5 is not installed I'm getting the following output:

SKIPPED: tidy5 is not installed. Visit https://w3c.github.io/tidy-html5/line 22, col 1: Error: Separation of concerns: Javascript should be defined in its own file: Move the contents of this tag to its own JS file.

but it should be:

SKIPPED: tidy5 is not installed. Visit https://w3c.github.io/tidy-html5/
line 22, col 1: Error: Separation of concerns: Javascript should be defined in its own file: Move the contents of this tag to its own JS file.

Make Php CodeSniffer use the PSR standards

Php CodeSniffer should use the standards defined by PSR.

rubocop regular expression reports wrong lines

The RE used for rubocop is incorrect as it matches lines that it should not.

The problem is that the first component is being captured with a .*, then comes the line number and then optionally the column number. However, as the .* can match a :, the line number component matches also the column number, generating thus many false positives.

The solution is to generate the report with the option --format emacs which generates the report having an absolute path and with a line and column number.

configure to use --force by default

As mentioned in #32, I'd like to always run with --force. Right now, it looks like the only way to provide the option is on the command line. It'd be nice to be able to set it in per-project configuration instead.

Does caching works with branches or reverts?

Check what happens with the cached versions when changing branches or reverting (checking out) the changes.

I'm pretty sure there's a bug.

Add --commits command line option

For CI, it would be great to have a --commits options that allows one to specify a comma separated list of commits id to track.

Add an XML linter/validator

Normalize the output

All linters have different output and sometimes you have very long and verbose lines like:

/home/repo/my/file/being/linted line 5 col 5: Error: Constant not defined

And in some other cases the information is very succinct.

The idea is to just output the messges sorted and normalized. That is extract the relevant information from the original message and format it according to a defined pattern.

This is related to issue #23 .

Add way to match files based on complete filename and not only their extension.

Currently the rules for deciding what linters is going to be used only takes into account the extension of the file. This has several problems, one is that some files don't have an extension (for example, .gitlint), and in some other cases there are specific linters that can be used for some files. For example for travis.yml there is a specific linter, taht is only valid for that kind of files. The same happens for the phpunit configuration file, which can be called phpunit.xml.dist, in which case we only want to apply the xml linter to those files and not to all files ending with .dist.

Add --json option

Add a --json option that would output json instead of plain text.

This is required to integrate the output with gerrit.

The format could be something like:

{
  "comments": {
    "file1": [
      {
        "line": 1,
        "message": "[nit] Foo",
        "linter": "linter1"
      },
      {
        "message": "[nit] Bar",
        "linter": "linter2"
      }
    ],
    "file2": [
      {
        "message": "Please add documentation",
        "line": 5,
        "column": 0,
        "linter": "linter"
      }
    ]
  }
}

This output will also need to have a summary, whether the file had errors or not or if one of the linters was skipped.

Sort the linter output by line numbers

Currently the output of the linters is not sorted by line number, which is annoying as you have to go back and forth between different context in the target file. This is because the order of the linters is typically not sorted either. As an example below is the output of linting gitlint/linters.py:

git lint gitlint/linters.py --force
Linting file: gitlint/linters.py
line 54, col 0: Warning: [fixme]: TODO(skreft): add test case for result already in cache.
line 125, col 0: Warning: [fixme]: TODO(skreft): validate data['filter'], ie check that only has valid fields.
line 25, col 0: Warning: [no-init]: Partial: Class has no __init__ method
line 28, col 54: Error: [no-member]: Partial.__eq__: Instance of 'Partial' has no 'args' member
line 29, col 20: Error: [no-member]: Partial.__eq__: Instance of 'Partial' has no 'keywords' member
line 34, col 13: Error: [no-member]: Partial.__repr__: Instance of 'Partial' has no 'func' member
line 34, col 33: Error: [no-member]: Partial.__repr__: Instance of 'Partial' has no 'args' member
line 34, col 44: Error: [no-member]: Partial.__repr__: Instance of 'Partial' has no 'keywords' member
line 25, col 0: Refactor: [too-few-public-methods]: Partial: Too few public methods (0/1)
line 55, col 0: Refactor: [too-many-locals]: lint_command: Too many local variables (16/15)
line 34, col 13: [E128]: continuation line under-indented for visual indent

Add variable to configuration file that points to the location of the installation

It would be really useful to have a variable called $CONFIG or something, so we can distribute some default configuration files. Then in the configuration it could be used as:

command: linter
args:

• --config=$CONFIG/linter.ini

Integrate PMD for java

Integrate PMD for java

Filenames with spaces are not handled correctly

git lint
Linting file: "a b.py"
SKIPPED: no linter is defined or enabled for files with extension ".py""

This is because git status porcelain return the name of the files surrounded by quotes in case it has some spaces.

$ git status --porcelain
A  "a b.py"

Add a way to remove the cache

When you change the configuration and the reuslt is cached there's no way other than touching all the file to force a rerun.

Several options are possible:

• Add a flag to remove the cache
• If the configuration file changed then disable the cache
• Maybe the --force option could also disable the cache.

Integrate SASS linter

https://github.com/causes/scss-lint

Add --rails option to rubocop

rubocop has some validations for rails that can be activated with the --rails option.

Support the Pep257 linter for python docstrings

Add support for the PEP 257 linter (https://pypi.python.org/pypi/pep257).

Use format syntax in config file

Currently the config file uses the old interpolation syntax, which is harder to read.

Compare "%(DEFAULT_CONFIGS)s/jshint.json" with "{DEFAULT_CONFIGS}/jshint.json".

add linters pyflakes, cppcheck, cpplint

I have tried something similar to git-lint in the past, but had no time to continue on this:
https://github.com/tkruse/unilint
But the code to scape the output from the given tools could be reused.

Also Java linters should include Findbugs

Ruby linter

It would be great if git lint could have support for Ruby