Coder Social home page Coder Social logo

codacy-plugins-test's Introduction

Codacy Badge Build Status

Codacy Plugins Test

Provide a testing interface for the external docker tools.

Test definition

The test files should be placed in /docs/tests on the docker of the tool being tested.

Definition

//#Patterns: <PATTERN_NAME> : { "<PARAMETER_NAME>": "<PARAMETER_VALUE>" }

var people = {};
//#<PATTERN_LEVEL>: <PATTERN_NAME>
for (var i = 0, person; (person = people[i]); i++) {}

var variable;
function test() {
  //#<PATTERN_LEVEL>: <PATTERN_NAME>
  return (variable = "test");
}

Example:

//#Patterns: boss

var people = {};
//#Warn: boss
for (var i = 0, person; (person = people[i]); i++) {}

var variable;
function test() {
  //#Warn: boss
  return (variable = "test");
}

You can also look for multiple patterns in the same file, just separate them with a comma:

Example:

//#Patterns: big, boss
var people = {};
//#Warn: big
for (var i = 0, person; (person = people[i]); i++) {}

var variable;
function test() {
  //#Warn: boss
  return (variable = "test");
}

Instead of commenting in the line before the error, you can alternatively specify the line of the warning with this syntax:

<LANGUAGE_COMMENT>#Issue: {"severity": "<ERROR_LEVEL>", "line": <LINE_NUMBER_WITH_ISSUE>, "patternId": "PATTERN_ID"}

Example:

//#Patterns: design_tag_todo
//#Issue: {"severity": "Info", "line": 3, "patternId": "design_tag_todo"}

var people = {};
//TODO: remove empty for
for (var i = 0, person; (person = people[i]); i++) {}

var variable;

Multiple test definition

The multiple tests are defined inside the /docs/tests/multiple-tests/ directory of the tool's docker image being tested.

You can have a subdirectory of /docs/tests/multiple-tests/ for every run of the tool.

For example, you can add two subdirectories like these:

  • with-config-file
  • without-config-file

To check that the tool works with native configuration file (for example .eslintrc.json) and with Codacy configuration.

Each test folder should have a src directory (containining the source files to test and the tool's native configuration (if it exists)), two files named patterns.xml and results.xml with the following structure:

patterns.xml Structure

<?xml version="1.0" encoding="UTF-8"?>

<module name="root">
    <property name="tool-extra-parameter-name" value="tool-extra-parameter-value" />
    <module name="rule-name" />
    <module name="rule-with-parameters">
        <property name="parameter-key" value="parameter-value" />
    </module>
    <!-- To ignore config files from analysis -->
    <module name="BeforeExecutionExclusionFileFilter">
        <!-- value can be a regex matching files to ignore -->
        <property name="fileNamePattern" value="config-file\.xml"/>
    </module>
</module>

results.xml Structure

<?xml version="1.0" encoding="utf-8"?>
<checkstyle version="4.3">
    <file name="file-name.ext"> <!-- severity can be one of info, warning or error -->
        <error source="rule-name" line="20" message="reported message from the tool" severity="info|warning|error" />
    </file>
</checkstyle>

Usage

JsonTests

Checks if the patterns definitions are in the correct format

sbt "runMain codacy.plugins.DockerTest json <DOCKER_NAME>:<DOCKER_VERSION>"

Options:

  • codacy.tests.ignore.descriptions - if this variable is defined we do not check if all the patterns have descriptions

PatternTests

Check if all the patterns defined in the test files occur in the specified line

sbt "runMain codacy.plugins.DockerTest pattern <DOCKER_NAME>:<DOCKER_VERSION>"

Options:

  • codacy.tests.languages - languages supported by the tool. If this option isn't provided, the languages will be inferred from the test files. Example: -Dcodacy.tests.languages=ruby,java,javascript

Alternatively, you can run a specific test file:

sbt "runMain codacy.plugins.DockerTest pattern <DOCKER_NAME>:<DOCKER_VERSION> no-curly-brackets"

MetricsTests

Check if the metrics defined in the test files match with same complexieties in the specified lines

sbt "runMain codacy.plugins.DockerTest metrics <DOCKER_NAME>:<DOCKER_VERSION>"

MultipleTests

Check if the tool runs with multiple patterns and test files at the same time and configuration file behavior as well

sbt "runMain codacy.plugins.DockerTest multiple <DOCKER_NAME>:<DOCKER_VERSION>"

Options:

  • --only by adding this flag, followed by the folder name, you can run a single test of the multiple folder sbt "runMain codacy.plugins.DockerTest multiple <DOCKER_NAME>:<DOCKER_VERSION>" --only <folderName>

Docs

Information about the integration with external analysis tools at Codacy available here.

Troubleshooting

OSx

Change the java tmp dir to your home so that boot2docker can access the tmp files

-Djava.io.tmpdir=$HOME/tmp

What is Codacy?

Codacy is an Automated Code Review Tool that monitors your technical debt, helps you improve your code quality, teaches best practices to your developers, and helps you save time in Code Reviews.

Among Codacy’s features:

  • Identify new Static Analysis issues
  • Commit and Pull Request Analysis with GitHub, BitBucket/Stash, GitLab (and also direct git repositories)
  • Auto-comments on Commits and Pull Requests
  • Integrations with Slack, HipChat, Jira, YouTrack
  • Track issues in Code Style, Security, Error Proneness, Performance, Unused Code and other categories

Codacy also helps keep track of Code Coverage, Code Duplication, and Code Complexity.

Codacy supports PHP, Python, Ruby, Java, JavaScript, and Scala, among others.

Free for Open Source

Codacy is free for Open Source projects.

License

Licensed under the MIT License terms.

codacy-plugins-test's People

Contributors

andreatp avatar carlospinho04 avatar caxaria avatar codacy-vrhpires avatar h314to avatar jllopes avatar josemiguelmelo avatar kidsoncoffee avatar ljmf00 avatar lolgab avatar lventura-codacy avatar machadoit avatar miguelmarcelino avatar mrfyda avatar pedrocodacy avatar rrdelaney avatar rtfpessoa avatar rubencodacy avatar tfrgoncalves avatar tiagonbotelho avatar xplosunn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

codacy-plugins-test's Issues

[CY-2542] Add integration and unit tests

We should have unit tests and integration tests based on our example tools.

All the modules should be tested:

  • multiple tests
  • metrics tests
  • duplication tests
  • pattern tests
  • json tests
  • plugin tests

Try to make tests parallel

Tests take a lot of time to execute. Instead of running tests for every file sequentially, it should run them in parallel.

Temporary resources should be disposed

This is what I'm getting on my /tmp folder:

-rw-r--r--  1 luis luis      157 Nov 19 17:08  codacy-config1007526028416712285.json
-rw-r--r--  1 luis luis      200 Nov 18 16:05  codacy-config10747433491757880506.json
-rw-r--r--  1 luis luis       79 Nov 19 17:25  codacy-config10900967276462879896.json
-rw-r--r--  1 luis luis      217 Nov 18 16:05  codacy-config11410438314241888228.json
-rw-r--r--  1 luis luis      149 Nov 18 16:05  codacy-config11822540078454293596.json
-rw-r--r--  1 luis luis       79 Nov 19 17:25  codacy-config11911206478924219462.json
-rw-r--r--  1 luis luis       79 Nov 19 19:21  codacy-config12434335306641727915.json
-rw-r--r--  1 luis luis       79 Nov 19 19:15  codacy-config1265006343722513455.json
-rw-r--r--  1 luis luis      265 Nov 18 16:04  codacy-config12670940352005711481.json
-rw-r--r--  1 luis luis      209 Nov 18 16:04  codacy-config127891875580660594.json
-rw-r--r--  1 luis luis       79 Nov 19 19:19  codacy-config1287372457872075850.json
-rw-r--r--  1 luis luis       98 Nov 19 17:35  codacy-config1300165936894879169.json
-rw-r--r--  1 luis luis      193 Nov 18 16:04  codacy-config13434302983527744820.json
-rw-r--r--  1 luis luis       79 Nov 19 17:37  codacy-config13555396985233648836.json
-rw-r--r--  1 luis luis      148 Nov 18 16:05  codacy-config1393972590260023534.json
-rw-r--r--  1 luis luis       98 Nov 19 17:25  codacy-config14474274534026548549.json
-rw-r--r--  1 luis luis       79 Nov 19 19:14  codacy-config15084458851814817938.json
-rw-r--r--  1 luis luis       98 Nov 19 19:19  codacy-config15098556566484717069.json
-rw-r--r--  1 luis luis      152 Nov 18 16:05  codacy-config15271630667541655195.json
-rw-r--r--  1 luis luis      159 Nov 18 16:05  codacy-config15691328719902239136.json
-rw-r--r--  1 luis luis       98 Nov 19 17:37  codacy-config15821327109610682986.json
-rw-r--r--  1 luis luis      149 Nov 18 16:05  codacy-config15869809963288542345.json
-rw-r--r--  1 luis luis       98 Nov 19 19:15  codacy-config16067553781456026966.json
-rw-r--r--  1 luis luis       98 Nov 19 19:21  codacy-config16839850794640517327.json
-rw-r--r--  1 luis luis       79 Nov 19 17:37  codacy-config1689094944599038493.json
-rw-r--r--  1 luis luis       79 Nov 19 16:58  codacy-config17992669195186763846.json
-rw-r--r--  1 luis luis      222 Nov 19 15:14  codacy-config18015813333882120689.json
-rw-r--r--  1 luis luis      124 Nov 18 16:05  codacy-config18250713333072630703.json
-rw-r--r--  1 luis luis      199 Nov 18 16:05  codacy-config1831785825699012283.json
-rw-r--r--  1 luis luis       79 Nov 19 19:22  codacy-config2486525055355653683.json
-rw-r--r--  1 luis luis      120 Nov 18 16:04  codacy-config3190501608293627513.json
-rw-r--r--  1 luis luis      144 Nov 18 16:05  codacy-config3210097122543542943.json
-rw-r--r--  1 luis luis       98 Nov 19 19:22  codacy-config3444547403911708746.json
-rw-r--r--  1 luis luis       79 Nov 19 16:58  codacy-config3902639737347313187.json
-rw-r--r--  1 luis luis       79 Nov 19 19:15  codacy-config421743889396390786.json
-rw-r--r--  1 luis luis      171 Nov 18 16:05  codacy-config4400560258477692330.json
-rw-r--r--  1 luis luis      146 Nov 18 16:05  codacy-config545230600222820656.json
-rw-r--r--  1 luis luis       79 Nov 19 19:21  codacy-config5460487036895619913.json
-rw-r--r--  1 luis luis       79 Nov 19 17:35  codacy-config55515038679635350.json
-rw-r--r--  1 luis luis      200 Nov 18 16:05  codacy-config5839303068815253982.json
-rw-r--r--  1 luis luis       87 Nov 19 17:08  codacy-config5845559959395570924.json
-rw-r--r--  1 luis luis      152 Nov 19 15:14  codacy-config6053448131594488476.json
-rw-r--r--  1 luis luis       98 Nov 19 16:58  codacy-config6272942676596234653.json
-rw-r--r--  1 luis luis      134 Nov 18 16:05  codacy-config6419809868952686999.json
-rw-r--r--  1 luis luis      165 Nov 18 16:05  codacy-config6600843440668169142.json
-rw-r--r--  1 luis luis      136 Nov 18 16:05  codacy-config7167663024977500651.json
-rw-r--r--  1 luis luis      116 Nov 18 16:04  codacy-config7227064179414984144.json
-rw-r--r--  1 luis luis       79 Nov 19 19:14  codacy-config7285628020022164388.json
-rw-r--r--  1 luis luis       79 Nov 19 17:35  codacy-config7694785198187634367.json
-rw-r--r--  1 luis luis       79 Nov 19 19:19  codacy-config9090325255378247049.json
-rw-r--r--  1 luis luis       98 Nov 19 19:14  codacy-config9403930691344897162.json
-rw-r--r--  1 luis luis      162 Nov 18 16:05  codacy-config9440793196503670134.json
-rw-r--r--  1 luis luis      132 Nov 18 16:05  codacy-config9705875068491258255.json
-rw-r--r--  1 luis luis      238 Nov 18 16:05  codacy-config9734121320401609171.json
-rw-r--r--  1 luis luis      223 Nov 19 15:14  codacy-config9924484628529014914.json

I assume this resources should be disposed on exit.

Error running plugins-test using sbt

The logging framework seems having some issues at the end, when using sbt to run the tests.

Since we run the native image, this is consider a mid level issue. Consider using it to avoid errors.

[warn] Thread[pool-2-thread-1,5,main] loading wvlet.log.LogLevel$ after test or run has completed. This is a likely resource leak
2019-11-27 17:12:22,104 pool-2-thread-1 ERROR Unable to unregister MBeans java.lang.NoClassDefFoundError: wvlet/log/LogLevel$
	at wvlet.log.LogRecord$.apply(LogRecord.scala:32)
	at wvlet.log.LogFormatter.format(LogFormat.scala:32)
	at wvlet.log.LogFormatter.format$(LogFormat.scala:29)
	at wvlet.log.LogFormatter$SimpleLogFormatter$.format(LogFormat.scala:120)
	at wvlet.log.ConsoleLogHandler.publish(Handler.scala:13)
	at java.util.logging.Logger.log(Logger.java:738)
	at java.util.logging.Logger.doLog(Logger.java:765)
	at java.util.logging.Logger.logp(Logger.java:931)
	at com.sun.jmx.mbeanserver.Repository.retrieve(Repository.java:482)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
	at org.apache.logging.log4j.core.jmx.Server.unregisterAllMatching(Server.java:345)
	at org.apache.logging.log4j.core.jmx.Server.unregisterLoggerContext(Server.java:261)
	at org.apache.logging.log4j.core.jmx.Server.unregisterLoggerContext(Server.java:249)
	at org.apache.logging.log4j.core.LoggerContext.stop(LoggerContext.java:340)
	at org.apache.logging.log4j.core.LoggerContext$1.run(LoggerContext.java:281)
	at org.apache.logging.log4j.core.util.DefaultShutdownCallbackRegistry$RegisteredCancellable.run(DefaultShutdownCallbackRegistry.java:109)
	at org.apache.logging.log4j.core.util.DefaultShutdownCallbackRegistry.run(DefaultShutdownCallbackRegistry.java:74)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: wvlet.log.LogLevel$
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at sbt.internal.ManagedClassLoader$ZombieClassLoader.lookupClass(ManagedClassLoader.java:48)
	at sbt.internal.ManagedClassLoader.findClass(ManagedClassLoader.java:92)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 21 more

NoSuchElementException on empty output stream

[error] (run-main-0) java.util.NoSuchElementException: head of empty stream
[error] java.util.NoSuchElementException: head of empty stream
[error]         at scala.collection.immutable.Stream$Empty$.head(Stream.scala:1106)
[error]         at scala.collection.immutable.Stream$Empty$.head(Stream.scala:1104)
[error]         at codacy.plugins.test.DockerHelpers$.testFoldersInDocker(DockerHelpers.scala:19)
[error]         at codacy.plugins.DockerTest$.$anonfun$main$4(DockerTest.scala:30)
[error]         at codacy.plugins.DockerTest$.$anonfun$main$4$adapted(DockerTest.scala:27)
[error]         at scala.Option.fold(Option.scala:251)
[error]         at codacy.plugins.DockerTest$.$anonfun$main$2(DockerTest.scala:27)
[error]         at codacy.plugins.DockerTest$.$anonfun$main$2$adapted(DockerTest.scala:24)
[error]         at scala.Option.fold(Option.scala:251)
[error]         at codacy.plugins.DockerTest$.main(DockerTest.scala:24)
[error]         at codacy.plugins.DockerTest.main(DockerTest.scala)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]         at java.lang.reflect.Method.invoke(Method.java:498)
[error] Nonzero exit code: 1

Running the same test twice getting different results

luis@cactus ~/Workspace/Programming/Company/Github/codacy/codacy-plugins-test 130 ► sbt "runMain codacy.plugins.DockerTest multiple codacy-sonar-csharp:dev"
[info] Loading settings for project codacy-plugins-test-build from plugins.sbt ...
[info] Loading project definition from /home/luis/Workspace/Programming/Company/Github/codacy/codacy-plugins-test/project
[info] Loading settings for project codacy-plugins-test from build.sbt ...
[info] Set current project to codacy-plugins-test (in build file:/home/luis/Workspace/Programming/Company/Github/codacy/codacy-plugins-test/)
[info] running codacy.plugins.DockerTest multiple codacy-sonar-csharp:dev
[MultipleTests] Running MultipleTests:
[MultipleTests] without-config-file should have 1 results
[ResultPrinter] Got 1 results.
[MultipleTests] with-config-file should have 3 results
[ResultPrinter] Tool results don't match expected results:
[ResultPrinter] Extra: 
[ResultPrinter] Set()
[ResultPrinter] Missing:
[ResultPrinter] Set()
[ResultPrinter] Got 3 results.
[DockerTest] [Success] MultipleTests$
[DockerTest] [Success] All tests passed!
[success] Total time: 9 s, completed Nov 19, 2019, 7:59:17 PM
luis@cactus ~/Workspace/Programming/Company/Github/codacy/codacy-plugins-test ► sbt "runMain codacy.plugins.DockerTest multiple codacy-sonar-csharp:dev"
[info] Loading settings for project codacy-plugins-test-build from plugins.sbt ...
[info] Loading project definition from /home/luis/Workspace/Programming/Company/Github/codacy/codacy-plugins-test/project
[info] Loading settings for project codacy-plugins-test from build.sbt ...
[info] Set current project to codacy-plugins-test (in build file:/home/luis/Workspace/Programming/Company/Github/codacy/codacy-plugins-test/)
[info] running codacy.plugins.DockerTest multiple codacy-sonar-csharp:dev
[MultipleTests] Running MultipleTests:
[MultipleTests] without-config-file should have 1 results
[ResultPrinter] Got 1 results.
[MultipleTests] with-config-file should have 3 results
[ResultPrinter] Got 3 results.
[DockerTest] [Success] MultipleTests$
[DockerTest] [Success] All tests passed!
[success] Total time: 6 s, completed Nov 19, 2019, 7:59:42 PM

Files passed to `multiple` test are not being filtered

In case you have a configuration file or another file that is not intended to be analyzed by the tool inside /docs/multiple-tests/with-config-file/src/, it will be passed as a language file.

For example: You have config.xml and test.cs, for a C# tool, you may only pass the language files which is test.cs and filter config.xml or another useless file.

This may report an error or bad results on some tools.

Results are being filtered by codacy-plugins

Results are being filtered by codacy-plugins based on files and by removing duplicates.

This can hide some problems with the tools, e.g., if the tool runs too many times, the test will not fail because we remove duplicates.

Acceptance Criteria:

  • Tools can have tests that validate the tool output directly (without it being filtered by codacy-plugins)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.