Support the analysis of JS code snippets within any html files (Polymer, JSP, PHP, JSF, ...) about sonarjs

[chkpnt]

Don't know what's an MMF, but sounds good.

Please keep in mind it's nowadays not only about JavaScript but TypeScript, too. I suggest to respect the lang-attribute:
The content of <script lang="ts">...</script> or <script lang="typescript">...</script> should be delegated to the TypeScript-scanner.

[inverno]

@fmallet I think that analyzing the code snippets within any html file is anyway a good first step. This way we also bring value beyond polymer and it's similar to the work needed to support Vue.js

[vilchik-elena]

@exhuma unfortunately yet no news on this feature. It's not a simple one as it's "breaks" our current approach: one file - one script.

[exhuma]

In case someone else comes across this before the support is integrated in SonarJS, I've written a small Python script to extract <script> tags into .js files. This can be run before running sonar-scanner so it will find .js files. Note that this requires the package beautifulsoup4 to work. It should be fairly self-explanatory and should also be easy to adapt it to your needs. This current version expects exactly one <script> tag. If you have more it will definitely need to be modified.

For those that don't know Python well: It is possible to create a "virtual environment" containing a separate Python interpreter with isolated packages/dependencies, so it is possible to use this without installing beautifulsoup as root. To do this, use one of the following commands:

pyvenv /path/to/isolated/environment
python -m venv  /path/to/isolated/environment   # if the previous does not work or does not exist

Then you can run:

/path/to/isolated/environment/bin/pip install beautifulsoup4

This installs the dependency to beautifulsoup into that environment. To run the script below, run:

/path/to/isolated/environment/bin/python extract.py --help

Here is the script:

'''
This module defines a main entry point which will recursively walk over all
HTML files in a folder and extract the first <script> tag it finds. This script
tag will be saved as a new ".js" file which sits right along-side the original
HTML file.

The script's main purpose is to make JavaScript code available to SonarQube
which currently cannot read <script> tags in HTML files.
'''
import logging
from argparse import ArgumentParser
from os import walk
from os.path import join, abspath

from bs4 import BeautifulSoup

LOG = logging.getLogger(__name__)


def extract_js(filename):
    '''
    Extract the first script tag from *filename* and save it to <filename>.js
    '''
    with open(filename) as fptr:
        html_doc = fptr.read()
    soup = BeautifulSoup(html_doc, 'html.parser')
    script = soup.script
    if not script or not script.string or not script.string.strip():
        LOG.debug('Skipping %r: Empty or no <script> tag in', filename)
        return
    outfile = filename + '.js'
    with open(outfile, 'w') as fptr:
        print(script.string, file=fptr)
        LOG.info('Written %r.', abspath(outfile))

def main(root):
    '''
    Recursively walk over *root* and extract JS code.
    '''
    LOG.info('Extracting <script> tags from HTML files in %r', abspath(root))
    for root, dirs, files in walk(root):
        for basename in files:
            filename = join(root, basename)
            if not basename.endswith('.html'):
                LOG.debug('Skipping %r: Not a HTML file!', filename)
                continue
            extract_js(filename)

def parse_args():
    '''
    Parse CLI arguments
    '''
    parser = ArgumentParser()
    parser.add_argument(
        'root_path',
        help='The root-path which is used to find input files.')
    return parser.parse_args()


if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)
    args = parse_args()
    main(args.root_path)

[vilchik-elena]

Closing this ticket as feature is tracked by MMF

[304NotModified]

Direct link: https://jira.sonarsource.com/browse/MMF-2427

[fmallet]

By supporting Polyme @FridaCai, do you mean supporting the analysis of JavaScript code snippets part of any HTML files or are you thinking about anything really specific to the Polymer framework ? Thanks

[fmallet]

Agreed @inverno and that's also my feeling, I would just like to make sure that @FridaCai was not thinking about anything else.

[ganncamp]

Reaction to parsing contents of random <script> tags in random files: +10

[fmallet]

@inverno and @ganncamp, does that mean that this issue title should be updated to something like "Analyze JS code located between <script></script> tags in any HTML files/templates" ?

[ganncamp]

IMO, @fmallet it should be "...in any file".

[inverno]

Yes... in fact, I wonder if instead of a parsing approach (which would have us define the grammar of the hosting file), we could do something very rough and very generic, extracting the content of paired <script/> tags wherever they appear (even when present in formats we have no clue of).

…

On Tue, Jun 13, 2017 at 1:26 PM G. Ann Campbell ***@***.***> wrote: IMO, @fmallet <https://github.com/fmallet> it should be "...in any file". — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#581 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAeUwMevc7W5vN88nrCuD-Ofw9dKZtUxks5sDnHfgaJpZM4NM-4s> .

[vilchik-elena]

@inverno @ganncamp what are the examples of js code inside <script> but file not being html?

[fmallet]

@vilchik-elena, we're thinking about JSP, JSF, PHP, ... pages

[ganncamp]

What he said ^

[inverno]

Also, less frequent, but : js embedded in XML

[ilawrencewu]

As mentioned this question, I installed the SonarJS plugin in SonarQube, but the scanned result is
ERROR: Unable to parse file: /data/jenkins/workspace/develop/vueportal/lib/components/picker/src/picker-slot.vue
ERROR: Parse error at line 111 column 1:

101: overflow: visible
102: }
103:
104: .picker-3d .picker-item {
105: transform-origin: center center;
106: backface-visibility: hidden;
107: transition-timing-function: ease-out
108: }
109: </style>
110:
111: <script type="text/babel">
^
112: import draggable from './draggable'
113: import translateUtil from './translate'
114: import { once, addClass, removeClass } from './dom'
115: import emitter from './emitter'
116: import Vue from 'vue'
117: if (!Vue.prototype.$isServer) {
118: require('raf.js')
119: }
120:
121: var rotateElement = function (element, angle) {....}

So, what's the matter ?

[vilchik-elena]

@ilawrencewu Answered in issue #843 you've created. Next time please post your problem once.

[ilawrencewu]

@vilchik-elena sure, thanks

[exhuma]

Is there any update on this? I just wanted to scan a Polymer project only to realise that it skips the code almost completely. We have around 30 Polymer projects and I would really like to get a feel for the technical debt in those. sonar-qube would provide a really helpful tool for this.

I can see that this issue was removed from the 4.1 milestone recently which is kind of a bummer. I was waiting for this to hit.

Why was it removed?

[exhuma]

I understand. Thanks for the clarification. I'll keep an eye on this thread 😉

[chkpnt]

@inverno @ganncamp what are the examples of js code inside <script> but file not being html?

And .svelte for @sveltejs files.

[vilchik-elena]

FYI we have started drafting this MMF, but nothing planned