microsoft / applicationinspector

A source code analyzer built for surfacing features of interest and other characteristics to answer the question 'What's in the code?' quickly using static analysis with a json based rules engine. Ideal for scanning components before use or detecting feature level changes.

Introduction

Microsoft Application Inspector is a software source code characterization tool that helps identify coding features of first or third party software components based on well-known library/API calls and is helpful in security and non-security use cases. It uses hundreds of rules and regex patterns to surface interesting characteristics of source code to aid in determining what the software is or what it does from what file operations it uses, encryption, shell operations, cloud API's, frameworks and more and has received industry attention as a new and valuable contribution to OSS on ZDNet, SecurityWeek , CSOOnline , Linux.com/news , HelpNetSecurity, Twitter and more and was first featured on Microsoft.com.

Application Inspector is different from traditional static analysis tools in that it doesn't attempt to identify "good" or "bad" patterns; it simply reports what it finds against a set of over 400 rule patterns for feature detection including features that impact security such as the use of cryptography and more. This can be extremely helpful in reducing the time needed to determine what Open Source or other components do by examining the source directly rather than trusting to limited documentation or recommendations.

The tool supports scanning various programming languages including C, C++, C#, Java, JavaScript, HTML, Python, Objective-C, Go, Ruby, PowerShell and more and can scan projects with mixed language files. It supports generating results in HTML, JSON and text output formats with the default being an HTML report similar to the one shown here.

Be sure to see our complete project wiki page https://Github.com/Microsoft/ApplicationInspector/wiki for additional information and help.

Quick Start

Obtain Application Inspector

.NET Tool (recommended)

• Download and install the .NET 6 SDK
• Run dotnet tool install --global Microsoft.CST.ApplicationInspector.CLI

See more in the wiki

Platform Dependent Binary

• Download Application Inspector by selecting the pre-built package for the operating system of choice shown under the Assets section of the Releases.

Run Application Inspector

• Nuget Tool: appinspector analyze -s path/to/src.
• Platform Specific: applicationinspector.cli.exe analyze -s path/to/src

Goals

Microsoft Application Inspector helps you in securing your applications from start to deployment.

Design Choices - Enables you to choose which components meet your needs with a smaller footprint of unnecessary or unknowns features for keeping your application attack surface smaller as well as help to verify expected ones i.e. industry standard crypto only.

Identifying Feature Deltas - Detects changes between component versions which can be critical for detecting injection of backdoors.

Automating Security Compliance Checks - Use to identify components with features that require additional security scrutiny, approval or SDL compliance as part of your build pipeline or create a repository of metadata regarding all of your enterprise application.

Contribute

We have a strong default starting base of Rules for feature detection. But there are many feature identification patterns yet to be defined and we invite you to submit ideas on what you want to see or take a crack at defining a few. This is a chance to literally impact the open source ecosystem helping provide a tool that everyone can use. See the Rules section of the wiki for more.

Official Releases

Application Inspector is in GENERAL AUDIENCE release status. Your feedback is important to us. If you're interested in contributing, please review the CONTRIBUTING.md.

Application Inspector is available as a command line tool or NuGet package and is supported on Windows, Linux, or MacOS.

Platform specific binaries of the ApplicationInspector CLI are available on our GitHub releases page.

The C# library is available on NuGet as Microsoft.CST.ApplicationInspector.Commands .

The .NET Global Tool is available on NuGet as Microsoft.CST.ApplicationInspector.CLI.

If you use the .NET Core version, you will need to have .NET 6.0 or later installed. See the JustRunIt.md or Build.md files for more.

CLI Usage Information

> appinspector --help
ApplicationInspector.CLI 1.8.4-beta+976ee3cdd1
c Microsoft Corporation. All rights reserved.

  analyze        Inspect source directory/file/compressed file (.tgz|zip)
                 against defined characteristics

  tagdiff        Compares unique tag values between two source paths

  exporttags     Export the list of tags associated with the specified rules.
                 Does not scan source code.

  verifyrules    Verify custom rules syntax is valid

  packrules      Combine multiple rule files into one file for ease in
                 distribution

  help           Display more information on a specific command.

  version        Display version information.

Examples:

Command Help

To get help for a specific command run appinspector <command> --help.

Analyze Command

The Analyze Command is the workhorse of Application Inspector.

Simple Default Analyze

This will produce an output.html of the analysis in the current directory using default arguments and rules.

appinspector analyze -s path/to/files

Output Sarif

appinspector analyze -s path/to/files -f sarif -o output.sarif

Excluding Files using Globs

This will create a json output named data.json of the analysis in the current directory, excluding all files in test and .git folders using the provided glob patterns.

appinspector analyze -s path/to/files -o data.json -f json -g **/tests/**,**/.git/**

Additional Usage Information

For additional help on use of the console interface see CLI Usage.

For help using the NuGet package see NuGet Support

Build Instructions

See build.md

applicationinspector's Issues

Can't analyze paths with spaces in names

Describe the bug
When trying to analyze source directories with spaces in the names, windows appends double quotes to the beginning and end of the path.

To Reproduce
Steps to reproduce the behavior:

1. Create a folder with spaces in the name
2. Run AppInspector.exe analyze --source "C:\users\bgates\source\repos\My Badly Named Project\"
3. See error in log. It is ignoring the double quotes and terminates the string evaluation at the space

Expected behavior
Evaluate the project in the badly named directory

Additional context
Workaround-- Take the spaces out of the project directory's name

Log file level exception

If user specifies invalid log file level, v1.0.10 catches it but attempts to write the error to the log which has not yet been successfully created thus causing an additional exception for invalid object.

Exclude sample code from scan results

Folders with sample, example or test code should not be included as features in a project. The primary objective of the tools is the application features not ancillary files.

Add More Support for Compressed Package Types

Application Inspector currently supports .zip, .tgz, .tar, .gz, .jar source files.

The enhancement is to add support for p7zip, rar, and bzip2.

Dart Support

Please provide Dart Langugae support.

Assess and drive down false positive rate < 10% (min) < 5% (preferred)

Perform new rounds and level of testing to reassess current level of false positives and drive down (as needed) within acceptable margins of error i.e. 10% minimum overall and 5% for high confidence patterns (minimum).

Add a guideline to describe how to compile and use in linux

Insufficient memory to continue the execution of large program.

Tried running this against a large project, and I get an Insufficient memory error. Looking at task manager, the .NET Core host process gets up to about 3GB and then fails. Computer still has plenty of memory available

log.txt

General inspection of rules coverage per language

Several language specific rule patterns exists in addition to the common set. Ensure good coverage of rules across all supported languages that are indicated in the tool.

Package ApplicationInspector as Nuget package to consume in other .NET based projects

It would be great to have ApplicationInspector packaged as a Nuget library, so that other .NET projects can consume it's core functionality programmatically.

Getting an OutOfMemoryException

Describe the bug
I am running AppInspector.exe analyze -s "D:\Work\HL\Half-Life VR" -f html -v TRACE on the code in this repo: https://github.com/maxvollmer/Half-Life-VR

After about 31% it dies saying A runtime error has occured. Please see log file for more information.. The log then reveals that an OutOfMemoryException was thrown while trying to load a file. Unfortunately even with -v TRACE it doesn't say which file it was trying to load, probably because the exception kills the process before the file name would've been printed in the trace output.

To Reproduce
Steps to reproduce the behavior:

1. Clone https://github.com/maxvollmer/Half-Life-VR
2. Run AppInspector.exe analyze -s "<path to where you cloned the repo>" -f html -v TRACE

Expected behavior
Ideally no exception at all, and at least have -v TRACE print the filename before it starts doing things that might go wrong. Or when catching the exception, print the filename in the error output, so users don't need to run with TRACE at all to know which file caused an error.

I also saw that it skips files it considers too big, so maybe the threshold for this skip needs adjustment.

Screenshots
cmdline output:

Microsoft Application Inspector 1.0.23
Analyze command running
31% source files processedA runtime error has occured. Please see log file for more information.

log.txt:

2020-01-25T11:17:17 1 ERROR - A runtime error has occured. Please see log file for more information.
2020-01-25T11:17:17 1 ERROR - Runtime error: Exception of type 'System.OutOfMemoryException' was thrown. at System.Text.StringBuilder.ToString()
at System.IO.StreamReader.ReadToEnd()
at System.IO.File.InternalReadAllText(String path, Encoding encoding)
at System.IO.File.ReadAllText(String path)
at Microsoft.AppInspector.AnalyzeCommand.ProcessAsFile(String filename)
at Microsoft.AppInspector.AnalyzeCommand.Run()
at Microsoft.AppInspector.Program.RunAnalyzeCommand(AnalyzeCommandOptions opts)
at Microsoft.AppInspector.Program.<>c.

b__6_0(AnalyzeCommandOptions opts)
at CommandLine.ParserResultExtensions.MapResult[T1,T2,T3,T4,T5,TResult](ParserResult1 result, Func2 parsedFunc1, Func2 parsedFunc2, Func2 parsedFunc3, Func2 parsedFunc4, Func2 parsedFunc5, Func`2 notParsedFunc)
at Microsoft.AppInspector.Program.Main(String[] args)

Desktop:

• OS: Windows 10
• Browser: NA
• Version: NA

Smartphone:

• NA

Additional context
NA

tagtest command throwing Newtonsoft json reader exception

Describe the bug
I wanted to use the tagtest feature against a custom rule set. I took one rule from the default rule set into my own file to test this against an application.

analyze seems to work against my project

To Reproduce
Steps to reproduce the behavior:

1. Use the command in PowerShell 5.1.18362.145

.\AppInspector.exe tagtest -s $projectDir -r "C:\temp\\myrules\one_custom_rule.json" --log-file-path $logDir --log-file-level Info  --console-verbosity high

2. run it against any project
3. See error
   the following logs I got:

2020-01-24T13:43:32 1 INFO - [1/24/2020 2:43:32 PM] //////////////////////////////////////////////////////////
2020-01-24T13:43:32 1 INFO - Rule added: AI000700,Cloud Service: Hosting (Microsoft Azure),Cloud Service: Hosting (Microsoft Azure)
2020-01-24T13:43:32 1 INFO - Rule added: AI000700,Cloud Service: Hosting (Microsoft Azure),Cloud Service: Hosting (Microsoft Azure)
2020-01-24T13:43:32 1 WARN - no support for compressed type: .targets
2020-01-24T13:43:32 1 WARN - no support for compressed type: .targets
2020-01-24T13:43:32 1 WARN - no support for compressed type: .targets
2020-01-24T13:43:32 1 WARN - no support for compressed type: .targets
2020-01-24T13:43:32 1 ERROR - A runtime error has occured.  Please see log file for more information.
2020-01-24T13:43:32 1 ERROR - Runtime error: Unexpected character encountered while parsing value: M. Path '', line 0, position 0.    at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.JsonTextReader.Read()
   at Newtonsoft.Json.JsonReader.ReadAndMoveToContent()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at Microsoft.AppInspector.TagTestCommand.Run()
   at Microsoft.AppInspector.Program.RunTagTestCommand(TagTestCommandOptions opts)
   at Microsoft.AppInspector.Program.<>c.<Main>b__6_2(TagTestCommandOptions opts)
   at CommandLine.ParserResultExtensions.MapResult[T1,T2,T3,T4,T5,TResult](ParserResult`1 result, Func`2 parsedFunc1, Func`2 parsedFunc2, Func`2 parsedFunc3, Func`2 parsedFunc4, Func`2 parsedFunc5, Func`2 notParsedFunc)
   at Microsoft.AppInspector.Program.Main(String[] args)

Expected behavior
That I see a positive or negative answer if the feature is included in my solution or not.

Desktop (please complete the following information):

• OS: Windows 10
• Version 18363.592

Additional context
The file one_custom_rule.json contains the following JSON

[
  {
    "name": "Cloud Service: Hosting (Microsoft Azure)",
    "id": "AI000700",
    "description": "Cloud Service: Hosting (Microsoft Azure)",
    "tags": [
      "Metadata.CloudServices.Hosting.Microsoft.Azure"
    ],
    "severity": "moderate",
    "patterns": [
      {
        "pattern": "azure",
        "type": "string",
        "scopes": [ "code" ],
        "modifiers": [ "i" ],
        "confidence": "high"
      }
    ]
  } 
]

Turn this into a dotnet global tool?

It would be great if this could be installed as a dotnet global tool.

That would remove a lot of the friction required to get up and running with the inspector.

Error running analyze command

Describe the bug
Trying to run the analyze command I get an error at the end of the process.

To Reproduce
Steps to reproduce the behavior:
executed command

dotnet AppInspector.dll analyze -s [path] -l ff.log

at the end of the process this error occurs:

2020-01-20T21:03:37 1 ERROR - A runtime error has occured.  Please see log file for more information.
2020-01-20T21:03:37 1 ERROR - Runtime error: Unexpected EOF    at ICSharpCode.SharpZipLib.Zip.Compression.Streams.InflaterInputStream.Fill()
   at ICSharpCode.SharpZipLib.Zip.Compression.Streams.InflaterInputStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at ICSharpCode.SharpZipLib.GZip.GZipInputStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at ICSharpCode.SharpZipLib.Tar.TarBuffer.ReadRecord()
   at ICSharpCode.SharpZipLib.Tar.TarBuffer.ReadBlock()
   at ICSharpCode.SharpZipLib.Tar.TarInputStream.GetNextEntry()
   at Microsoft.AppInspector.AnalyzeCommand.GetTarGzFileCount(String filename)
   at Microsoft.AppInspector.AnalyzeCommand.ProcessTarGzFile(String filename)
   at Microsoft.AppInspector.AnalyzeCommand.UnZipAndProcess(String filename)
   at Microsoft.AppInspector.AnalyzeCommand.Run()
   at Microsoft.AppInspector.Program.RunAnalyzeCommand(AnalyzeCommandOptions opts)
   at Microsoft.AppInspector.Program.<>c.<Main>b__6_0(AnalyzeCommandOptions opts)
   at CommandLine.ParserResultExtensions.MapResult[T1,T2,T3,T4,T5,TResult](ParserResult`1 result, Func`2 parsedFunc1, Func`2 parsedFunc2, Func`2 parsedFunc3, Func`2 parsedFunc4, Func`2 parsedFunc5, Func`2 notParsedFunc)
   at Microsoft.AppInspector.Program.Main(String[] args)

Expected behavior
A clear and concise description of what you expected to happen.

Desktop (please complete the following information):

• OS: Macos Mojave

Typo in README.md

Rudy -> Ruby (credit to HN user mitchty)

Ensure all output strings are localizable

Look for any output strings written to the console or output files that are not content specific or data structures i.e. json structure and ensure they can be easily localized by using the built-in resource object and reading from there.

Fix scan metadata values not appearing

labels show no values on report summary for metadata subsection

Output encoding for html report

Describe the bug
The html report format is missing html encoding which then allows malicious source with XSS to execute in a browser. The bug was submitted to Microsoft using requested policy and corrected in commit #106 and Release v.1.0.24 by Ahmad Khan (@ahmsec). Our thanks!

Problem:
The tool reflects strings from 3rd party source files into its HTML output, and didn't escape the strings for XSS.

Impact:
A malicious source file may contain JavaScript that reads the tool's output file and send it to an external server. Since the output file contains source code, this can result in theft of sensitive data.

Attack flow:

• A software engineer at a company develops an application, and inadvertently includes a malicious 3rd party dependency.
• A security engineer at the company is tasked with reviewing the source code, and runs Application Inspector on the application.
• When the security engineer views the output HTML file, malicious JavaScript is executed that reads internal code snippets and ex-filtrates them out to an external server.

tagdiff command Runtime error: Unexpected character encountered

Describe the bug
dotnet AppInspector.dll tagdiff --src1 /media/emailba1 --src2 /media/emailba2

2020-01-23T14:45:21 1 ERROR - A runtime error has occured. Please see log file for more information.
2020-01-23T14:45:21 1 ERROR - Runtime error: Unexpected character encountered while parsing value: M. Path '', line 0, position 0. at Newtonsoft.Json.JsonTextReader.ParseValue()
at Newtonsoft.Json.JsonReader.ReadAndMoveToContent()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value)
at Microsoft.AppInspector.TagDiffCommand.Run()
at Microsoft.AppInspector.Program.RunTagDiffCommand(TagDiffCommandOptions opts)
at Microsoft.AppInspector.Program.<>c.

b__6_1(TagDiffCommandOptions opts)
at CommandLine.ParserResultExtensions.MapResult[T1,T2,T3,T4,T5,TResult](ParserResult1 result, Func2 parsedFunc1, Func2 parsedFunc2, Func2 parsedFunc3, Func2 parsedFunc4, Func2 parsedFunc5, Func`2 notParsedFunc)
at Microsoft.AppInspector.Program.Main(String[] args)

To Reproduce
Execute
dotnet AppInspector.dll tagdiff --src1 /media/emailba1 --src2 /media/emailba2

Expected behavior
End The command Succesfully and generates output.html

Desktop (please complete the following information):

• OS: Debian Linux
• Version 5.2

Add more auth detection support for LDAP and improved Oauth and more

Several key rules/patterns needed to address missing auth patterns for SAML, Basic Auth, a missing Microsoft login URL, active directory, Oauth, NTML, and more.

AppInspector errors when trying to set output filetype as HTML

Describe the bug
When trying to output the file as html, I will see that AppInspector tries looking for a non existent file "html/index.html" inside the current folder where the command is run.

Command line

dotnet /path/to/AppInspector.dll analyze -s file.c -f html

Error message in log.txt

Runtime error: Could not find a part of the path "/current/path/to/source/code/files/html/index.html"

If I tried the following, I see another error message

dotnet /path/to/AppInspector.dll analyze -s file.c -f html -o ./output.html

Error message in log.txt

Runtime error: Object reference not set to an instance of an object

To Reproduce
Try to analyze a file and select the filetype as html and use either of the command line arguments listed above. There are no issues setting the output filetype as JSON or Text.

Expected behavior
AppInspector should be able to output a proper html file

Screenshots

Desktop (please complete the following information):

• OS: Mac OS 10.14.6 Mojave
• Browser Google Chrome
• Version 79.0.3945.117

Smartphone (please complete the following information):

Additional context

Confidence level image quality needs improvement

Quality of confidence images is low and especially noticeable on some resolutions more than others. Improve the images quality e.g. don't shrink but create at the right size etc.

Report Show Skipped Code as Analyzed

Describe the bug
The final report output.html show analyzed 0.99% but detects several kinds of code, I think the result is wrong and this could be the skipped %.

Desktop (please complete the following information):

• OS: Windows 10 - Linux
• Browser FireFox

Failed to analyze with exception. Possible zip file related issue for some cases.

Describe the bug
Failed to write the result to a specified floder using "-o" arguements.
Please check out the pics

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
write the result to anywhere I specified.

Screenshots

Desktop (please complete the following information):

• OS: [Windows 10]
• Browser [chrome]
• Version [win 10 18363.592, ]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Expand HTML report to show confidence values

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
The tool has a confidence value associated with each pattern match. The HTML report needs to be updated to illustrate that both at the subgroup level or icon level as well as the individual match lists somehow similar to the JSON report.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Repair disabled default rules

A small set or rules are disabled at present including media file parsing, .NET crypto detection and crypto infra. Review, correct any duplication with current ruleset and re-add valid delta.

Match details exclusion from json

Describe the bug
With a recent change over in the html report we can no longer have an indicator for a given feature tag in the html report without all instances appearing as match records also I the json output. Fix the issue so that the two possible so we don't have multiple records or noise for things we don't want to see e.g. ability to list dependencies I the html and aggregated sections of each report without a large list of matches in the json output.

Add unit tests for known cases and languages

Add test cases for supported languages in rules
Add test cases for verifying subset of patterns

Clean up any extraneous files before GA release

Examine and finalize set of files needed and those not needed before general audience final 1.0 release.

[Request] Azure DevOps Extension

Would be nice to have an Azure DevOps extension doing all the boilerplate instead of
writing a few commands to invoke with all variables

Processing more then 100%

Describe the bug
The Processing status shows more then 100%

To Reproduce
Steps to reproduce the behavior:
Run the command seen in the screenshot.

Expected behavior
Some work status between 0% and 100%

Or change the Message.

Screenshots

Desktop (please complete the following information):

• OS: Windows 10
• Version 1.0.20

Improve method ProcessInMemmory in AnalyzeCommand.

Hello!
Thank's for this project. I found it useful! You guys did a great job!

Is your feature request related to a problem? Please describe.

Some people faced with an OutOfMemmoryException during the execution of AnalyzeCommand
#103, #91 .

I dived into this class(AnalyzeCommand.cs) and find a code, that I think could be improved.

I'm talking about method ProcessAsFile(string filename) in AnalyzeCommand.cs.

void ProcesAsFile(string filename)
        {
            if (File.Exists(filename))
            {
                _appProfile.MetaData.FileNames.Add(filename);
                _appProfile.MetaData.PackageTypes.Add(ErrMsg.GetString(ErrMsg.ID.ANALYZE_UNCOMPRESSED_FILETYPE));

                string fileText = File.ReadAllText(filename);
                ProcessInMemory(filename, fileText);
            }
            else
            {
                throw new OpException(ErrMsg.FormatString(ErrMsg.ID.CMD_INVALID_FILE_OR_DIR, filename));
            }
        }

I'm confused with these lines:

string fileText = File.ReadAllText(filename);
ProcessInMemory(filename, fileText);

Here we read a whole file content into memory using ReadAllText method and then call method ProcessInMemmory which verify if a length of the string exceded specified threshold:

if (fileText.Length > MAX_FILESIZE)
{
   //Some stuff goes here.
   return;
}

Fact, that we perform this operation in a loop, for each file trouble me a little bit.
Potentially this could create memory pressure.

Describe the solution you'd like

What if before reading the whole file into memory we will check the actual file size and only after that decide proceed file or not.
I think this could be achieved using FileInfo.Length property.

What do you think of this?
Feel free to correct me if I'm mistaken or didn't notice something.

Have a nice day!

Ref: #75 ApplicationInspector errors trying to produce html output when launched from outside app directory

Describe the bug
A reference to Issue #75 where ApplicationInspector could not produce output file in HTML format

To Reproduce
Using the following command line arguments without any output arguments and it will show the error.

dotnet /path/to/AppInspector.dll analyze -s file.c

Runtime error: Could not find a part of the path '/path/to/source_code/files/html/index.html'

The tool keeps looking for a non existent index.html file

Expected behavior
ApplicationInspector should produce a HTML file.

Screenshots

Desktop (please complete the following information):

• OS: Mac OS X 10.14.6 Mojave
• Browser Google Chrome
• Version 79.0.3945.130

Smartphone (please complete the following information):

Additional context

Fails to write output

Describe the bug

2020-01-17T15:50:19 1 ERROR - A runtime error has occured.  Please see log file for more information.
2020-01-17T15:50:19 1 ERROR - Runtime error:    at System.IO.FileStream.ValidateFileHandle(SafeFileHandle fileHandle)
   at System.IO.FileStream.CreateFileOpenHandle(FileMode mode, FileShare share, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.StreamReader.ValidateArgsAndOpenPath(String path, Encoding encoding, Int32 bufferSize)
   at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks)
   at System.IO.File.InternalReadAllText(String path, Encoding encoding)
   at System.IO.File.ReadAllText(String path)
   at Microsoft.AppInspector.LiquidWriter.WriteApp(AppProfile app)
   at Microsoft.AppInspector.AnalyzeCommand.FlushAll()
   at Microsoft.AppInspector.AnalyzeCommand.Run()
   at Microsoft.AppInspector.Program.RunAnalyzeCommand(AnalyzeCommandOptions opts)
   at Microsoft.AppInspector.Program.<>c.<Main>b__6_0(AnalyzeCommandOptions opts)
   at CommandLine.ParserResultExtensions.MapResult[T1,T2,T3,T4,T5,TResult](ParserResult`1 result, Func`2 parsedFunc1, Func`2 parsedFunc2, Func`2 parsedFunc3, Func`2 parsedFunc4, Func`2 parsedFunc5, Func`2 notParsedFunc)
   at Microsoft.AppInspector.Program.Main(String[] args)

To Reproduce
Steps to reproduce the behavior:

1. Run the tool on any directory containing C# source code (analyze -s .)
2. Receive A runtime error has occured. Please see log file for more information.
3. Check log file - always same stack trace

Expected behavior
Output file generated successfully.

Desktop (please complete the following information):

• OS: Windows 10 (1809)

Also tried running with elevated privilege. Same result.

Add Support for Wildcards in Path or Add option to ignore certain filetypes

Is your feature request related to a problem? Please describe.
In a big repo with heavy zip files i want to ignore them in the scan.

Describe the solution you'd like
-o path\to\source*.java

or
--ignore-filetypes .zip .gz

Describe alternatives you've considered
deleting the zips before scan --> not practical

Additional context
Mainly because the scanner crashes with the Zip too large Error

Add highlighting of source code sample section for improved visibility

Describe the bug
When you navigate a report and click on one of the featured discovered, you get a snapshot of the source code. There is also an hyperlink above the snapshot to the source file.
However, when you click on that link, nothing happens.
It woudl be great if you could click on that link and see the full source code so we get more context.
Also, if the snapshot could be highlgihted in the full source code that would be a bonus.

Regards,
S.

Exception: Cannot access a disposed object.\r\nObject name: 'ZipFile'.

Describe the bug
Running the tool using parameters "analyze -s ", I get this error when the tool tries to read a .zip file.

To Reproduce
Steps to reproduce the behavior:

1. Run tool using parameters "analyze -s "
2. Exception will occur on a certain type of zip file.
3. Exception occurs.

Expected behavior
Should skip the file, and log the problem.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Windows 10 (10.0.17763 N/A Build 17763)

Add composition rule matching support

Add ability to test for the presence of multiple rules as a set for an evaluation to be considered a match e.g.
1: Rule 1 + Rule 2 + Rule 3
2: Rule 1 + Rule 2 + !Rule3

List of all unique tags found in HTML report would be nice

Is your feature request related to a problem? Please describe.
The icons are great but would be nice to see a list of all unique tags found in one glance. Related is that you don't know what the icons represent until or unless you mouse over them. Should be able to tell what they represent easier on the Profile page. Our first effort on the HTML report UI didn't cover everything and this serves as a work item for it.

Describe the solution you'd like

1. a place that just lists all unique features
2. ability to tell what icons are like a label that is shown for each on the profile page

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Reduce structure duplication for matchlist

The analyze command uses a class MatchIssue to wrap the Issue objects returned from the rules engine. Most of the added properties in the MatchIssue are valid for the Issue object and should be moved to avoid creating a new class and for improving performance.

Source File Listing Width

Source listing width is really wide and doesn't look fantastic. Reduce and enable overflow-x:auto or something.

Add DynamicEval Detection for Python

Missing a detection rule for Python. Add a rule to detect the following:

python pattern to add for external proc

Run an external command, block until it completes

def run_command(cmd):
print '***** Running command: %s' % ' '.join(map(str,cmd))
p = subprocess.Popen(cmd)
p.wait()

Rules Pattern Prioritization Replacement

Currently, the rules engine processes rules in order they are deserialized originally and if a pattern match that is a lower confidence level has already been added for the same code block, a higher confidence pattern match will be discarded. Note this issue is not present if the engine is filtered for high confidence only.

Improve the solution so that it checks to see if a new match has a higher confidence level than a previous match and replace the old one with the new.

Python: File write uses 'read' tag, missing system() call

Describe the bug
Code Sample:

import requests
import subprocess

response = requests.get('https://example.com/file.zip')
with open('downloaded_file.zip') as f:
  f.write(response.read())

system('ls -l downloaded_file.zip')

First, Unique Tag:

"uniqueTags": [
  "OS.FileOperation.Read",   <-- Should be .Write or just OS.FileOperation
  "OS.Network.Connection.Http",
  "Data.Zipfile"
],

Second, the system() call should be flagged as execution. Maybe add Python to AI033700.

Found in v1.0.11.

publish official docker image

Document the supported programming languages explicitly

Installation instructions

Is your feature request related to a problem? Please describe.
Provide step by step instruction for installing on Windows 10 for use with Visual Studio Community.

Describe the solution you'd like
A clear and concise set of steps on how to install

ApplicationInspector does not render HTML report page properly if run outside its folder

Describe the bug
When running AppInspector.dll from outside its folder to generate HTML report, the page does not get rendered properly due to relative paths that reference where AppInspector.dll is located.

<link rel="stylesheet" href="html/resources/css/bootstrap.min.css"

The page will try to look for these files relative to where the command line was run (example the folder where I store the source code files, which is /path/to/source_code/files).

dotnet /path/to/AppInspector.dll analyze -s /path/to/source_code/files/file.c

To Reproduce
Run the command outside of AppInspector.dll's folder.

dotnet /path/to/AppInspector.dll analyze -s /path/to/source_code/files/file.c

Expected behavior
The HTML report page should be rendered properly

Screenshots

Desktop (please complete the following information):

• OS: Mac OS 10.14.6 Mojave
• Browser Google Chrome
• Version 79.0.3945.130

Smartphone (please complete the following information):

Additional context

Blazor GUI Interface

Is your feature request related to a problem? Please describe.
The local HTML file can get very large; architecturally, embedding everything in it may not be the best solution.

Describe the solution you'd like
Instead of generating a static HTML file, use a local web-server that serves the content. The advantage of this is that for large reports, we don't need to create a large HTML file, and can instead stream the results as needed, and potentially perform operations on the "server" instead of within the browser.

Describe alternatives you've considered
n/a

Additional context
n/a

Provide option to disable automatic archive decompression

Is your feature request related to a problem? Please describe.
When attempting to analyze the Chromium source code, I run into an error where the application attempts to inflate an intentionally-corrupt zip file.

Describe the solution you'd like
I'd like the option to pass in a flag to disable the built-in archive decompression. Alternatively, a flag to ignore files matching certain patterns.