checkmarx-ts / cxanalytix Goto Github PK

Describe the problem

Currently all projects and their scans are extracted during a sweep. In some cases, it may be desirable to only include scans for projects that are considered production and avoid scans for projects no meeting this criteria.

Proposed solution

Apply pass criteria via regex match using the following order:

1. Regex match on the team to which the project is assigned
2. Regex match to the project name.

Additional details

This filtering should be disabled by default. No filtering steps should be taken unless the filtering is explicitly included in the configuration.

Description

When an XML report is generated, the generation code is supposed to check for the status of the report periodically.

Expected Behavior

There should be some sort of delay between checks.

Actual Behavior

There is no delay, the code just hammers the web service looking for status updates.

Reproduction

• Execute v1.1.6 with network trace enabled
• Observe there is no time delay between the calls to the status check

The custom fields array contains an "id" property and it confuses the project parsing which is using "id" as the property for the project id.

Description

The initial run is concatenating paths, and the initial path is .\, which is causing an issue during writing of the State file.

Expected Behavior

File path should be proper format

Actual Behavior

.\ is concatenated with issues, causing exception (see below).

Reproduction

Execution of the Linux Daemon on Mac, Linux (WSL 2), the following exception occurs:

[2020-07-10 12:23:15,197] ERROR [4] [CxAnalytix.TransformLogic.Transformer] (?:?) - Error resolving scans, server may be unavailable.
System.IO.DirectoryNotFoundException: Could not find a part of the path '/App/.\/CxAnalytixExportState.json'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.StreamWriter.ValidateArgsAndOpenPath(String path, Boolean append, Encoding encoding, Int32 bufferSize)
   at System.IO.StreamWriter..ctor(String path)
   at CxAnalytix.TransformLogic.ProjectResolver.saveProjectCheckState() in /Users/kenmcdonald/Documents/checkmarx-ts/cxanalytix/TransformLogic/ProjectResolver.cs:line 85
   at CxAnalytix.TransformLogic.ScanResolver.Resolve(DateTime lastCheckDate) in /Users/kenmcdonald/Documents/checkmarx-ts/cxanalytix/TransformLogic/ScanResolver.cs:line 145
   at CxAnalytix.TransformLogic.Transformer..ctor(CxRestContext ctx, CancellationToken token, String previousStatePath) in /Users/kenmcdonald/Documents/checkmarx-ts/cxanalytix/TransformLogic/Transformer.cs:line 328

Environment Details

Confirmed on Mac, Docker on mac, WSL 2

Description

The SinkLine and SinkColumn value in vulnerability details is actually the line of the vulnerability source. It is unknown if this was the case at the time CxAnalytix was implemented. It is observed that XML reports produce "Result" nodes with "Line" and "Column" attributes that are the same as the first node in the data flow (i.e. the source node).

Expected Behavior

SinkFileName, SinkLine, and SinkColumn should be the Line/Column of the last node in the result.

Actual Behavior

SinkFileName/SinkLine/SinkColumn are values coming from attributes in the Result node of the XML report. They should be values from the last node in the data flow.

Reproduction

Run against SAST 9.4 or beyond to observe SinkFileName/SinkLine/SinkColumn values reporting the source line/column.

Environment Details

SAST 9.4 HF6

Describe the problem

Currently the StateDataStoragePath is set to empty, causing the state data file to be written to the working directory. The state data file will usually be mapped to some persistent storage so that state is maintained across runs

Proposed solution

A clear description of what you want to happen.

Additional details

Add any other details / contexts / screenshots about the feature request.

The "partialScanReasons" element in the json causes the parsing to have issues.

Description

When crawling, an exception is thrown when the audit trail database connection string is not defined.

Expected Behavior

If the audit trail connection is not defined, the audit trail crawl should be silently bypassed.

Actual Behavior

An exception is thrown indicating the configuration can't be found.

Reproduction

Run CxAnalytix 2.0.0 without the CxDB connection string defined.

Scenario: Your first run works from command line (user key store), but fails when you later run as a service (system key store)

System.PlatformNotSupportedException: Windows Data Protection API (DPAPI) is not supported on this platform.

Reason: When running as a service, the system account cannot retrieve the key from the user key store.

Solution: Restore you CxCredentials node to original. Set Username and Password. Now, only run as service.

Describe the problem

If using Splunk, the data is logged with a timestamp assumed to be the time the data is indexed by Splunk.

Proposed solution

Document how to modify the timestamp extraction logic during indexing so that Splunk uses the scan finish time during indexing.

Describe the problem

Some record types are not needed in some cases, so an ability to filter those record types should be provided in the configuration.

Proposed solution

Add an ability to suppress output for non-audit records the same way audit records can be suppressed.

Describe the problem

When CxAnalytix performs a sweep, it will calculate all scans that need to be extracted based on a recorded timestamp. The timestamp of the current sweep is then recorded for use as the starting time for the next sweep.

If the current sweep is interrupted or a scan can't be extracted, scans that were resolved in the current sweep may never be extracted.

Proposed solution

Update the last sweep timestamp per project using the timestamp of the last scan extracted for that project. This should mean that scans for each project are extracted in chronological order.

Describe the problem

The data crawl is currently limited to SAST/OSA. CxASP should also be supported.

Proposed solution

Perform the same crawl and extraction logic using the CxASP APIs. Data fields should be mapped to the existing field specification as much as possible.

The field specification should change to a matrix to indicate support for field names based on product being crawled.

Describe the problem

Newer versions of SAST have a scan statistics API (/cxrestapi/sast/scans/{id}/statistics) that should be a new record type.

Proposed solution

Each scan has a statistics result, and each result has aggregate contents (e.g. product version, engine version, memory peak, etc) but also has breakdowns by language. Flatten by language type so that there is one record entry per language.

Describe the problem

Scan custom fields have been available since ~v9.2. These should be included in the data crawl.

Proposed solution

Include the scan custom fields in the scan detail and summary data records.

Description

When OSA is not licensed, the reporting API returns a status code to indicate it is not there. CxAnalytix handles this, but will repeatedly ask for more OSA reports since it treats the response as an error. The error should indicate that OSA requests should stop completely.

Expected Behavior

When a request for an OSA report returns a status code indicating OSA is not licensed, no further OSA requests should be made.

Actual Behavior

Requests for OSA reports are repeated for every scan.

Reproduction

Run CxAnalytix with debug logging on, it will show the repeated requests in the log output.

Each record type should correlate to a document collection. Single-field indexes on time fields and any single-field unique identifiers. Compound indexes where appropriate to identify documents uniquely.

Partitioning/shard keys should be hashes calculated on fields and added to the document for sharding purposes. Shard key should be in the root of the document to potentially support Mongo-API compatible DBs like CosmoDB PaaS in Azure.

TBD: the fields used for sharding may need to be configurable so the shard can be tuned based on the expected volume of results.

Describe the problem

CxAnalytix does a periodic crawl of the vulnerability platform. When it executes and the time it takes to execute are variable and unpredictable.

Transactional commits to data storage is not always possible since not all data output mechanisms support transactions. Even the support of transactions is not consistent since some transaction-supporting output methods (like Mongo) have limits on the transaction size that are not compatible with the volume of vulnerability data.

Proposed solution

CxAnalytix can be configured to execute a shell process post-crawl and wait for it to finish before considering the crawl complete. This will allow external analysis processes to be performed while the CxAnalytix data is in a steady state.

Additional details

This can currently be accomplished by executing a script that first executes CxAnalytixCLI using a scheduled task/cron job. In the script, subsequent steps can be executed after CxAnalytixCLI finishes to achieve the ability to work in a steady state.

Describe the problem

Currently CxAnalytix is not serverless and needs to be run on an instance.

Proposed solution

Provide a CxAnalytix public lambda layer that would grant an AWS Lambda access to run CxAnalytix. See Klayers as an example of public layers that can be used by Lambda.

Additional details

This would allow extract transform and load jobs to run serverless and would remove the overhead of maintaining an instance.

Ideally the reporting generated by CxAnalytix should be available as endpoints of the Checkmarx API

Describe the problem

If using MongoDB, the configuration section in dotnet.exe.config leaves plaintext credentials in the connection string when running under Windows. It should be encrypted in-place as it is done with CxCredentials

Proposed solution

Update the MongoDB configuration to encrypt the connection string in-place.

Custom fields can be optionally defined on a per-project basis. If they exist, they should be added to the project output. To avoid clashes with existing field names, they should be prefixed with "Custom_".

Describe the problem

CxAnalytix traffic to a SAST server should be observable for troubleshooting purposes. API operations currently don't provide a meaningful user agent string that would allow CxAnalytix traffic to be explicitly identified.

Proposed solution

The user agent for CxAnalytix should be set to "CxAnalytix/"/

In \CxRestClient\CxScaScans.cs:
if (!scans.IsSuccessStatusCode)
throw new InvalidOperationException(scans.ReasonPhrase);

When OSA is not licensed, 403 error code is thrown causing processing to stop. Processing should continue without the need for OSA scan information.

Describe the problem

Data is currently output as soon as elements are transformed. If the transformation of a scan is interrupted this leaves the scan with only partial data having been output.

Proposed solution

Use transactional capabilities for output methods that support them, implement internal buffering of data and write-on-commit for output methods that lack transactional capabilities (log4net, AMQP, etc).

Describe the problem

Best Fix Location is not currently being return in the results.

Proposed solution

Please include BFL node in scan detail results.

Additional details

NA

Description

The unit tests are currently not part of the CI due to some incompatibility in the build docker image. It looks like the unit tests need to run under .Net core 3.1.

Expected Behavior

Unit tests should execute successfully in the build.

Actual Behavior

The build times out because the unit test execution appears to try to download many unnecessary dependencies.

Reproduction

Enable unit tests in the CI pipeline, execute a build.

Environment Details

N/A

Description

Authentication fails in some cases with an error message "can't use a disposed object".

Expected Behavior

Authentication should work.

Actual Behavior

Initial connection and resume after error fail to re-authenticate with this message.

Description

When not using AMQP as an output, the first run injects a partially configured AMQP configuration into the config file. This appears to be an encrypted connection configuration that may result from default values normally set by the AMQP client library. Subsequent runs of the program fail because the configuration is now malformed.

A workaround is to comment the following lines in the configSections element:

<!--
<section name="AMQPConnection" type="CxAnalytix.Out.AMQPOutput.Config.AmqpConnectionConfig, AMQPOutput"/>
<section name="AMQPConfig" type="CxAnalytix.Out.AMQPOutput.Config.AmqpConfig, AMQPOutput"/>
-->

Expected Behavior

No AMQP connection element should appear spontaneously in the config file; only when the config exists in the file should it write the encrypted version of the connection element.

Actual Behavior

The AMQP connection config appears after the first run.

Reproduction

Configure v1.3.0 to use log output using the default configuration files, execute one crawl. Attempt to crawl again and the program will stop with an error indicating the AMQP connection config is malformed.

Environment Details

Observed on Windows

Describe the problem

Some scans may initially show a large number of results until the project has been properly onboarded. This can occasionally cause some very large data outputs with data that has a potential to incorrectly introduce statistical skews during analysis.

Proposed solution

Enable the following filtering criteria with the numbered precedence for checking:

1. Filter all detail output for scans with vulnerability counts exceeding a maximum threshold
2. Filter detail output for vulnerabilities of a given result severity if the result severity exceeds a maximum threshold
3. Filter detail output for vulnerabilities of a given query severity if the query severity exceeds a maximum threshold

Additional details

By default, no filtering should be performed unless explicitly configured.

Describe the problem

When CxAnalytix performs a crawl, it currently only looks at new scans in each project since the last crawl. This means that updated triage states for each project's latest may be out of date if the states were changed after the CxAnalytix crawl.

Some output methods (such as MongoDB) can perform updates to details if detail outputs are differentiated between Add (currently the only method) and Update. Even the streaming outputs (like AMQP) could potentially handle update vs add a bit differently.

For file output, this obviously would not be option.

Proposed solution

There should be a configuration option to enable this feature, and it should be "off" by default.

1. When a project has no new scans, find the last scan and include it as an "Update" crawl.
2. Output the updated data as an "Update", the output method should understand how to locate and perform the update.
3. For output methods that don't support update, do nothing.

Environment :
Windows Server 2012R2
Dotnet 3.1 runtime
Checkmarx 890 HF30 / 920 HF17

Description：
I try many times to set the config in Environmental variables or inline but it always fails in login token issue, the log message is below...

[2021-08-12T11:19:28+08:00][ INFO][1][CxAnalytixCLI.Program] Start
[2021-08-12T11:19:28+08:00][ INFO][1][CxAnalytixCLI.Program] CWD: C:\Users\Administrator\Desktop\CxAnalytix1.3.1-560.win-x64
[2021-08-12T11:19:28+08:00][ WARN][1][CxRestClient.CxRestContext] Unable to obtain login token due to an unexpected exception. Login will retry on the next operation.
System.InvalidOperationException: Not Found
at CxRestClient.CxRestContext.GetLoginToken(String url, HttpContent authContent) in /root/project/CxRestClient/CxRestContext.cs:line 146
[2021-08-12T11:19:28+08:00][ WARN][1][CxRestClient.CxRestContext] Unable to obtain login token due to an unexpected exception. Login will retry on the next operation.
System.InvalidOperationException: Not Found
at CxRestClient.CxRestContext.GetLoginToken(String url, HttpContent authContent) in /root/project/CxRestClient/CxRestContext.cs:line 146

And the same at the Checkmarx 920. Is it support the 9.2 version?

Thanks.

Add the ability to configure a string value as an "instance identifier" to be added to each record. This allows aggregation of data from multiple SAST instances into a single data store.

Currently, CxAnalytix works great with 8.x. However, the Logintoken behaviour with 9.x ends up with error. This is an expected behaviour with the current code-base with the introduction of new Access Control Module.

It would good to see CxAnalytix be extended for 9.x in the near-future.

Error details: [CxRestClient.CxRestContext]

Unable to obtain login token due to an unexpected exception

Description

In reports from an 8.9 server instance, the transformed JSON contained the full path (e.g. "\CxServer\foo\bar"). When the server instance is 9.x, it truncates the team to the end of the path (e.g. "bar").

Expected Behavior

Normalized data should have the full path rather than just the specific name of the team.

Actual Behavior

The name of the team without the full path is output in the data. This means data for teams with identical names but under a different team structure can't be analyzed properly.

Reproduction

Crawl a 9.x instance, compare normalized data to the data created when crawling an 8.9 instance.

Environment Details

SAST 9.2 HF3 vs SAST 8.9 HF 27

Describe the problem

The current data crawl uses the SAST REST API to retrieve OSA vulnerability data. SCA is a different service and should be incorporated into the data output if it is being used.

Proposed solution

Configuration and logic needs to be added to pull data from the SCA API.

Additional details

Data field naming compatibility should be maintained if possible in the current SCA summary and detail records.

Description

An exception gets thrown when the details column has a null value.

[2021-04-04T16:28:21+10:00][ERROR][7][CxAnalytixService.ServiceLifecycleControl] Audit data transformation aborted due to unhandled exception.
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.ArgumentNullException: Value cannot be null. (Parameter 's')
   at System.IO.StringReader..ctor(String s)
   at CxAnalytix.CxAuditTrails.DBCrawler.<>c.<CxDB_accesscontrol_AuditTrail>b__6_0(Object val) in /root/project/CxAuditTrailsDirectDB/DBCrawler.cs:line 69
   at CxAnalytix.CxAuditTrails.DBCrawler.OutputRecords(SqlDataReader reader, IOutput output, Dictionary`2 customColumnConverters) in /root/project/CxAuditTrailsDirectDB/DBCrawler.cs:line 44
   at CxAnalytix.CxAuditTrails.DBCrawler.CxDB_accesscontrol_AuditTrail(DateTime sinceDate, IOutput output) in /root/project/CxAuditTrailsDirectDB/DBCrawler.cs:line 72
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)
   at System.Type.InvokeMember(String name, BindingFlags invokeAttr, Binder binder, Object target, Object[] args)
   at CxAnalytix.AuditTrails.Crawler.AuditTrailCrawler.InvokeCrawlMethod(String methodName, IAuditTrailCrawler crawler, CancellationToken token) in /root/project/CxAuditTrailsCrawler/AuditTrailCrawler.cs:line 77
   at CxAnalytix.AuditTrails.Crawler.AuditTrailCrawler.CrawlAuditTrails(IOutputFactory outFactory, CancellationToken token) in /root/project/CxAuditTrailsCrawler/AuditTrailCrawler.cs:line 101
   at CxAnalytixService.ServiceLifecycleControl.<>c__DisplayClass8_0.<<OnStart>b__0>d.MoveNext() in /root/project/CxAnalytixService/ServiceLifecycleControl.cs:line 149

Expected Behavior

The null value should be handled gracefully and not throw an exception.

Actual Behavior

The exception is thrown, which causes the audit trail crawl to halt prematurely.

Reproduction

Use a database that has a record where the details column is null. It appears that creating an SMTP server records NULL as details.

Describe the problem

Starting v2.2 of CxSAST API, additional information related branching is available as part of the Get / project API
relevant part:
... "isBranched": true, "originalProjectId": "string", "branchedOnScanId": "string", "relatedProjects": [ 0 ]
Would be good to include this data into CxAnalytix too

Proposed solution

As part of any other information which we are already capturing for the projects - add the branching info as well

Additional details

Description

An exception is thrown when the CxAnalytixExportState.json file is empty which is likely to occur when users want to "restart" CxAnalytix from the beginning.

Expected Behavior

An empty state file will be treated the same as if the file does not exist and begin extracting all data available.

Actual Behavior

An exception is thrown and the extraction stops when the state file is empty.

Reproduction

Installed CxAnalytix using the Getting Started w/the CLI steps.

dotnet.exe.config file contents:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="CxCredentials" type="CxAnalytix.Configuration.CxCredentials, Configuration" />
    <section name="CxConnection" type="CxAnalytix.Configuration.CxConnection, Configuration" />
    <section name="CxAnalyticsService" type="CxAnalytix.Configuration.CxAnalyticsService, Configuration" />
    <section name="CxLogOutput" type="CxAnalytix.Out.Log4NetOutput.LogOutputConfig, Log4NetOutput" />
  </configSections>

  <!-- Common config parameters -->
  <CxConnection URL="http://localhost"
                mnoURL=""
                TimeoutSeconds="600" ValidateCertificates="true" />
  <CxCredentials configProtectionProvider="DataProtectionConfigurationProvider">
    <EncryptedData>
      <CipherData>
        <CipherValue>AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAD+Yfz3TTtkuNUwiA2mnaNQQAAAACAAAAAAAQZgAAAAEAACAAAABEFil6yJornAfayWri4jhnYx8ZcVKlCbdK0MKf0OYbPQAAAAAOgAAAAAIAACAAAAAoSneyAzYRay+umoLa1CEvdb/54SM3v6CwWl8QMcgPOZAAAADSspNbRLZr9vwTmqOeZhm05gVNs3yONMWuKvhfwodTOF7jGtg9uHVbc5lH8cpNxU7Qb072JAjiiCYrAjy1aCjMO5NH0ibJViL0n9euH2jJz6mibUo0VNoNfid8KQhRZqogivlzpL/rEOSOdX0qEzu3ABu35g9knhcCb8wL2kwawXAIn3vYYE8vAszERfZ8fbpAAAAACU6pegd1dCoitWECWzFd5oPxW2BLsCRkJqG30yqFsmwD0jKEh8WWyK5QPVxZ9x8TURIliyJdtMhLK/yHGKiq6Q==</CipherValue>
      </CipherData>
    </EncryptedData>
  </CxCredentials>
  <CxAnalyticsService ConcurrentThreads="2" StateDataStoragePath=""
                      ProcessPeriodMinutes="120"
                      OutputFactoryClassPath="CxAnalytix.Out.Log4NetOutput.LoggerOutFactory, Log4NetOutput"
                      SASTScanSummaryRecordName="RECORD_SAST_Scan_Summary"
                      SASTScanDetailRecordName="RECORD_SAST_Scan_Detail"
                      SCAScanSummaryRecordName="RECORD_SCA_Scan_Summary"
                      SCAScanDetailRecordName="RECORD_SCA_Scan_Detail"
                      ProjectInfoRecordName="RECORD_Project_Info"
                      PolicyViolationsRecordName="RECORD_Policy_Violations"
                      />


  <!-- Specific output method configuration parameters -->
  <CxLogOutput DataRetentionDays="3" OutputRoot="logs\">
    <PurgeSpecs>
      <spec MatchSpec="*.log.*" />
    </PurgeSpecs>
  </CxLogOutput>


</configuration>

Run CxAnalytix one time. Your state file will be populated. For example:

{"1":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":1,"ProjectName":"dvna_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"2":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":2,"ProjectName":"WebGoat_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"3":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":3,"ProjectName":"NodeGoat_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"4":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":4,"ProjectName":"NAudio_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"5":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":5,"ProjectName":"dvna_sandbox","TeamId":"11c02153-343a-4629-bd01-6aa51c28fd33","TeamName":"\\CxServer\\serviceprovider\\company\\dvna","PresetId":36},"6":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":6,"ProjectName":"WebGoat_sandbox","TeamId":"d84ae381-3bdf-4e23-b57b-a754662957cd","TeamName":"\\CxServer\\serviceprovider\\company\\WebGoat","PresetId":36},"7":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":7,"ProjectName":"NodeGoat_sandbox","TeamId":"6d6b6dd4-0205-415d-8fd1-102e02490c90","TeamName":"\\CxServer\\serviceprovider\\company\\NodeGoat","PresetId":36},"8":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":8,"ProjectName":"NAudio_sandbox","TeamId":"af6336f4-7913-44a3-aa3a-2dc3e3dd2eb1","TeamName":"\\CxServer\\serviceprovider\\company\\NAudio","PresetId":36},"9":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":9,"ProjectName":"nopCommerce_sandbox","TeamId":"74935482-3e41-4d61-8eb8-8f7db42d6245","TeamName":"\\CxServer\\serviceprovider\\company\\nopCommerce","PresetId":36},"10":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":10,"ProjectName":"FluentEmail_sandbox","TeamId":"225fb029-0589-4a22-83ac-8d5b8a5222ab","TeamName":"\\CxServer\\serviceprovider\\company\\FluentEmail","PresetId":36},"11":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":11,"ProjectName":"DVWA_sandbox","TeamId":"13bc700b-e3a2-4d9a-95a1-131919ccf0b6","TeamName":"\\CxServer\\serviceprovider\\company\\DVWA","PresetId":36},"12":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":12,"ProjectName":"notepad-plus-plus_sandbox","TeamId":"af7e5264-6963-490b-9b63-7fbe2d8f94af","TeamName":"\\CxServer\\serviceprovider\\company\\notepad-plus-plus","PresetId":36},"13":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":13,"ProjectName":"dvja_sandbox","TeamId":"50fa5d73-0311-47b7-85fc-9599a765f830","TeamName":"\\CxServer\\serviceprovider\\company\\dvja","PresetId":36},"14":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":14,"ProjectName":"JavaVulnerableLab_sandbox","TeamId":"4b0d6b0d-f552-49d7-ac27-ad468dbd6b4c","TeamName":"\\CxServer\\serviceprovider\\company\\JavaVulnerableLab","PresetId":36},"15":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":15,"ProjectName":"OWASP-GoatDroid-Project_sandbox","TeamId":"9ed648cf-0bfd-4abb-bc7e-6b37e8cbfc57","TeamName":"\\CxServer\\serviceprovider\\company\\OWASP-GoatDroid-Project","PresetId":36},"16":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":16,"ProjectName":"DVWS_sandbox","TeamId":"011aa991-9203-454b-b0b9-349e0d652ecf","TeamName":"\\CxServer\\serviceprovider\\company\\DVWS","PresetId":36},"17":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":17,"ProjectName":"dvws-node_sandbox","TeamId":"eebd2236-f487-4c8e-9008-395d2016661f","TeamName":"\\CxServer\\serviceprovider\\company\\dvws-node","PresetId":36},"18":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":18,"ProjectName":"pivaa_sandbox","TeamId":"8d2048b1-f189-4a16-85f0-020bf447f9b0","TeamName":"\\CxServer\\serviceprovider\\company\\pivaa","PresetId":36},"19":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":19,"ProjectName":"diva-android_sandbox","TeamId":"af0dfef0-165b-4b8a-ac2f-d78466111ebc","TeamName":"\\CxServer\\serviceprovider\\company\\diva-android","PresetId":36},"20":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":20,"ProjectName":"DodoVulnerableBank_sandbox","TeamId":"5a52624e-317a-458e-b074-674b0e12687b","TeamName":"\\CxServer\\serviceprovider\\company\\DodoVulnerableBank","PresetId":36},"21":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":21,"ProjectName":"Android-InsecureBankv2_sandbox","TeamId":"be5a8d1b-da8e-44e2-ba62-9f85adb5bffe","TeamName":"\\CxServer\\serviceprovider\\company\\Android-InsecureBankv2","PresetId":36},"22":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":22,"ProjectName":"VulnerableAndroidAppOracle_sandbox","TeamId":"aa1c180c-aacf-4f36-9041-ed98b6f16b6d","TeamName":"\\CxServer\\serviceprovider\\company\\VulnerableAndroidAppOracle","PresetId":36},"23":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":23,"ProjectName":"Digitalbank_sandbox","TeamId":"99595f9d-822d-4910-8ea6-8d2866929389","TeamName":"\\CxServer\\serviceprovider\\company\\Digitalbank","PresetId":36},"24":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":24,"ProjectName":"DVHMA_sandbox","TeamId":"f5e04c11-573f-49fd-ab2d-74553b1b37c6","TeamName":"\\CxServer\\serviceprovider\\company\\DVHMA","PresetId":36},"25":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":25,"ProjectName":"juice-shop_sandbox","TeamId":"95f00e80-737a-4f16-b4d5-8758c056daae","TeamName":"\\CxServer\\serviceprovider\\company\\juice-shop","PresetId":36},"26":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":26,"ProjectName":"DVIA_sandbox","TeamId":"48745f59-80cc-48d2-9126-e23ea90e614a","TeamName":"\\CxServer\\serviceprovider\\company\\DVIA","PresetId":36},"27":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":27,"ProjectName":"DVIA-v2_sandbox","TeamId":"4f0c4656-0061-449c-9fc8-657d8df49aab","TeamName":"\\CxServer\\serviceprovider\\company\\DVIA-v2","PresetId":36},"28":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":28,"ProjectName":"AltoroJ_sandbox","TeamId":"1c0d6132-d2ac-45ab-835a-66b35212320a","TeamName":"\\CxServer\\serviceprovider\\company\\AltoroJ","PresetId":36},"29":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":29,"ProjectName":"nopCommerce_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"30":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":30,"ProjectName":"FluentEmail_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"31":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":31,"ProjectName":"DVWA_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"32":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":32,"ProjectName":"notepad-plus-plus_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"33":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":33,"ProjectName":"dvja_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"34":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":34,"ProjectName":"JavaVulnerableLab_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"35":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":35,"ProjectName":"OWASP-GoatDroid-Project_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"36":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":36,"ProjectName":"DVWS_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"37":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":37,"ProjectName":"dvws-node_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"38":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":38,"ProjectName":"pivaa_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"39":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":39,"ProjectName":"diva-android_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"40":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":40,"ProjectName":"DodoVulnerableBank_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"41":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":41,"ProjectName":"Android-InsecureBankv2_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"42":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":42,"ProjectName":"VulnerableAndroidAppOracle_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"43":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":43,"ProjectName":"Digitalbank_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"44":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":44,"ProjectName":"DVHMA_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"45":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":45,"ProjectName":"juice-shop_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"46":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":46,"ProjectName":"DVIA_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"47":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":47,"ProjectName":"DVIA-v2_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36},"48":{"LastScanCheckDate":"2020-08-11T20:26:24.5533576+00:00","ProjectId":48,"ProjectName":"AltoroJ_policy","TeamId":"26af7de0-042e-43b0-ad7c-48fabb038b0d","TeamName":"\\CxServer\\serviceprovider\\company","PresetId":36}}

Run the CLI again to verify that there is no more data to extract.

PS C:\programdata\checkmarx\cxanalytix\artifacts\Release> dotnet CxAnalytixCLI.dll
[2020-08-11 20:29:39,797]  INFO [1] [CxAnalytixCLI.Program] (?:?) - Start
[2020-08-11 20:29:39,809]  INFO [1] [CxAnalytixCLI.Program] (?:?) - CWD: C:\programdata\checkmarx\cxanalytix\artifacts\Release
[2020-08-11 20:29:40,143]  WARN [1] [CxAnalytix.TransformLogic.Transformer] (?:?) - Policy data is not available.
System.InvalidOperationException: Unable to retrieve policies.
   at CxRestClient.CxMnoPolicies.GetAllPolicies(CxRestContext ctx, CancellationToken token) in c:\programdata\checkmarx\CxAnalytix\CxRestClient\CxMnoPolicies.cs:line 128
   at CxAnalytix.TransformLogic.Transformer..ctor(CxRestContext ctx, CancellationToken token, String previousStatePath) in c:\programdata\checkmarx\CxAnalytix\TransformLogic\Transformer.cs:line 260
[2020-08-11 20:29:40,577]  INFO [1] [CxAnalytix.TransformLogic.ProjectResolver] (?:?) - 48 projects are targets for check for new scans. Since last scan: 0 projects removed, 0 new projects.
[2020-08-11 20:29:41,138]  INFO [1] [CxAnalytix.TransformLogic.ScanResolver] (?:?) - Resolved 0 scans to check in 0 projects since 8/11/2020 8:29:40 PM.
[2020-08-11 20:29:41,142]  INFO [1] [CxAnalytixCLI.Program] (?:?) - End

**Delete the state file content using a text editor so that the file still exists but is empty.

Run the CLI again and note an exception occurs.**

PS C:\programdata\checkmarx\cxanalytix\artifacts\Release> dotnet CxAnalytixCLI.dll
[2020-08-11 20:30:45,196]  INFO [1] [CxAnalytixCLI.Program] (?:?) - Start
[2020-08-11 20:30:45,210]  INFO [1] [CxAnalytixCLI.Program] (?:?) - CWD: C:\programdata\checkmarx\cxanalytix\artifacts\Release
[2020-08-11 20:30:45,571]  WARN [1] [CxAnalytix.TransformLogic.Transformer] (?:?) - Policy data is not available.
System.InvalidOperationException: Unable to retrieve policies.
   at CxRestClient.CxMnoPolicies.GetAllPolicies(CxRestContext ctx, CancellationToken token) in c:\programdata\checkmarx\CxAnalytix\CxRestClient\CxMnoPolicies.cs:line 128
   at CxAnalytix.TransformLogic.Transformer..ctor(CxRestContext ctx, CancellationToken token, String previousStatePath) in c:\programdata\checkmarx\CxAnalytix\TransformLogic\Transformer.cs:line 260

Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at CxAnalytix.TransformLogic.ProjectResolver.Resolve(Dictionary`2 productAction) in c:\programdata\checkmarx\CxAnalytix\TransformLogic\ProjectResolver.cs:line 125
   at CxAnalytix.TransformLogic.Transformer..ctor(CxRestContext ctx, CancellationToken token, String previousStatePath) in c:\programdata\checkmarx\CxAnalytix\TransformLogic\Transformer.cs:line 314
   at CxAnalytix.TransformLogic.Transformer.DoTransform(Int32 concurrentThreads, String previousStatePath, String instanceId, CxRestContext ctx, IOutputFactory outFactory, RecordNames records, CancellationToken token) in c:\programdata\checkmarx\CxAnalytix\TransformLogic\Transformer.cs:line 406
   at CxAnalytixCLI.Program.Main(String[] args) in c:\programdata\checkmarx\CxAnalytix\CxAnalytixCLI\Program.cs:line 41

Environment Details

I built from source, but it was building version 1.1.3

PS C:\programdata\checkmarx\cxanalytix> git describe --tags
v1.1.3
PS C:\programdata\checkmarx\cxanalytix> git rev-parse HEAD
9c7725e6075524f0c9b2ba405e163c0faa989456
PS C:\programdata\checkmarx\cxanalytix>

OS Environment is server 2016.

Description

When crawling projects, attempts to pull vulnerability data for OSA scans report the scan data can not be found.

Expected Behavior

Existing OSA scan vulnerabilities should be found and output.

Actual Behavior

The log reports Could not obtain vulnerability data for scan <GUID> in project <ID>: <NAME>. Vulnerability data will not be available. and does not output the OSA vulnerability data.

Reproduction

Reproduction is still under investigation.

Environment Details

N/A

Description

The CxAnalytix crawl fails due to a date parsing error when run on a machine with a non-US locale.

Expected Behavior

Dates from the server should be parsed without failure.

Actual Behavior

Date parsing fails with an error:

System.FormatException: String '08/13/2020 00:02:37' was not recognized as a valid DateTime.

Reproduction

Set the client in a non-US display locale, such as Austrailia, that uses a different date format.

Environment Details

The locale of the server does not appear to format the date according to the display rules. The server appears to send the date in the US locale format regardless of the server-side locale.

Describe the problem

Database, collection, and indexes on collections are created at first start if these elements are not detected in the database. This means that the user account for connecting to MongoDB must have elevated privileges for at least the first execution.

Proposed solution

Create an external script that is used to create the MongoDB schema. On startup, the process should exit if the required schema elements are not detected.

Description

Trying to retrieve policy violations from projects with a 9.0 server results in the following errors and no policy violation data being retrieved.

[2020-07-16T09:24:39-05:00][ WARN][1][CxAnalytix.TransformLogic.Transformer] Unable to correlate policies to project 15: SimplyVulnerable-master. Policy statistics will be unavalable.
System.InvalidOperationException: Unable to retrieve policies for project 15.
at CxRestClient.CxMnoPolicies.GetPolicyIdsForProject(CxRestContext ctx, CancellationToken token, Int32 projectId) in /root/project/CxRestClient/CxMnoPolicies.cs:line 156
at CxAnalytix.TransformLogic.Transformer..ctor(CxRestContext ctx, CancellationToken token, String previousStatePath) in /root/project/TransformLogic/Transformer.cs:line 288

[2020-07-16T09:24:41-05:00][DEBUG][1][CxAnalytix.TransformLogic.Transformer] Policy violations for project 15: SimplyVulnerable-master are unavailable.
System.InvalidOperationException: Unable to retrieve rule violations for project 15.
at CxRestClient.CxMnoRetreivePolicyViolations.GetViolations(CxRestContext ctx, CancellationToken token, Int32 projectId, PolicyCollection policies) in /root/project/CxRestClient/CxMnoRetreivePolicyViolations.cs:line 118
at CxAnalytix.TransformLogic.Transformer.b__74_0(ScanDescriptor scan) in /root/project/TransformLogic/Transformer.cs:line 354

Expected Behavior

If there are policy violations available, they should be retrieved.

Actual Behavior

No policy violations reported.

Reproduction

Execute against a 9.0 server.

Environment Details

Windows client, 9.0 server,

Description

The scan summary record produces severity totals that don't match the project state totals at the time the scan report is consumed.

Expected Behavior

When the result report is produced, the totals should be calculated by filtering out Not Exploitable (False Positive) results and using the ResultSeverity field for the totals.

The totals can diverge from project state totals if triage is performed after the CxAnalytix crawl. It is not the expectation that triage changes will update previously written totals.

Actual Behavior

The counts are produced using QuerySeverity and no filtering of NE results.

Describe the problem

GCP supports gRPC endpoints for Cloud PubSub. Messages sent using this protocol may be used the same way as messages sent via the AMQP protocol.

Proposed solution

Use gRPC to provide some similar header/subject routing capabilities when used by Cloud PubSub. This should deliver content as JSON messages wrapped in a Protocol Buffer definition that is compatible with Pub/Sub features of Cloud PubSub. Avoid attempting to make the message specification rigid by translating the message specification to a Protocol Buffer interface.

Description

When running CxAnalytix with lots of scan data to process (in this case 52 projects with 48 scans) an exception can occur that causes all data to not be extracted due to a System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted.

Expected Behavior

No exceptions should occur regardless of how many projects or scans are being extracted in a single run or lifetime of the CxAnalytix process.

Actual Behavior

PS C:\programdata\checkmarx\cxanalytix\artifacts\Release> dotnet CxAnalytixCLI.dll
[2020-08-11 21:21:31,419]  INFO [1] [CxAnalytixCLI.Program] (?:?) - Start
[2020-08-11 21:21:31,432]  INFO [1] [CxAnalytixCLI.Program] (?:?) - CWD: C:\programdata\checkmarx\cxanalytix\artifacts\Release
[2020-08-11 21:21:31,884]  INFO [1] [CxAnalytix.Out.MongoDBOutput.MongoDBOutFactory] (?:?) - 2 calculated shard keys have been defined.
[2020-08-11 21:21:32,299]  WARN [1] [CxAnalytix.Out.MongoDBOutput.MongoDBOutFactory] (?:?) - Database CxAnalytix does not exist, it will be created.
[2020-08-11 21:21:32,316]  INFO [1] [CxAnalytix.Out.MongoDBOutput.MongoUtil] (?:?) - Creating collection RECORD_SAST_Scan_Detail
[2020-08-11 21:21:32,343]  INFO [1] [CxAnalytix.Out.MongoDBOutput.MongoUtil] (?:?) - Creating collection RECORD_SAST_Scan_Summary
[2020-08-11 21:21:32,354]  INFO [1] [CxAnalytix.Out.MongoDBOutput.MongoUtil] (?:?) - Creating collection RECORD_SCA_Scan_Summary
[2020-08-11 21:21:32,365]  INFO [1] [CxAnalytix.Out.MongoDBOutput.MongoUtil] (?:?) - Creating collection RECORD_SCA_Scan_Detail
[2020-08-11 21:21:32,376]  INFO [1] [CxAnalytix.Out.MongoDBOutput.MongoUtil] (?:?) - Creating collection RECORD_Project_Info
[2020-08-11 21:21:32,386]  INFO [1] [CxAnalytix.Out.MongoDBOutput.MongoUtil] (?:?) - Creating collection RECORD_Policy_Violations
[2020-08-11 21:21:32,408]  WARN [1] [CxAnalytix.TransformLogic.Transformer] (?:?) - Policy data is not available.
System.InvalidOperationException: Unable to retrieve policies.
   at CxRestClient.CxMnoPolicies.GetAllPolicies(CxRestContext ctx, CancellationToken token) in c:\programdata\checkmarx\CxAnalytix\CxRestClient\CxMnoPolicies.cs:line 128
   at CxAnalytix.TransformLogic.Transformer..ctor(CxRestContext ctx, CancellationToken token, String previousStatePath) in c:\programdata\checkmarx\CxAnalytix\TransformLogic\Transformer.cs:line 260
[2020-08-11 21:21:32,856]  INFO [1] [CxAnalytix.TransformLogic.ProjectResolver] (?:?) - 48 projects are targets for check for new scans. Since last scan: 0 projects removed, 48 new projects.
[2020-08-11 21:21:33,437]  INFO [1] [CxAnalytix.TransformLogic.ScanResolver] (?:?) - Resolved 52 scans to check in 48 projects since 8/11/2020 9:21:32 PM.
[2020-08-11 21:23:00,476]  WARN [1] [CxAnalytix.TransformLogic.Transformer] (?:?) - Error attempting to retrieve the SAST XML report for 1000006 in project 7: NodeGoat_sandbox.
System.AggregateException: One or more errors occurred. (Only one usage of each socket address (protocol/network address/port) is normally permitted) ---> System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at CxRestClient.CxSastScanReportGenStatus.GetReportGenerationStatus(CxRestContext ctx, CancellationToken token, String reportId) in c:\programdata\checkmarx\CxAnalytix\CxRestClient\CxSastScanReportGenStatus.cs:line 44
   at CxRestClient.CxSastXmlReport.GetXmlReport(CxRestContext ctx, CancellationToken token, String scanId) in c:\programdata\checkmarx\CxAnalytix\CxRestClient\CxSastXmlReport.cs:line 33
   at CxAnalytix.TransformLogic.Transformer.SastReportOutput(ScanDescriptor scan, Transformer inst) in c:\programdata\checkmarx\CxAnalytix\TransformLogic\Transformer.cs:line 68
---> (Inner Exception #0) System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)<---

[2020-08-11 21:23:00,494]  WARN [1] [CxAnalytix.TransformLogic.Transformer] (?:?) - Error attempting to retrieve the SAST XML report for 1000004 in project 5: dvna_sandbox.
System.AggregateException: One or more errors occurred. (Only one usage of each socket address (protocol/network address/port) is normally permitted) ---> System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at CxRestClient.CxSastGenerateScanReport.GetGeneratedReportId(CxRestContext ctx, CancellationToken token, String scanId, ReportTypes type) in c:\programdata\checkmarx\CxAnalytix\CxRestClient\CxSastGenerateScanReport.cs:line 63
   at CxRestClient.CxSastXmlReport.GetXmlReport(CxRestContext ctx, CancellationToken token, String scanId) in c:\programdata\checkmarx\CxAnalytix\CxRestClient\CxSastXmlReport.cs:line 26
   at CxAnalytix.TransformLogic.Transformer.SastReportOutput(ScanDescriptor scan, Transformer inst) in c:\programdata\checkmarx\CxAnalytix\TransformLogic\Transformer.cs:line 68
---> (Inner Exception #0) System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)<---

[2020-08-11 21:23:00,515]  WARN [1] [CxAnalytix.TransformLogic.Transformer] (?:?) - Error attempting to retrieve the SAST XML report for 1000001 in project 2: WebGoat_policy.
System.AggregateException: One or more errors occurred. (Only one usage of each socket address (protocol/network address/port) is normally permitted) ---> System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at CxRestClient.CxSastGenerateScanReport.GetGeneratedReportId(CxRestContext ctx, CancellationToken token, String scanId, ReportTypes type) in c:\programdata\checkmarx\CxAnalytix\CxRestClient\CxSastGenerateScanReport.cs:line 63
   at CxRestClient.CxSastXmlReport.GetXmlReport(CxRestContext ctx, CancellationToken token, String scanId) in c:\programdata\checkmarx\CxAnalytix\CxRestClient\CxSastXmlReport.cs:line 26
   at CxAnalytix.TransformLogic.Transformer.SastReportOutput(ScanDescriptor scan, Transformer inst) in c:\programdata\checkmarx\CxAnalytix\TransformLogic\Transformer.cs:line 68
---> (Inner Exception #0) System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)<---

Reproduction

CxAnalytix 1.1.3 on Windows Server 2016. Starting with a clean State File and needing to extract data for 52 projects and 48 scans. Configuration is set for MongoDB:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="CxCredentials" type="CxAnalytix.Configuration.CxCredentials, Configuration" />
    <section name="CxConnection" type="CxAnalytix.Configuration.CxConnection, Configuration" />
    <section name="CxAnalyticsService" type="CxAnalytix.Configuration.CxAnalyticsService, Configuration" />
    <section name="CxLogOutput" type="CxAnalytix.Out.Log4NetOutput.LogOutputConfig, Log4NetOutput" />
	<section name="CxMongoOutput" type="CxAnalytix.Out.MongoDBOutput.MongoOutConfig, MongoDBOutput" />
  </configSections>

  <!-- Common config parameters -->
  <CxConnection URL="http://localhost"
                mnoURL=""
                TimeoutSeconds="600" ValidateCertificates="true" />
  <CxCredentials configProtectionProvider="DataProtectionConfigurationProvider">
    <EncryptedData>
      <CipherData>
        <CipherValue>AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAD+Yfz3TTtkuNUwiA2mnaNQQAAAACAAAAAAAQZgAAAAEAACAAAABEFil6yJornAfayWri4jhnYx8ZcVKlCbdK0MKf0OYbPQAAAAAOgAAAAAIAACAAAAAoSneyAzYRay+umoLa1CEvdb/54SM3v6CwWl8QMcgPOZAAAADSspNbRLZr9vwTmqOeZhm05gVNs3yONMWuKvhfwodTOF7jGtg9uHVbc5lH8cpNxU7Qb072JAjiiCYrAjy1aCjMO5NH0ibJViL0n9euH2jJz6mibUo0VNoNfid8KQhRZqogivlzpL/rEOSOdX0qEzu3ABu35g9knhcCb8wL2kwawXAIn3vYYE8vAszERfZ8fbpAAAAACU6pegd1dCoitWECWzFd5oPxW2BLsCRkJqG30yqFsmwD0jKEh8WWyK5QPVxZ9x8TURIliyJdtMhLK/yHGKiq6Q==</CipherValue>
      </CipherData>
    </EncryptedData>
  </CxCredentials>
  <CxAnalyticsService ConcurrentThreads="2" StateDataStoragePath=""
                      ProcessPeriodMinutes="120"
                      OutputFactoryClassPath="CxAnalytix.Out.MongoDBOutput.MongoDBOutFactory, MongoDBOutput"
                      SASTScanSummaryRecordName="RECORD_SAST_Scan_Summary"
                      SASTScanDetailRecordName="RECORD_SAST_Scan_Detail"
                      SCAScanSummaryRecordName="RECORD_SCA_Scan_Summary"
                      SCAScanDetailRecordName="RECORD_SCA_Scan_Detail"
                      ProjectInfoRecordName="RECORD_Project_Info"
                      PolicyViolationsRecordName="RECORD_Policy_Violations"
                      />


  <!-- Specific output method configuration parameters -->
  <CxLogOutput DataRetentionDays="3" OutputRoot="logs\">
    <PurgeSpecs>
      <spec MatchSpec="*.log.*" />
    </PurgeSpecs>
  </CxLogOutput>

  <CxMongoOutput ConnectionString="mongodb://localhost:27017/CxAnalytix">
    <!-- This section is optional -->
    <GeneratedShardKeys>
      <!-- Each of these are optional -->
      <Spec KeyName="pkey" CollectionName="SAST_Scan_Summary" FormatSpec="{ScanType}-{ScanFinished:yyyy-dddd}"  />
      <Spec KeyName="pkey" CollectionName="SAST_Scan_Detail" FormatSpec="{ScanType}-{QueryGroup}-{ScanFinished:yyyy-dddd}" />
    </GeneratedShardKeys>
  </CxMongoOutput>

</configuration>

Build the tool as per the tutorial and run the command line. Notice that the exception (above) occurs and you do not have all of your data loaded into MongoDB.

Note:

Initial research indicates this is a known issue with HttpClient and the solution is instantiate it once (singleton) and use many times.

This issue report from Azure Functions seems to be relevant to this bug report in CxAnalytix:
Azure/azure-functions-host#1806

Environment Details

CxAnalytix v1.1.3 on Server 2016.

Values provided in configs should be able to resolve from environment variables. In essence, this would be a way to Dockerize components and not store secrets in the config xml. Example:

<CxCredentials Username="${SAST_USER}" Password ="${SAST_PASSWORD}" />

Description

Error messages about the SAST version API not working are displayed on startup if the SAST version is < 9.5. Eventually the error times out and CxAnalytix fails.

Expected Behavior

Versions 9.x should be compatible with CxAnalytix.

Actual Behavior

Failure to execute.

Environment Details

CxAnalytix 2.0.1, SAST 9.5

Description

Refreshing the auth token is currently done on first operation after token expiration. Slight clock drifts between server and client make this occasionally cause scan extractions to fail until the timeout threshold is passed and the code performs a re-auth.

Expected Behavior

401 errors trigger a token refresh, operations are retried on authentication error.

Actual Behavior

The token is not refreshed until the expiration time is passed.

Describe the problem

Currently the StateDataStoragePath is set to empty, causing the state data file to be written to the working directory. The state data file will usually be mapped to some persistent storage so that state is maintained across runs. This is difficult to do with the state data sitting in the same directory as the application.

Proposed solution

Use an ARG value with the default of "/var/cxanalytix", assign the ARG value to an ENV variable, reference ENV variable in StateDataStoragePath. Containers using checkmarxts/cxanalytix as a base container can override it at build or runtime as needed or map persistent storage to /var/cxanalytix.

Describe the problem

Output data should be able to be written to an AMQP endpoint for delivery to multiple destinations.

Proposed solution

Define an AMQP endpoint for an exchange or a queue for each record type. Record outputs are sent as messages to the AMQP endpoint. Routing and consumption logic is configured on the broker that is hosting the AMQP endpoint.

Additional details

It should support adding message attributes (headers/routing keys) per record type with values composed of elements in the record.

Message TTL should be configurable per record type.