finos / legend-engine Goto Github PK

View Code? Open in Web Editor NEW

75.0 14.0 230.0 81.63 MB

Legend Engine module

Home Page: https://legend.finos.org

License: Apache License 2.0

Java 97.72% ANTLR 2.22% HTML 0.04% CSS 0.01% Dockerfile 0.01% Shell 0.01%

legend modeling

legend-engine's Introduction

legend-engine

Execution engine for Legend. It provides:

A Pure parser and compiler.
An execution engine, generating and/or executing execution plans when provided with a Pure function, a Mapping and a Runtime.
Access point for model transformers written using the Pure language (soon to come).

Development setup

This applications uses Maven 3.6+ and JDK 11. Simply run mvn install to compile.
In order to start the server, please use the Main class org.finos.legend.engine.server.Server with the parameters: server legend-engine-server/src/test/resources/org/finos/legend/engine/server/test/userTestConfig.json.
You can test by trying http://127.0.0.1:6300 in a web browser. The swagger page can be accessed at http://127.0.0.1:6300/api/swagger

Given the size of the project, developers should consider using paralellism to speed up build times, given enough RAM is provided. In IntelliJ IDEA, this can be done by setting Thread Count (in Preferences > Build, Execution, Deployment > Build Tools > Maven) and Shared build process heap size (Mbytes) (in Preferences > Build, Execution, Deployment > Compiler > Java Compiler). For example, setting Thread Count to 4 and Shared build process heap size (Mbytes) to 30000 can significantly reduce build times, you need to fine-tune this to match your machine's specs.

⚠️ It's also important to disable Clear output directory on rebuild in Preferences > Build, Execution, Deployment > Complier as this blows away generated resources needed for running the servers from the IDE.

Starting Pure IDE

If you're making changes to the Pure codebase, it's highly recommended that you also spin up our Pure IDE application:

In order to start the server, please use the Main class org.finos.legend.engine.ide.PureIDELight with the parameters: server legend-engine-pure-ide-light/src/main/resources/ideLightConfig.json.
You can now access the IDE at http://127.0.0.1:9200/ide in a web browser.

Roadmap

Visit our roadmap to know more about the upcoming features.

Contributing

Visit Legend Contribution Guide to learn how to contribute to Legend.

License

Distributed under the Apache License, Version 2.0.

SPDX-License-Identifier: Apache-2.0

legend-engine's People

Contributors

Stargazers

Watchers

Forkers

akphi kevin-m-knight-gs aziemchawdhary-gs pierredebelen dave-wathen cptteddy hardikmaheshwari davidharte-gs fagan2888 emilia-sokol-gs stjordanis yannangao-gs markseavers junjikatto brooklynrob gs-ssh16 rjwinchester gs-gunjan inahanninger-gs diffblue-benchmarks afine-gs rajamony erwin-jeanlouis-gs hagerupe mauriciouyaguari epsstan megamuskeln maoo piy0999 kshradhan beekemarie varunmaddipati maverick729 annemxu ten-bits aayush81198 gayathrir11 abhishoya-gs agpooja umarphaarook prateekgarg-gs marcinjalmuzna rafaelbey stephanof ahmed-hassan-gs saisha95 jinanisha ivan-kyosev-gs nsprimmer saisha17 thejuanandonly99 samjcs bhatans davidsunglee hausea prasar-ashutosh jackie-h marinaelson kumuwu gagan0711 kwaseth-gs aamend christian-gallagher gs-hg ghub-real dihuibao-gs vj2050gs hmmuthus-gs poojaparlapalli nickyhof chloeminkyung gs-manvig handong890 aormerod-gs rishav-1024 dmitrietkine roasher tharunk-gs openceo surya-palaniswamy jayita-m sharvanigs sahil37 krishnapriya-04 rashyd-timkaiev-gs kishalayghosh shubham43 filipfutera detkin82 gs-kotang prishajeeva dtomcik-gs siaka-akash chiragsaluja itdnk sravyaduggirala relationalai gbrgr ankitamritraj mkadziela-gs

legend-engine's Issues

CVE-2017-18640 (High) detected in snakeyaml-1.24.jar

CVE-2017-18640 - High Severity Vulnerability

Vulnerable Library - snakeyaml-1.24.jar

YAML 1.1 parser and emitter for Java

Library home page: http://www.snakeyaml.org

Path to dependency file: legend-engine/legend-engine-server/pom.xml

Path to vulnerable library: /home/wss-scanner/.m2/repository/org/yaml/snakeyaml/1.24/snakeyaml-1.24.jar

Dependency Hierarchy:

legend-shared-pac4j-0.0.7.jar (Root Library)
- jackson-dataformat-yaml-2.10.3.jar
  - ❌ snakeyaml-1.24.jar (Vulnerable Library)

Found in HEAD commit: e1ef8a7daf4f0c08b2b72e8b800166776a25d6cb

Found in base branch: master

Vulnerability Details

The Alias feature in SnakeYAML 1.18 allows entity expansion during a load operation, a related issue to CVE-2003-1564.

Publish Date: 2019-12-12

URL: CVE-2017-18640

CVSS 3 Score Details (7.5)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: None
- Integrity Impact: None
- Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://bitbucket.org/asomov/snakeyaml/commits/da11ddbd91c1f8392ea932b37fa48110fa54ed8c

Release Date: 2020-03-08

Fix Resolution: 1.26

Upgrade Eclipse Collections library dependencies from 10.2 to 11.1

Feature Request

Description of Problem:

Most of the repositories for Legend currently depend on Eclipse Collections 10.2 which is over two years old. The latest 10.x release was 10.4 and was released in August 2020. Eclipse Collections 11.1 was released in July 2022 and is the current release.

I would recommend upgrading to the latest version of Eclipse Collections (11.1) in each Legend Java Repository to stay up to date with the latest features and bug fixes.

Potential Solutions:

Upgrade dependencies in pom files replacing Eclipse Collections 10.2 with 11.1 in the following repositories.

legend-pure
legend-engine
legend-sdlc
legend-depot

I looked across all of the Legend Java language repositories for references to Eclipse Collections in the parent pom. It's possible I missed a repository.

Representation in the JSON extract of objects that are inherited somewhere else in the model

Bug Report

Steps to Reproduce:

Go to the CDM project
Generate a JSON representation of the FloatingRate class

Expected Result:

The floatingRateMultiplierSchedule attribute should be represented as below in the JSON extract (see attached FloatingRate_Expected.txt for the full extract)
"floatingRateMultiplierSchedule": {
"$ref": "RateSchedule.json",

Actual Result:

In the JSON representation the floatingRateMultiplierSchedule is specified as below in the JSON extract (see attached FloatingRate_Actual.txt for the full extract)
"floatingRateMultiplierSchedule": {
"oneOf": [
{
"$ref": "RateSchedule.json"
},
{
"$ref": "SpreadSchedule.json"
},
{
"$ref": "StrikeSchedule.json"
}
],

Environment:

legend-studio: 6.6.0
legend-sdlc-server: 0.83.1
legend-engine-server: 3.9.2

Additional Context:

This behavior is happening when a class is inherited in other parts of the model. The Legend file generation then represents that class alongside the classes that inherit it. If you look at the CDM model, you can see that the RateSchedule is a base class for the SpreadSchedule and the StrikeSchedule. Those correspond to the 3 entries in the oneOf list above.
One way around that could be to create base classes that aren't actually used in the model (e.g. a RateScheduleBase that would exist alongside the RateSchedule), which is what I have done to generate the FloatingRate_Expected.txt extract.
The broader issue is that this creates a problem for supporting a CDM model representation through Legend, as this design pattern is widely used by the CDM.
FloatingRate_Actual.txt
FloatingRate_Expected.txt

CVE-2018-1000873 (Medium) detected in jackson-datatype-jsr310-2.9.6.jar - autoclosed

CVE-2018-1000873 - Medium Severity Vulnerability

Vulnerable Library - jackson-datatype-jsr310-2.9.6.jar

Add-on module to support JSR-310 (Java 8 Date & Time API) data types.

Library home page: https://github.com/FasterXML/jackson-modules-java8/

Path to dependency file: legend-engine/legend-engine-server/pom.xml

Path to vulnerable library: /home/wss-scanner/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.9.6/jackson-datatype-jsr310-2.9.6.jar

Dependency Hierarchy:

dropwizard-core-1.3.5.jar (Root Library)
- dropwizard-jackson-1.3.5.jar
  - ❌ jackson-datatype-jsr310-2.9.6.jar (Vulnerable Library)

Found in HEAD commit: e1ef8a7daf4f0c08b2b72e8b800166776a25d6cb

Found in base branch: master

Vulnerability Details

Fasterxml Jackson version Before 2.9.8 contains a CWE-20: Improper Input Validation vulnerability in Jackson-Modules-Java8 that can result in Causes a denial-of-service (DoS). This attack appear to be exploitable via The victim deserializes malicious input, specifically very large values in the nanoseconds field of a time value. This vulnerability appears to have been fixed in 2.9.8.

Publish Date: 2018-12-20

URL: CVE-2018-1000873

CVSS 3 Score Details (6.5)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: Required
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: None
- Integrity Impact: None
- Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nvd.nist.gov/vuln/detail/CVE-2018-1000873

Release Date: 2018-12-20

Fix Resolution: 2.9.8

Add support for executing a chain of mappings in graphFetch queries

Feature Request

Add support for executing a chain of mappings in graphFetch queries

Description of Problem:

Currently, execution of a graphFetch query on a single mapping is supported. Ability to execute a chain of mappings ( M2M -> (M2M)* -> (Any Store) ) would be a good addition.

Potential Solutions:

Add support for ModelChainConnection in graphFetch queries

service execution runtime shows invalid validation warnings due to runtimes not being checked in dependencies

Bug Report

The following bug shows for the execution runtime:
runtime is not associated with specified mapping

Steps to Reproduce:

use valid dependency runtime in service execution
warning will show up but runtime is valid

The solution is to check dependency runtimes as well as ownRuntimes when doing validation.

Service Store supports recursive models

Feature Request

Description of Problem:

Currently, if a model has a property of type (same as the class itself) we are unable to compile ServiceStore Mapping because we end in an infinite loop finding property mappings.

Such models are generally speaking non-serializable but the platform may provide a way for users to work with such models (some potential ideas listed below)

Steps to reproduce :

    public void testRecursiveModelWithServiceStore()
    {
        String grammar = "###Pure\n" +
                "import meta::external::store::service::showcase::domain::*;\n" +
                "\n" +
                "Class meta::external::store::service::showcase::domain::Firm\n" +
                "{\n" +
                "    firmName     : String[1];\n" +
                "    firmId       : Integer[1];\n" +
                "    subsidiaries : Firm[*];\n" +
                "}\n" +
                "\n" +
                "###ServiceStore\n" +
                "ServiceStore meta::external::store::service::showcase::store::FirmServiceStore\n" +
                "(\n" +
                "   Service FirmService\n" +
                "   (\n" +
                "      path     : '/firms';\n" +
                "      method   : GET;\n" +
                "      security : [];\n" +
                "      response : [meta::external::store::service::showcase::domain::Firm <- meta::external::store::service::showcase::store::FirmResponseSchemaBinding];\n" +
                "   )\n" +
                ")\n" +
                "\n" +
                "###ExternalFormat\n" +
                "Binding meta::external::store::service::showcase::store::FirmResponseSchemaBinding\n" +
                "{\n" +
                "  contentType   : 'application/json';\n" +
                "  modelIncludes : [\n" +
                "                    meta::external::store::service::showcase::domain::Firm\n" +
                "                  ];\n" +
                "}\n\n" +
                "###Mapping\n" +
                "Mapping meta::external::store::service::showcase::mapping::ServiceStoreMapping\n" +
                "(\n" +
                "    *meta::external::store::service::showcase::domain::Firm: ServiceStore\n" +
                "    {\n" +
                "        ~service [meta::external::store::service::showcase::store::FirmServiceStore] FirmService\n" +
                "    }\n" +
                ")\n\n";

        Pair<PureModelContextData, PureModel> result = org.finos.legend.engine.language.pure.compiler.test.TestCompilationFromGrammar.TestCompilationFromGrammarTestSuite.test(grammar);
    }

Potential Solutions:

Some ideas

Platform may provide a way for users to indicate depth till which recursiveness is considered/property mappings are computed
Platform can compute such property mappings on the fly in context of query

CVE-2019-10241 (Medium) detected in multiple libraries - autoclosed

CVE-2019-10241 - Medium Severity Vulnerability

Vulnerable Libraries - jetty-util-9.4.11.v20180605.jar, jetty-servlet-9.4.11.v20180605.jar, jetty-server-9.4.11.v20180605.jar

jetty-util-9.4.11.v20180605.jar

Utility classes for Jetty

Library home page: http://www.eclipse.org/jetty

Path to dependency file: legend-engine/legend-engine-server/pom.xml

Path to vulnerable library: /home/wss-scanner/.m2/repository/org/eclipse/jetty/jetty-util/9.4.11.v20180605/jetty-util-9.4.11.v20180605.jar

Dependency Hierarchy:

jetty-servlets-9.4.11.v20180605.jar (Root Library)
- ❌ jetty-util-9.4.11.v20180605.jar (Vulnerable Library)

jetty-servlet-9.4.11.v20180605.jar

Jetty Servlet Container

Library home page: http://www.eclipse.org/jetty

Path to dependency file: legend-engine/legend-engine-server/pom.xml

Path to vulnerable library: /home/wss-scanner/.m2/repository/org/eclipse/jetty/jetty-servlet/9.4.11.v20180605/jetty-servlet-9.4.11.v20180605.jar

Dependency Hierarchy:

dropwizard-jetty-1.3.5.jar (Root Library)
- ❌ jetty-servlet-9.4.11.v20180605.jar (Vulnerable Library)

jetty-server-9.4.11.v20180605.jar

The core jetty server artifact.

Library home page: http://www.eclipse.org/jetty

Path to dependency file: legend-engine/legend-engine-server/pom.xml

Path to vulnerable library: canner/.m2/repository/org/eclipse/jetty/jetty-server/9.4.11.v20180605/jetty-server-9.4.11.v20180605.jar

Dependency Hierarchy:

❌ jetty-server-9.4.11.v20180605.jar (Vulnerable Library)

Found in HEAD commit: e1ef8a7daf4f0c08b2b72e8b800166776a25d6cb

Found in base branch: master

Vulnerability Details

In Eclipse Jetty version 9.2.26 and older, 9.3.25 and older, and 9.4.15 and older, the server is vulnerable to XSS conditions if a remote client USES a specially formatted URL against the DefaultServlet or ResourceHandler that is configured for showing a Listing of directory contents.

Publish Date: 2019-04-22

URL: CVE-2019-10241

CVSS 3 Score Details (6.1)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: Required
- Scope: Changed
Impact Metrics:
- Confidentiality Impact: Low
- Integrity Impact: Low
- Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10241

Release Date: 2019-04-22

Fix Resolution: org.eclipse.jetty:jetty-server:9.2.27,9.3.26,9.4.16,org.eclipse.jetty:jetty-servlet:9.2.27,9.3.26,9.4.16,org.eclipse.jetty:jetty-util:9.2.27,9.3.26,9.4.16

Snowflake connector - NPE when getting a database connection

Bug Report

Code that acquires a database connection fails with an NPE. See stack trace under 'Additional Context'

Steps to Reproduce:

Run an instance of Legend
Model and execute a relational query/service

Expected Result:

Legend should be able to connect to the database and execute the execution plan

Actual Result:

Legend fails with a NPE.

DataSourceSpecification.java [1] attempts to create a connection. This code uses ProfileManagerHelper.java to extract a javax.subject that represents the currently logged in user [2]. However ProfileManagerHelper.java supports only the KerberosProfile. Obviously in cases where Kerberos is not used, this returns a null.

DataSourceSpecification.java then delegates to the authentication strategy's getAlternativePrinicipal method which in the case of Snowflake returns a null.

Eventually the null subject returns in a null principal which leads to the NPE.

[1]

legend-engine/legend-engine-executionPlan-execution-store-relational-connection/src/main/java/org/finos/legend/engine/plan/execution/stores/relational/connection/ds/DataSourceSpecification.java

Line 121 in 35a598e

    
           public Connection getConnectionUsingProfiles(MutableList<CommonProfile> profiles)

[2]

legend-engine/legend-engine-shared-core/src/main/java/org/finos/legend/engine/shared/core/kerberos/ProfileManagerHelper.java

Line 35 in 35a598e

    
           public static Subject extractSubject(Iterable<? extends CommonProfile> profiles)

[3] https://github.com/finos/legend-engine/blob/35a598e94c736b13c9481e9a4bc7c5f2918521c7/legend-engine-executionPlan-execution-store-relational-connection/src/main/java/org/finos/legend/engine/plan/execution/stores/relational/connection/authentication/strategy/SnowflakePublicAuthenticationStrategy.java

Environment:

Reproducible with the latest version of Legend and the Snowflake connector.

Additional Context:

See below for a sample stack trace. This trace is from my fork with a BigQuery connector. But the same is true for the Snowflake connector as well.

INFO [dw-53 - POST /api/pure/v1/execution/execute] org.finos.legend.engine.plan.execution.stores.relational.connection.ds.DataSourceSpecification - Get Connection for null from DataSourceSpecification[BigQueryDataSourceSpecification,BigQuerySpecificationKey{projectId='project1',datasetName='dataset1'},org.finos.legend.engine.plan.execution.stores.relational.connection.ds.specifications.BigQueryDataSourceSpecification@7948af2b] 2021-06-04 21:31:33.919 ERROR [dw-53 - POST /api/pure/v1/execution/execute] Alloy Execution Server - {"duration":0.0,"eventType":"EXECUTE_INTERACTIVE_ERROR","info":{"code":-1,"message":"NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null","status":"error","trace":"java.lang.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null\n\tat org.eclipse.collections.impl.map.mutable.ConcurrentHashMap.hash(ConcurrentHashMap.java:186)\n\tat org.eclipse.collections.impl.map.mutable.ConcurrentHashMap.get(ConcurrentHashMap.java:709)\n\tat org.finos.legend.engine.plan.execution.stores.relational.connection.ds.DataSourceSpecification.getConnection(DataSourceSpecification.java:166)\n\tat org.finos.legend.engine.plan.execution.stores.relational.connection.ds.DataSourceSpecification.getConnectionUsingProfiles(DataSourceSpecification.java:129)\n\tat org.finos.legend.engine.plan.execution.stores.relational.connection.manager.ConnectionManagerSelector.getDatabaseConnection(ConnectionManagerSelector.java:62)\n\tat org.finos.legend.engine.plan.execution.stores.relational.RelationalExecutor.getConnection(RelationalExecutor.java:352)\n\tat org.finos.legend.engine.plan.execution.stores.relational.RelationalExecutor.getConnection(RelationalExecutor.java:334)\n\tat org.finos.legend.engine.plan.execution.stores.relational.RelationalExecutor.execute(RelationalExecutor.java:229)\n\tat org.finos.legend.engine.plan.execution.stores.relational.plugin.RelationalExecutionNodeExecutor.visit(RelationalExecutionNodeExecutor.java:237)\n\tat org.finos.legend.engine.plan.execution.stores.relational.plugin.RelationalExecutionNodeExecutor.visit(RelationalExecutionNodeExecutor.java:251)\n\tat org.finos.legend.engine.plan.execution.stores.relational.plugin.RelationalExecutionNodeExecutor.visit(RelationalExecutionNodeExecutor.java:96)\n\tat org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.RelationalTdsInstantiationExecutionNode.accept(RelationalTdsInstantiationExecutionNode.java:22)\n\tat org.finos.legend.engine.plan.execution.stores.relational.RelationalExecutionExtension.lambda$getExtraNodeExecutors$fbe53481$1(RelationalExecutionExtension.java:50)\n\tat org.finos.legend.engine.plan.execution.nodes.ExecutionNodeExecutor.lambda$visit$0(ExecutionNodeExecutor.java:99)\n\tat

Graph query execution with adaptive batching

Feature Request

Legend engine provides batch execution capability for graph fetch queries. Batch execution capability lets the engine perform "graphFetch" operation (fetch the nested property structure as per the query) on a list of objects in one go.

This feature significantly improves performance of graph queries, specially in the case of relational stores where the cost of executing a SQL query is high. Let's consider a situation where a graph fetch operation is being performed on 2000 objects and the fetch tree requires 5 SQLs to be executed. Without batching, the number of SQL executions would be (2000 * 5 = 10000). The same graph fetch operation when done with a batch size of 1000 would require (2 * 5 = 10) SQL executions which is a lot less (2 is the number of batches). However, executing graph fetch with batching would increase the memory requirement from server perspective.

Description of Problem:

Currently, batch size is configured in the query as an extra parameter to graph fetch operation. It remains constant during the execution. Also, there is a memory limit which is put up to cap the memory being consumed for each batch in the execution (can result in failures if memory limit is breached). It requires some effort to tune the batch size and arrive at a good number, by doing a tradeoff between performance and memory utilization.

Potential Solution:

One of the ways we can tackle this problem is to build an adaptive batching mechanism. Recently, some extra stats were added to graph fetch execution in this PR: #492. The idea is to use the stats from previous batches to decide on the batch size for next batch thereby adapting to the dynamics of the execution (increasing batch size if the memory consumption is less compared to the memory limit or reducing batch size if memory consumption exceeds or is close to the memory limit). This feature can help improve performance in the case of executions which are less memory intensive and also avoid failures in executions which are memory intensive.

Add parameter values option to `execute` and `executePlan` api

Feature Request

Description of Problem:

Currently, users are not able to use the execute, execute plan API to execute queries with parameter values. The goal is to provide users with a method to provide parameter values when executing.

Potential Solutions:

https://legend-acct.finos.org/exec/api/swagger#/Pure%20-%20Execution/execute
https://legend-acct.finos.org/exec/api/swagger#/Pure%20-%20Execution/executePlan

Add a parameterValues values to executePlan and execute api to take parameterValues. We want to carry these values down after generating the execution plan and plug in those values into the plan.

Compilation failure

Bug Report

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project legend-engine-language-pure-grammar: Compilation failure: Compilation failure:
[ERROR] /D:/code/finos/legend-engine/legend-engine-language-pure-grammar/src/main/java/org/finos/legend/engine/language/pure/grammar/to/HelperEmbeddedDataGrammarComposer.java:[28,30] cannot find symbol
[ERROR] symbol: variable extraEmbeddedDataComposers
[ERROR] location: variable context of type org.finos.legend.engine.language.pure.grammar.to.PureGrammarComposerContext
[ERROR] /D:/code/finos/legend-engine/legend-engine-language-pure-grammar/src/main/java/org/finos/legend/engine/language/pure/grammar/from/data/embedded/HelperEmbeddedDataGrammarParser.java:[70,47] cannot find symbol
[ERROR] symbol: method getExtraEmbeddedDataParser(java.lang.String)
[ERROR] location: variable extensions of type org.finos.legend.engine.language.pure.grammar.from.extension.PureGrammarParserExtensions
[ERROR] /D:/code/finos/legend-engine/legend-engine-language-pure-grammar/src/main/java/org/finos/legend/engine/language/pure/grammar/from/data/embedded/ReferenceEmbeddedDataParser.java:[35,52] cannot find symbol
[ERROR] symbol: method validatePath(java.lang.String,org.finos.legend.engine.protocol.pure.v1.model.SourceInformation)
[ERROR] location: class org.finos.legend.engine.language.pure.grammar.from.PureGrammarParserUtility

Steps to Reproduce:

I was trying to build legend-engine on win10( jdk11, maven3.6.1) by ’mvn install -DskipTests‘ in the root directory of this project,
and I got the compilation failure above. I didn't find any variable named 'extraEmbeddedDataComposers' in the class 'PureGrammarComposerContext' in the source code. Besides, git-resetting the codebase to a few different previous versions did not help a little bit. Is there any step I missed in the process of building the project?

CVE-2019-17359 (High) detected in bcprov-jdk15on-1.60.jar - autoclosed

CVE-2019-17359 - High Severity Vulnerability

Vulnerable Library - bcprov-jdk15on-1.60.jar

The Bouncy Castle Crypto package is a Java implementation of cryptographic algorithms. This jar contains JCE provider and lightweight API for the Bouncy Castle Cryptography APIs for JDK 1.5 to JDK 1.8.

Library home page: http://www.bouncycastle.org/java.html

Path to dependency file: legend-engine/legend-engine-query-pure/pom.xml

Path to vulnerable library: /home/wss-scanner/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.60/bcprov-jdk15on-1.60.jar,/home/wss-scanner/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.60/bcprov-jdk15on-1.60.jar,/home/wss-scanner/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.60/bcprov-jdk15on-1.60.jar,/home/wss-scanner/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.60/bcprov-jdk15on-1.60.jar,/home/wss-scanner/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.60/bcprov-jdk15on-1.60.jar,/home/wss-scanner/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.60/bcprov-jdk15on-1.60.jar,/home/wss-scanner/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.60/bcprov-jdk15on-1.60.jar,/home/wss-scanner/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.60/bcprov-jdk15on-1.60.jar,/home/wss-scanner/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.60/bcprov-jdk15on-1.60.jar,/home/wss-scanner/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.60/bcprov-jdk15on-1.60.jar

Dependency Hierarchy:

legend-engine-language-pure-compiler-0.0.1-SNAPSHOT.jar (Root Library)
- legend-pure-runtime-java-engine-compiled-0.0.1-SNAPSHOT.jar
  - legend-pure-runtime-java-engine-shared-0.0.1-SNAPSHOT.jar
    - ❌ bcprov-jdk15on-1.60.jar (Vulnerable Library)

Found in HEAD commit: e1ef8a7daf4f0c08b2b72e8b800166776a25d6cb

Found in base branch: master

Vulnerability Details

The ASN.1 parser in Bouncy Castle Crypto (aka BC Java) 1.63 can trigger a large attempted memory allocation, and resultant OutOfMemoryError error, via crafted ASN.1 data. This is fixed in 1.64.

Publish Date: 2019-10-08

URL: CVE-2019-17359

CVSS 3 Score Details (7.5)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: None
- Integrity Impact: None
- Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17359

Release Date: 2019-10-08

Fix Resolution: org.bouncycastle:bcprov-jdk15on:1.64

Bug: Relational Association Mapping omits one field when switching between form and hacker mode

🐛 Bug Report

AssociationMapping defined under Mapping omits the second property, after round trip, which is syntactically incorrect.

In the code snippet below:

model::FirmPerson: Relational
  {
    AssociationMapping
    (
      firm[model_Person, model_Firm]: [model::Test]@FirmPerson,
      employee[model_Firm, model_Person]: [model::Test]@FirmPerson
    )
  }

the second property model_Firm in firm and the second property model_Person in employee are omitted:

model::FirmPerson: Relational
  {
    AssociationMapping
    (
      firm[model_Person]: [model::Test]@FirmPerson,
      employee[model_Firm]: [model::Test]@FirmPerson
    )
  }

Step to Reproduce

Use the snippet below to create the models and mappings
Switch to form mode
Switch back to text mode
Observe AssociationMapping definition inside Mapping each property would have the second property omitted.

###Relational
Database model::Test
(
  Table FirmTable
  (
    id INTEGER PRIMARY KEY,
    Legal_name VARCHAR(200)
  )
  Table PersonTable
  (
    id INTEGER PRIMARY KEY,
    firm_id INTEGER,
    firstName VARCHAR(200),
    lastName VARCHAR(200)
  )

  Join FirmPerson(PersonTable.firm_id = FirmTable.id)
)


###Pure
Profile model::MyExtension
{
  stereotypes: [important];
  tags: [doc];
}

Enum model::IncType
{
  Corp,
  LLC
}

Class <> {model::MyExtension.doc = 'This is a model of a firm'} model::Firm extends model::LegalEntity
[
  validName: $this.legalName->startsWith('_')
]
{
  employees: model::Person[1..*];
  incType: model::IncType[1];
  employeeSize() {$this.employees->count()}: Number[1];
}

Class model::LegalEntity
{
  legalName: String[1];
}

Class model::Person
{
  firstName: String[1];
  lastName: String[1];
}

Association model::FirmPerson
{
  firm: model::Firm[1];
  employee: model::Person[1];
}


###Mapping
Mapping model::NewMapping
(
  *model::Firm: Relational
  {
    ~primaryKey
    (
      [model::Test]FirmTable.id
    )
    ~mainTable [model::Test]FirmTable
    legalName: concat([model::Test]FirmTable.Legal_name, '_LTD'),
    employees[model_Person]: [model::Test]@FirmPerson
  }
  *model::Person: Relational
  {
    ~primaryKey
    (
      [model::Test]PersonTable.id
    )
    ~mainTable [model::Test]PersonTable
    firstName: [model::Test]PersonTable.firstName,
    lastName: [model::Test]PersonTable.lastName
  }

  model::FirmPerson: Relational
  {
    AssociationMapping
    (
      firm[model_Person, model_Firm]: [model::Test]@FirmPerson,
      employee[model_Firm, model_Person]: [model::Test]@FirmPerson
    )
  }

)


###Connection
RelationalDatabaseConnection model::test
{
  store: model::Test;
  type: H2;
  specification: LocalH2
  {
    testDataSetupSqls: [
      'Drop table if exists FirmTable;\r\nDrop table if exists PersonTable;\r\nCreate Table FirmTable(id INT, Legal_Name VARCHAR(200));\r\nCreate Table PersonTable(id INT, firm_id INT, lastName VARCHAR(200), firstName VARCHAR(200));\r\nInsert into FirmTable (id, Legal_Name) values (1, \'FirmX\');\r\nInsert into FirmTable (id, Legal_Name) values (2, \'FirmY\');\r\nInsert into FirmTable (id, Legal_Name) values (3, \'FirmZ\');\r\nInsert into PersonTable (id, firm_id, lastName, firstName) values (1, 1, \'John\', \'Doe\');\r\nInsert into PersonTable (id, firm_id, lastName, firstName) values (2, 1, \'Johkiun\', \'Douip.e\');\r\nInsert into PersonTable (id, firm_id, lastName, firstName) values (3, 2, \'Jauaohn\', \'Douipe\');\r\nInsert into PersonTable (id, firm_id, lastName, firstName) values (4, 2, \'Johyyfpn\', \'Doeuid\');\r\n'
      ];
  };
  auth: DefaultH2;
}

Expected behavior

The syntax for AssociationMapping should remain the same as it is in the above snippet

Possible Solution

CVE-2019-10247 (Medium) detected in jetty-server-9.4.11.v20180605.jar - autoclosed

CVE-2019-10247 - Medium Severity Vulnerability

Vulnerable Library - jetty-server-9.4.11.v20180605.jar

The core jetty server artifact.

Library home page: http://www.eclipse.org/jetty

Path to dependency file: legend-engine/legend-engine-server/pom.xml

Path to vulnerable library: canner/.m2/repository/org/eclipse/jetty/jetty-server/9.4.11.v20180605/jetty-server-9.4.11.v20180605.jar

Dependency Hierarchy:

❌ jetty-server-9.4.11.v20180605.jar (Vulnerable Library)

Found in HEAD commit: e1ef8a7daf4f0c08b2b72e8b800166776a25d6cb

Found in base branch: master

Vulnerability Details

In Eclipse Jetty version 7.x, 8.x, 9.2.27 and older, 9.3.26 and older, and 9.4.16 and older, the server running on any OS and Jetty version combination will reveal the configured fully qualified directory base resource location on the output of the 404 error for not finding a Context that matches the requested path. The default server behavior on jetty-distribution and jetty-home will include at the end of the Handler tree a DefaultHandler, which is responsible for reporting this 404 error, it presents the various configured contexts as HTML for users to click through to. This produced HTML includes output that contains the configured fully qualified directory base resource location for each context.

Publish Date: 2019-04-22

URL: CVE-2019-10247

CVSS 3 Score Details (5.3)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: Low
- Integrity Impact: None
- Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://bugs.eclipse.org/bugs/show_bug.cgi?id=546577

Release Date: 2019-04-22

Fix Resolution: 9.2.28.v20190418

Check this box to open an automated fix PR

Failing to report error message for compilation issue within a nested `if` statement

Bug Report

Steps to Reproduce

Compile below PureModelContextData. There is an expected error message of an unknown variable (fullNameL) being used. However no error message is returned as part of the response.

An Error message of unkown variable message should be returned to provide user with a path to fix their code.

JSON Payload

{
  "_type": "data",
  "elements": [
    {
      "_type": "class",
      "constraints": [],
      "name": "Person",
      "originalMilestonedProperties": [],
      "package": "model",
      "properties": [
        {
          "multiplicity": {
            "lowerBound": 1,
            "upperBound": 1
          },
          "name": "firstName",
          "propertyTypeSourceInformation": {
            "endColumn": 19,
            "endLine": 3,
            "sourceId": "",
            "startColumn": 14,
            "startLine": 3
          },
          "sourceInformation": {
            "endColumn": 23,
            "endLine": 3,
            "sourceId": "",
            "startColumn": 3,
            "startLine": 3
          },
          "stereotypes": [],
          "taggedValues": [],
          "type": "String"
        },
        {
          "multiplicity": {
            "lowerBound": 1,
            "upperBound": 1
          },
          "name": "lastName",
          "propertyTypeSourceInformation": {
            "endColumn": 18,
            "endLine": 4,
            "sourceId": "",
            "startColumn": 13,
            "startLine": 4
          },
          "sourceInformation": {
            "endColumn": 22,
            "endLine": 4,
            "sourceId": "",
            "startColumn": 3,
            "startLine": 4
          },
          "stereotypes": [],
          "taggedValues": [],
          "type": "String"
        }
      ],
      "qualifiedProperties": [
        {
          "body": [
            {
              "_type": "func",
              "function": "if",
              "parameters": [
                {
                  "_type": "var",
                  "name": "noCheck",
                  "sourceInformation": {
                    "endColumn": 64,
                    "endLine": 5,
                    "sourceId": "",
                    "startColumn": 57,
                    "startLine": 5
                  }
                },
                {
                  "_type": "lambda",
                  "body": [
                    {
                      "_type": "boolean",
                      "multiplicity": {
                        "lowerBound": 1,
                        "upperBound": 1
                      },
                      "sourceInformation": {
                        "endColumn": 71,
                        "endLine": 5,
                        "sourceId": "",
                        "startColumn": 68,
                        "startLine": 5
                      },
                      "values": [
                        true
                      ]
                    }
                  ],
                  "parameters": [],
                  "sourceInformation": {
                    "endColumn": 71,
                    "endLine": 5,
                    "sourceId": "",
                    "startColumn": 67,
                    "startLine": 5
                  }
                },
                {
                  "_type": "lambda",
                  "body": [
                    {
                      "_type": "func",
                      "function": "if",
                      "parameters": [
                        {
                          "_type": "func",
                          "function": "equal",
                          "parameters": [
                            {
                              "_type": "func",
                              "function": "plus",
                              "parameters": [
                                {
                                  "_type": "collection",
                                  "multiplicity": {
                                    "lowerBound": 2,
                                    "upperBound": 2
                                  },
                                  "sourceInformation": {
                                    "endColumn": 110,
                                    "endLine": 5,
                                    "sourceId": "",
                                    "startColumn": 95,
                                    "startLine": 5
                                  },
                                  "values": [
                                    {
                                      "_type": "property",
                                      "parameters": [
                                        {
                                          "_type": "var",
                                          "name": "this",
                                          "sourceInformation": {
                                            "endColumn": 83,
                                            "endLine": 5,
                                            "sourceId": "",
                                            "startColumn": 79,
                                            "startLine": 5
                                          }
                                        }
                                      ],
                                      "property": "firstName",
                                      "sourceInformation": {
                                        "endColumn": 93,
                                        "endLine": 5,
                                        "sourceId": "",
                                        "startColumn": 85,
                                        "startLine": 5
                                      }
                                    },
                                    {
                                      "_type": "property",
                                      "parameters": [
                                        {
                                          "_type": "var",
                                          "name": "this",
                                          "sourceInformation": {
                                            "endColumn": 101,
                                            "endLine": 5,
                                            "sourceId": "",
                                            "startColumn": 97,
                                            "startLine": 5
                                          }
                                        }
                                      ],
                                      "property": "lastName",
                                      "sourceInformation": {
                                        "endColumn": 110,
                                        "endLine": 5,
                                        "sourceId": "",
                                        "startColumn": 103,
                                        "startLine": 5
                                      }
                                    }
                                  ]
                                }
                              ],
                              "sourceInformation": {
                                "endColumn": 110,
                                "endLine": 5,
                                "sourceId": "",
                                "startColumn": 95,
                                "startLine": 5
                              }
                            },
                            {
                              "_type": "var",
                              "name": "fullNameL",
                              "sourceInformation": {
                                "endColumn": 125,
                                "endLine": 5,
                                "sourceId": "",
                                "startColumn": 116,
                                "startLine": 5
                              }
                            }
                          ],
                          "sourceInformation": {
                            "endColumn": 114,
                            "endLine": 5,
                            "sourceId": "",
                            "startColumn": 113,
                            "startLine": 5
                          }
                        },
                        {
                          "_type": "lambda",
                          "body": [
                            {
                              "_type": "boolean",
                              "multiplicity": {
                                "lowerBound": 1,
                                "upperBound": 1
                              },
                              "sourceInformation": {
                                "endColumn": 132,
                                "endLine": 5,
                                "sourceId": "",
                                "startColumn": 129,
                                "startLine": 5
                              },
                              "values": [
                                true
                              ]
                            }
                          ],
                          "parameters": [],
                          "sourceInformation": {
                            "endColumn": 132,
                            "endLine": 5,
                            "sourceId": "",
                            "startColumn": 128,
                            "startLine": 5
                          }
                        },
                        {
                          "_type": "lambda",
                          "body": [
                            {
                              "_type": "boolean",
                              "multiplicity": {
                                "lowerBound": 1,
                                "upperBound": 1
                              },
                              "sourceInformation": {
                                "endColumn": 140,
                                "endLine": 5,
                                "sourceId": "",
                                "startColumn": 136,
                                "startLine": 5
                              },
                              "values": [
                                false
                              ]
                            }
                          ],
                          "parameters": [],
                          "sourceInformation": {
                            "endColumn": 140,
                            "endLine": 5,
                            "sourceId": "",
                            "startColumn": 135,
                            "startLine": 5
                          }
                        }
                      ],
                      "sourceInformation": {
                        "endColumn": 76,
                        "endLine": 5,
                        "sourceId": "",
                        "startColumn": 75,
                        "startLine": 5
                      }
                    }
                  ],
                  "parameters": [],
                  "sourceInformation": {
                    "endColumn": 141,
                    "endLine": 5,
                    "sourceId": "",
                    "startColumn": 74,
                    "startLine": 5
                  }
                }
              ],
              "sourceInformation": {
                "endColumn": 55,
                "endLine": 5,
                "sourceId": "",
                "startColumn": 54,
                "startLine": 5
              }
            }
          ],
          "name": "isEqual",
          "parameters": [
            {
              "_type": "var",
              "class": "String",
              "multiplicity": {
                "lowerBound": 1,
                "upperBound": 1
              },
              "name": "fullName",
              "sourceInformation": {
                "endColumn": 29,
                "endLine": 5,
                "sourceId": "",
                "startColumn": 11,
                "startLine": 5
              }
            },
            {
              "_type": "var",
              "class": "Boolean",
              "multiplicity": {
                "lowerBound": 1,
                "upperBound": 1
              },
              "name": "noCheck",
              "sourceInformation": {
                "endColumn": 50,
                "endLine": 5,
                "sourceId": "",
                "startColumn": 32,
                "startLine": 5
              }
            }
          ],
          "returnMultiplicity": {
            "lowerBound": 1,
            "upperBound": 1
          },
          "returnType": "Boolean",
          "sourceInformation": {
            "endColumn": 156,
            "endLine": 5,
            "sourceId": "",
            "startColumn": 3,
            "startLine": 5
          },
          "stereotypes": [],
          "taggedValues": []
        }
      ],
      "sourceInformation": {
        "endColumn": 1,
        "endLine": 6,
        "sourceId": "",
        "startColumn": 1,
        "startLine": 1
      },
      "stereotypes": [],
      "superTypes": [],
      "taggedValues": []
    },
    {
      "_type": "sectionIndex",
      "name": "SectionIndex",
      "package": "__internal__",
      "sections": [
        {
          "_type": "importAware",
          "elements": [
            "model::Person"
          ],
          "imports": [],
          "parserName": "Pure",
          "sourceInformation": {
            "endColumn": 2,
            "endLine": 8,
            "sourceId": "",
            "startColumn": 1,
            "startLine": 1
          }
        }
      ]
    }
  ]
}

Grammar Payload

Class model::Person
{
  firstName: String[1];
  lastName: String[1];
  isEqual(fullName: String[1], noCheck: Boolean[1]) {if($noCheck, |true, |if(($this.firstName + $this.lastName) == $fullNameL, |true, |false))}: Boolean[1];
}

CVE-2019-17571 (High) detected in log4j-1.2.17.jar - autoclosed

CVE-2019-17571 - High Severity Vulnerability

Vulnerable Library - log4j-1.2.17.jar

Apache Log4j 1.2

Path to dependency file: legend-engine/legend-engine-server/pom.xml

Path to vulnerable library: canner/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar

Dependency Hierarchy:

❌ log4j-1.2.17.jar (Vulnerable Library)

Found in HEAD commit: e1ef8a7daf4f0c08b2b72e8b800166776a25d6cb

Found in base branch: master

Vulnerability Details

Included in Log4j 1.2 is a SocketServer class that is vulnerable to deserialization of untrusted data which can be exploited to remotely execute arbitrary code when combined with a deserialization gadget when listening to untrusted network traffic for log data. This affects Log4j versions up to 1.2 up to 1.2.17.

Publish Date: 2019-12-20

URL: CVE-2019-17571

CVSS 3 Score Details (9.8)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: High
- Integrity Impact: High
- Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17571

Release Date: 2019-12-20

Fix Resolution: org.apache.logging.log4j:log4j-core:2.0-alpha1

Check this box to open an automated fix PR

sdlc-server: SSL exception

Bug Report

Following along on Issue #949.

The next step is to install legend-sdlc. Checking out the repo and man install succeeded.

Next step is to start the sdlc server. However, the instructions for how to do so are a bit woolly:

A [sample configuration file](https://github.com/finos/legend-sdlc/blob/master/legend-sdlc-server/src/test/resources/config-sample.yaml) is included to help you get started. You will need to replace some placeholders, such as Gitlab instance your SDLC server is pointing at.

From this it is not clear exactly which fields are to be changed and how. I am attaching my best guess at the changes (vj-config-sample.yaml), please let me know if this is right.

After this I started the server:

java -cp legend-sdlc-server/target/*-shaded.jar org.finos.legend.sdlc.server.LegendSDLCServer server legend-sdlc-server/src/test/resources/vj-config-sample.yaml

On the console this gives me a lot of errors:

org.reflections.ReflectionsException: could not get type for name org.zeroturnaround.javarebel.ClassEventListener
[...]
org.reflections.ReflectionsException: could not get type for name org.glassfish.jersey.internal.inject.AbstractBinder
[...]
org.reflections.ReflectionsException: could not get type for name org.codehaus.groovy.runtime.GeneratedClosure
[...]
org.reflections.ReflectionsException: could not get type for name org.opentest4j.AssertionFailedError
[...]
org.reflections.ReflectionsException: could not get type for name org.jaxen.jdom.JDOMXPath

pls around 25 other such errors. These seem a lot. Am I missing some jar files?

After this the server does seem to come up. So I went to the url http://localhost:7070/api/info and got:

{"hostName":"Mahopac-2021.local","initTime":"2022-09-18T10:56:06.440939Z","platform":{"version":null,"buildTime":null,"buildRevision":null}}

This looks a bit fishy -- why no information for version, buildTime, buildRevision ...?

The next url http://localhost:7070/api/auth/authorize (per the instructions) gives me (in the browser):

<h2 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">HTTP ERROR 500 org.pac4j.core.exception.TechnicalException: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message</h2>

URI: | /api/auth/authorize
-- | --
500
org.pac4j.core.exception.TechnicalException: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
jersey
org.pac4j.core.exception.TechnicalException: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
javax.net.ssl.SSLException: Unsupported or unrecognized SSL message

This definitely looks like an error.

Would appreciate help!

Steps to Reproduce:

Please see above.

Expected Result:

...description of what you expected to see...

Actual Result:

...what actually happened, including full exceptions (please include the entire stack trace, including "caused by" entries), log entries, screen shots etc. where appropriate...

Environment:

As in #949.

Additional Context:

legend-sdlc-log.txt

Unable to connect to remote H2 database using StaticDataSourceSpecification and DefaultH2AuhenticationStrategy

Bug Report

Steps to Reproduce:

Start a remote H2 database server (not in-memory) and populate data
Try to execute queries on this H2 server from Studio

Expected Result:

Query execution should happen successfully

Actual Result:

Query execution fails as engine always tries to connect to an in-memory H2 database

NPE in parsing relational grammar

Bug Report

NPE exception in parsing relational grammar.

Steps to Reproduce:

Parse the attached model with Legend version 2021-08-17

finos/legend-studio:0.2.56 | finos/legend-engine-server:2.39.0 | finos/legend-sdlc-server:0.45.0

Expected Result:

No exceptions

Actual Result:

legend-engine-stacktrace.txt

Environment:

finos/legend-studio:0.2.56 | finos/legend-engine-server:2.39.0 | finos/legend-sdlc-server:0.45.0

Additional Context:

Model legend-model.txt

Repo fails to compile

Support Question

On the commit hash 92f8e5991124a795fc97094dc9c68a5be57cab96, it seems that mvn install is failing (even when skipping tests):

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project legend-engine-xt-relationalStore-pure: Compilation failure: Compilation failure:
[ERROR] /Users/gberger/work/rai/git/finos/tmp/legend-engine/legend-engine-xt-relationalStore-pure/target/generated-sources/org/finos/legend/pure/generated/core_relational_relational_protocols_pure_v1_20_0_extension_extension_relational.java:[125,271] incompatible types: org.eclipse.collections.api.RichIterable<java.lang.Object> cannot be converted to org.eclipse.collections.api.RichIterable<? extends org.finos.legend.pure.generated.Root_meta_protocols_pure_v1_20_0_metamodel_TableForeignColumns>
[ERROR] /Users/gberger/work/rai/git/finos/tmp/legend-engine/legend-engine-xt-relationalStore-pure/target/generated-sources/org/finos/legend/pure/generated/core_relational_relational_protocols_pure_v1_21_0_extension_extension_relational.java:[37,271] incompatible types: org.eclipse.collections.api.RichIterable<java.lang.Object> cannot be converted to org.eclipse.collections.api.RichIterable<? extends org.finos.legend.pure.generated.Root_meta_protocols_pure_v1_21_0_metamodel_TableForeignColumns>
[ERROR] /Users/gberger/work/rai/git/finos/tmp/legend-engine/legend-engine-xt-relationalStore-pure/target/generated-sources/org/finos/legend/pure/generated/core_relational_relational_protocols_pure_v1_25_0_extension_extension_relational.java:[1541,271] incompatible types: org.eclipse.collections.api.RichIterable<java.lang.Object> cannot be converted to org.eclipse.collections.api.RichIterable<? extends org.finos.legend.pure.generated.Root_meta_protocols_pure_v1_25_0_metamodel_TableForeignColumns>
[ERROR] /Users/gberger/work/rai/git/finos/tmp/legend-engine/legend-engine-xt-relationalStore-pure/target/generated-sources/org/finos/legend/pure/generated/core_relational_relational_protocols_pure_vX_X_X_extension_extension_relational.java:[725,269] incompatible types: org.eclipse.collections.api.RichIterable<java.lang.Object> cannot be converted to org.eclipse.collections.api.RichIterable<? extends org.finos.legend.pure.generated.Root_meta_protocols_pure_vX_X_X_metamodel_TableForeignColumns>
[ERROR] /Users/gberger/work/rai/git/finos/tmp/legend-engine/legend-engine-xt-relationalStore-pure/target/generated-sources/org/finos/legend/pure/generated/core_relational_relational_protocols_pure_v1_22_0_extension_extension_relational.java:[920,271] incompatible types: org.eclipse.collections.api.RichIterable<java.lang.Object> cannot be converted to org.eclipse.collections.api.RichIterable<? extends org.finos.legend.pure.generated.Root_meta_protocols_pure_v1_22_0_metamodel_TableForeignColumns>
[ERROR] /Users/gberger/work/rai/git/finos/tmp/legend-engine/legend-engine-xt-relationalStore-pure/target/generated-sources/org/finos/legend/pure/generated/core_relational_relational_protocols_pure_v1_23_0_extension_extension_relational.java:[1033,271] incompatible types: org.eclipse.collections.api.RichIterable<java.lang.Object> cannot be converted to org.eclipse.collections.api.RichIterable<? extends org.finos.legend.pure.generated.Root_meta_protocols_pure_v1_23_0_metamodel_TableForeignColumns>
[ERROR] /Users/gberger/work/rai/git/finos/tmp/legend-engine/legend-engine-xt-relationalStore-pure/target/generated-sources/org/finos/legend/pure/generated/core_relational_relational_protocols_pure_v1_24_0_extension_extension_relational.java:[803,271] incompatible types: org.eclipse.collections.api.RichIterable<java.lang.Object> cannot be converted to org.eclipse.collections.api.RichIterable<? extends org.finos.legend.pure.generated.Root_meta_protocols_pure_v1_24_0_metamodel_TableForeignColumns>
[ERROR] /Users/gberger/work/rai/git/finos/tmp/legend-engine/legend-engine-xt-relationalStore-pure/target/generated-sources/org/finos/legend/pure/generated/core_relational_relational_protocols_pure_v1_26_0_extension_extension_relational.java:[98,271] incompatible types: org.eclipse.collections.api.RichIterable<java.lang.Object> cannot be converted to org.eclipse.collections.api.RichIterable<? extends org.finos.legend.pure.generated.Root_meta_protocols_pure_v1_26_0_metamodel_TableForeignColumns>
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :legend-engine-xt-relationalStore-pure

(module legend-engine-xt-relationalStore-pure). Any resolution steps for that?

Tried with openjdk-15 and openjdk 11.0.15

Thanks!

Commit batch ingest library for use in persistence modules

Feature Request

Description of Problem:

We are introducing a common library to handle batch ingestion logic. The initial focus is batch milestoning into relational databases. This library consolidates the implementation across multiple backend persistence implementations. It also provides an in-memory implementation that will be of use for fast testing.

Empty test data for services with multi connection runtimes causes build failures

Bug Report

Steps to Reproduce:

Define a service with a multi connection runtime

Expected Result:

Ideally, the CI build should pass without any error. Empty test data should be assigned for each connection

Actual Result:

CI build fails with json deserialisation exception

Add protocol, grammar, and execution support for persistence tests

Add service persistence protocol, grammar, and compiler

Feature Request

Description of Problem:

Allow users to configure service persistence specifications easily and use them as the source in a model-to-model transform.

Potential Solutions:

Follow standard protocol / grammar / compiler pattern laid out in the system.

CVE-2020-5245 (High) detected in dropwizard-validation-1.3.5.jar - autoclosed

CVE-2020-5245 - High Severity Vulnerability

Vulnerable Library - dropwizard-validation-1.3.5.jar

Dropwizard is a Java framework for developing ops-friendly, high-performance, RESTful web applications.

Library home page: http://www.dropwizard.io/1.3.5

Path to dependency file: legend-engine/legend-engine-server/pom.xml

Path to vulnerable library: /home/wss-scanner/.m2/repository/io/dropwizard/dropwizard-validation/1.3.5/dropwizard-validation-1.3.5.jar

Dependency Hierarchy:

dropwizard-core-1.3.5.jar (Root Library)
- ❌ dropwizard-validation-1.3.5.jar (Vulnerable Library)

Found in HEAD commit: e1ef8a7daf4f0c08b2b72e8b800166776a25d6cb

Found in base branch: master

Vulnerability Details

Dropwizard-Validation before 1.3.19, and 2.0.2 may allow arbitrary code execution on the host system, with the privileges of the Dropwizard service account, by injecting arbitrary Java Expression Language expressions when using the self-validating feature. The issue has been fixed in dropwizard-validation 1.3.19 and 2.0.2.

Publish Date: 2020-02-24

URL: CVE-2020-5245

CVSS 3 Score Details (8.8)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: Low
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: High
- Integrity Impact: High
- Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-5245

Release Date: 2020-02-24

Fix Resolution: 1.3.19,2.0.2

RFC: Integrate binding with platform

Overview

A Binding is a concept used to define support in the platform for data in some format. Data is either described using a formal schema or understood through conventions.

Internal to the platform we have the data model to describe the data structurally and instances of it to represent data values. Their external equivalents are the schema (for example a JSON Schema) and the serialized form of the data (for example a JSON document). Models can be generated from schemas and vice versa.

As a part of the development for this RFC, we plan to integrate/refactor bindings into the legend platform so that users can integrate use-cases depending on schemas/data defined in some format

Implementation plan

We plan to tackle the development of binding integration in the following phases.
The ideology behind the implementation plan is to take a depth approach and get simple use-cases working first and then add features to bring feature parity compared to existing support for Flatdata/ Json/ Xsd.

These tasks would be expanded into individual issues containing concrete details.

Phase I: Simple use-cases supported end-to-end with binding

Phase II: Feature parity compared to existing support for FlatData / Json / Xsd

Support checked result with externalize
Studio supports binding queries seamlessly (Support should be at par with queries backed by stores)
Uplift binding / externalize function to account for custom configurations

Phase III: Delete code for legacy FlatData / Json / Xsd

Fix Binding metamodel and detach it from store hierarchy
Remove External Format Connection
Deprecate legacy code & migrate existing use cases to strategic flow
Safely delete legacy code

Additional context

Binding: https://github.com/finos/legend-engine/blob/master/docs/bindings/bindings.md
Query Semantics: https://github.com/finos/legend-engine/blob/master/docs/queries/queries.md

Discrepancies in the behavior of test runners for `Mapping` and `Service`

Mapping Test Runner vs Service Test Runner

Mapping Test

Mapping tests are currently run within Studio and SDLC pipeline. We haven't examined the pipeline mapping test runner, but for Studio, we do it by matching the hash of the actual data (created by building a test runtime and execute using the execute API) vs. the expected data.

Service Test

Service tests are currently run using doTest api call method in engine with test data and an assert lambda returning a boolean. This assert function uses the equalJsonStrings function in Pure.

Discrepancies

Discrepancies arise when the same protocol definition of the service expected data and the mapping expected data produces different test results. The service test fails with the below while the mapping test passes. Notice both have been escaped to \\'DATE_COL\\' in the protocol JSON. Studio is trying to make sure for either mapping test or service test, the assertion data strings are the same. However, as stated, this same string causes different response in the test runners. QUESTION: Is this is the intended behavior, should we fix either to match the other?

Now, one problem with finos/legend-engine is related to the general strategy we handle STRING fields in our grammar parsers. In certain places, the STRING fields mean JSON strings and we handle the escaping differently (see the pair of methods fromGrammarString() and convertString()). For MappingTest, the ExpectedOutputMappingTestAssert.expectedOutput is understood to be JSON string and for service assertion lambda, this is not the case, because the string value is wrapped in a CString and would not be interpreted as a JSON string. If we use the same string for both, we would produce different grammars!

So the question is, should we do anything about this behavior? This deserves its own Github issue, but I'm starting to think we can use base64 for this.

Model Grammar

###FlatData
FlatData model::FlatDataStore
{
  section default: DelimitedWithHeadings
  {
    scope.untilEof;
    delimiter: ',';
    quoteChar: '"';
    nullString: '';

    Record
    {
      DATE_COL: DATE(format='MM/dd/yyyy', optional);
    }
  }
}


###Service
Service model::PersonService
{
  pattern: '/a4a67e91-1f4f-4226-9949-43f872b8c8a8';
  documentation: '';
  autoActivateUpdates: true;
  execution: Single
  {
    query: |model::Person.all()->graphFetchChecked(#{model::Person{DOB}}#)->serialize(#{model::Person{DOB}}#, ^meta::pure::graphFetch::execution::AlloySerializationConfig(typeKeyName='@type' , removePropertiesWithNullValues=true));
    mapping: model::PersonMapping;
    runtime:
    #{
      mappings:
      [
        model::PersonMapping
      ];
      connections:
      [
        model::FlatDataStore:
        [
          connection_1:
          #{
            FlatDataConnection
            {
              store: model::FlatDataStore;
              url: 'executor:default';
            }
          }#
        ]
      ];
    }#;
  }
  test: Single
  {
    data: 'DATE_COL\r\n ,0001-01-01';
    asserts:
    [
      { [], res: meta::pure::mapping::Result[1]|$res.values->toOne()->toString()->equalJsonStrings('{"defects":[],"source":{"defects":[{"message":"Failed to read \'DATE_COL\' with value:  ,0001-01-01, error: ParseException Unparseable date: \\" ,0001-01-01\\"","enforcementLevel":"Error","ruleType":"InvalidInput","ruleDefinerPath":"model::FlatDataStore","path":[]}],"source":{"number":1,"lineNumber":2,"record":" ,0001-01-01","recordValues":[{"address":"DATE_COL","rawValue":" ,0001-01-01"}]},"value":{"typeName":"model::FlatDataStore.default.default","values":[]}},"value":{}}') }
    ];
  }
}


###Pure
Class model::Person
{
  DOB: StrictDate[0..1];
}


###Mapping
Mapping model::PersonMapping
(
  *model::Person[accounts_classA]: FlatData
  {
    ~src model::FlatDataStore.default
    DOB: $src['DATE_COL']
  }

  MappingTests
  [
    test_1
    (
      query: |model::Person.all()->graphFetchChecked(#{model::Person{DOB}}#)->serialize(#{model::Person{DOB}}#);
      data:
      [
        <FlatData, model::FlatDataStore, 'DATE_COL\r\n ,0001-01-01'>
      ];
      assert: '{"defects":[],"source":{"defects":[{"id":null,"externalId":null,"message":"Failed to read \'DATE_COL\' with value:  ,0001-01-01, error: ParseException Unparseable date: \" ,0001-01-01\"","enforcementLevel":"Error","ruleType":"InvalidInput","ruleDefinerPath":"model::FlatDataStore","path":[]}],"source":{"number":1,"lineNumber":2,"record":" ,0001-01-01","recordValues":[{"address":"DATE_COL","rawValue":" ,0001-01-01"}]},"value":{"typeName":"model::FlatDataStore.default.default","values":[]}},"value":{"DOB":null}}';
    )
  ]
)

Service protocol JSON vs. Mapping protocol JSON

{
  "_type": "service",
  "autoActivateUpdates": true,
  "documentation": "",
  "execution": {
    "_type": "pureSingleExecution",
    "func": {
      "_type": "lambda",
      "body": [
        {
          "_type": "func",
          "function": "serialize",
          "parameters": [
            {
              "_type": "func",
              "function": "graphFetchChecked",
              "parameters": [
                {
                  "_type": "func",
                  "function": "getAll",
                  "parameters": [
                    {
                      "_type": "packageableElementPtr",
                      "fullPath": "model::Person"
                    }
                  ]
                },
                {
                  "_type": "rootGraphFetchTree",
                  "class": "model::Person",
                  "subTrees": [
                    {
                      "_type": "propertyGraphFetchTree",
                      "parameters": [],
                      "property": "DOB",
                      "subTrees": []
                    }
                  ]
                }
              ]
            },
            {
              "_type": "rootGraphFetchTree",
              "class": "model::Person",
              "subTrees": [
                {
                  "_type": "propertyGraphFetchTree",
                  "parameters": [],
                  "property": "DOB",
                  "subTrees": []
                }
              ]
            },
            {
              "_type": "func",
              "function": "new",
              "parameters": [
                {
                  "_type": "packageableElementPtr",
                  "fullPath": "meta::pure::graphFetch::execution::AlloySerializationConfig"
                },
                {
                  "_type": "string",
                  "multiplicity": {
                    "lowerBound": 1,
                    "upperBound": 1
                  },
                  "values": []
                },
                {
                  "_type": "collection",
                  "multiplicity": {
                    "lowerBound": 1,
                    "upperBound": 1
                  },
                  "values": [
                    {
                      "_type": "keyExpression",
                      "add": false,
                      "expression": {
                        "_type": "string",
                        "multiplicity": {
                          "lowerBound": 1,
                          "upperBound": 1
                        },
                        "values": ["@type"]
                      },
                      "key": {
                        "_type": "string",
                        "multiplicity": {
                          "lowerBound": 1,
                          "upperBound": 1
                        },
                        "values": ["typeKeyName"]
                      }
                    },
                    {
                      "_type": "keyExpression",
                      "add": false,
                      "expression": {
                        "_type": "boolean",
                        "multiplicity": {
                          "lowerBound": 1,
                          "upperBound": 1
                        },
                        "values": [true]
                      },
                      "key": {
                        "_type": "string",
                        "multiplicity": {
                          "lowerBound": 1,
                          "upperBound": 1
                        },
                        "values": ["removePropertiesWithNullValues"]
                      }
                    }
                  ]
                }
              ]
            }
          ]
        }
      ],
      "parameters": []
    },
    "mapping": "model::PersonMapping",
    "runtime": {
      "_type": "engineRuntime",
      "connections": [
        {
          "store": {
            "path": "model::FlatDataStore",
            "type": "STORE"
          },
          "storeConnections": [
            {
              "connection": {
                "_type": "FlatDataConnection",
                "element": "model::FlatDataStore",
                "url": "executor:default"
              },
              "id": "connection_1"
            }
          ]
        }
      ],
      "mappings": [
        {
          "path": "model::PersonMapping",
          "type": "MAPPING"
        }
      ]
    }
  },
  "name": "PersonService",
  "owners": [],
  "package": "model",
  "pattern": "/a4a67e91-1f4f-4226-9949-43f872b8c8a8",
  "test": {
    "_type": "singleExecutionTest",
    "asserts": [
      {
        "assert": {
          "_type": "lambda",
          "body": [
            {
              "_type": "func",
              "function": "equalJsonStrings",
              "parameters": [
                {
                  "_type": "func",
                  "function": "toString",
                  "parameters": [
                    {
                      "_type": "func",
                      "function": "toOne",
                      "parameters": [
                        {
                          "_type": "property",
                          "parameters": [
                            {
                              "_type": "var",
                              "name": "res"
                            }
                          ],
                          "property": "values"
                        }
                      ]
                    }
                  ]
                },
                {
                  "_type": "string",
                  "multiplicity": {
                    "lowerBound": 1,
                    "upperBound": 1
                  },
                  "values": [
                    "{\"defects\":[],\"source\":{\"defects\":[{\"message\":\"Failed to read 'DATE_COL' with value:  ,0001-01-01, error: ParseException Unparseable date: \\\" ,0001-01-01\\\"\",\"enforcementLevel\":\"Error\",\"ruleType\":\"InvalidInput\",\"ruleDefinerPath\":\"model::FlatDataStore\",\"path\":[]}],\"source\":{\"number\":1,\"lineNumber\":2,\"record\":\" ,0001-01-01\",\"recordValues\":[{\"address\":\"DATE_COL\",\"rawValue\":\" ,0001-01-01\"}]},\"value\":{\"typeName\":\"model::FlatDataStore.default.default\",\"values\":[]}},\"value\":{}}"
                  ]
                }
              ]
            }
          ],
          "parameters": [
            {
              "_type": "var",
              "class": "meta::pure::mapping::Result",
              "multiplicity": {
                "lowerBound": 1,
                "upperBound": 1
              },
              "name": "res"
            }
          ]
        }
      }
    ],
    "data": "DATE_COL\r\n ,0001-01-01"
  }
}

{
  "_type": "mapping",
  "classMappings": [
    {
      "_type": "flatData",
      "class": "model::Person",
      "flatData": "model::FlatDataStore",
      "id": "accounts_classA",
      "propertyMappings": [
        {
          "_type": "flatDataPropertyMapping",
          "property": {
            "class": "model::Person",
            "property": "DOB"
          },
          "source": "accounts_classA",
          "transform": {
            "_type": "lambda",
            "body": [
              {
                "_type": "property",
                "parameters": [
                  {
                    "_type": "var",
                    "name": "src"
                  },
                  {
                    "_type": "string",
                    "multiplicity": {
                      "lowerBound": 1,
                      "upperBound": 1
                    },
                    "values": ["DATE_COL"]
                  }
                ],
                "property": "oneString"
              }
            ],
            "parameters": []
          }
        }
      ],
      "root": true,
      "sectionName": "default"
    }
  ],
  "enumerationMappings": [],
  "includedMappings": [],
  "name": "PersonMapping",
  "package": "model",
  "tests": [
    {
      "assert": {
        "_type": "expectedOutputMappingTestAssert",
        "expectedOutput": "{\"defects\":[],\"source\":{\"defects\":[{\"id\":null,\"externalId\":null,\"message\":\"Failed to read \\'DATE_COL\\' with value:  ,0001-01-01, error: ParseException Unparseable date: \\\" ,0001-01-01\\\"\",\"enforcementLevel\":\"Error\",\"ruleType\":\"InvalidInput\",\"ruleDefinerPath\":\"model::FlatDataStore\",\"path\":[]}],\"source\":{\"number\":1,\"lineNumber\":2,\"record\":\" ,0001-01-01\",\"recordValues\":[{\"address\":\"DATE_COL\",\"rawValue\":\" ,0001-01-01\"}]},\"value\":{\"typeName\":\"model::FlatDataStore.default.default\",\"values\":[]}},\"value\":{\"DOB\":null}}"
      },
      "inputData": [
        {
          "_type": "flatData",
          "data": "DATE_COL\r\n ,0001-01-01",
          "sourceFlatData": {
            "path": "model::FlatDataStore",
            "type": "STORE"
          }
        }
      ],
      "name": "test_1",
      "query": {
        "_type": "lambda",
        "body": [
          {
            "_type": "func",
            "function": "serialize",
            "parameters": [
              {
                "_type": "func",
                "function": "graphFetchChecked",
                "parameters": [
                  {
                    "_type": "func",
                    "function": "getAll",
                    "parameters": [
                      {
                        "_type": "packageableElementPtr",
                        "fullPath": "model::Person"
                      }
                    ]
                  },
                  {
                    "_type": "rootGraphFetchTree",
                    "class": "model::Person",
                    "subTrees": [
                      {
                        "_type": "propertyGraphFetchTree",
                        "parameters": [],
                        "property": "DOB",
                        "subTrees": []
                      }
                    ]
                  }
                ]
              },
              {
                "_type": "rootGraphFetchTree",
                "class": "model::Person",
                "subTrees": [
                  {
                    "_type": "propertyGraphFetchTree",
                    "parameters": [],
                    "property": "DOB",
                    "subTrees": []
                  }
                ]
              }
            ]
          }
        ],
        "parameters": []
      }
    }
  ]
}

WS-2019-0379 (Medium) detected in commons-codec-1.12.jar - autoclosed

WS-2019-0379 - Medium Severity Vulnerability

Vulnerable Library - commons-codec-1.12.jar

The Apache Commons Codec package contains simple encoder and decoders for various formats such as Base64 and Hexadecimal. In addition to these widely used encoders and decoders, the codec package also maintains a collection of phonetic encoding utilities.

Library home page: http://commons.apache.org/proper/commons-codec/

Path to dependency file: legend-engine/legend-engine-executionPlan-generation/pom.xml

Path to vulnerable library: /home/wss-scanner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar,/home/wss-scanner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar,/home/wss-scanner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar,/home/wss-scanner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar,/home/wss-scanner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar,/home/wss-scanner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar,/home/wss-scanner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar,/home/wss-scanner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar,/home/wss-scanner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar,/home/wss-scanner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar,/home/wss-scanner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar,canner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar,/home/wss-scanner/.m2/repository/commons-codec/commons-codec/1.12/commons-codec-1.12.jar

Dependency Hierarchy:

httpclient-4.5.5.jar (Root Library)
- ❌ commons-codec-1.12.jar (Vulnerable Library)

Found in HEAD commit: e1ef8a7daf4f0c08b2b72e8b800166776a25d6cb

Found in base branch: master

Vulnerability Details

Apache commons-codec before version “commons-codec-1.13-RC1” is vulnerable to information disclosure due to Improper Input validation.

Publish Date: 2019-05-20

URL: WS-2019-0379

CVSS 3 Score Details (6.5)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: Low
- Integrity Impact: Low
- Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: apache/commons-codec@48b6157

Release Date: 2019-05-12

Fix Resolution: 1.13-RC1

Production pure client version still points to v1_22_0

I could face this client version issue from SDLC as databricks code was missing from both v1_21_0 and v1_22_0 (see active PR finos/legend-pure#507).

Now that v1_23_0 has been released on legend-pure, shouldn't we point the same in https://github.com/finos/legend-engine/blob/master/legend-engine-protocol-pure/src/main/java/org/finos/legend/engine/protocol/pure/PureClientVersions.java#L31?

Embedded mapping for model-to-model mapping

Feature Request

Description of Problem:

Assuming a model where Trade has an executingBroker attribute which is of type Party, with the Party having an identifier attribute of type Identifier, mapping that model to another one currently involves cross-referencing, which makes the mapping difficult to navigate:

*Investment::Trade: Pure
{
~src Model::Transaction
executingBroker[executingBroker]: $src,
}
*Party::Party[executingBroker]: Pure
{
~src Model::BNYM_Transaction
partyId[executingBrokerId]: $src
}
*Shared::Identifier[executingBrokerId]: Pure
{
~src Model::BNYM_Transaction
identifier: $src.executingBroker->toOne()
}

Potential Solutions:

Query mapping supports embedded mapping, which would result in expressing this mapping as follows:

*Investment::Trade[trade]: Pure
{
~src Model::Transaction
executingBroker
(
party
(
identifier: $src.executingBroker->toOne()
)
)
}

This would provide 2 benefits:

Enhanced syntax consistency across Legend
Easier way to map model-to-model and navigate the outcome

Integrate External Formats into Generation Flow

Feature Request

Related to #703
As part of incorporating binding into the platform, we want to leverage the binding schema generation route for any new schema generation.

Approach 1: Introduce Schema and Code Generation Elements

Currently FileGeneration does not differentiate between code (python, java etc) and schema (json, protobuf) generation in its model definition (there is a distinction only in the file generation api names).
Moving forward we want to make the distinction clearer and deprecate the various schemaGeneration apis tied to a specific schema type (i.e v1/schemaGeneration/jsonSchema) and move to use the one external format api (v1/external/format/generateSchema) to generate any schema.
To do this we will move to define two new packageable elements called SchemaGeneration and CodeGeneration and deprecate FileGeneration

Approach 1 Things to Consider

Avro yet to migrated to new external format extension
Does not let user define an external format as a ModelGenerationSpecficiation. Do we envision users writing defining a schemaset and wanting generate models and binding as part of the pipeline ? If so should allow this in the schema generation definition as well?

Approach 1 Implementation steps

Add SchemaGeneration and CodeGeneration elements.
Deprecate FileGeneration element
Wire up schema and code generation to sdlc pipeline flow.
Maybe have one api for code generation to be able to scale similarly to schema generation
Move morphir and rosetta to code generation implementation
Parallel Wire-up supported schema generation to use external format :Json, Protobuf, Avro.

Approach 1 Current Grammar

###FileGeneration
JsonSchema model::Person_jsonSchema
{
  scopeElements: [model::Person];
  useConstraints: false;
}


###GenerationSpecification
GenerationSpecification model::MyGenerationSpecification
{
  fileGenerations: [
    model::Person_jsonSchema
  ];
}


###Pure
Class model::Person
{
  fullName: String[1];
}

Approach 1 Potential New Grammar

###FileGeneration
SchemaGeneration model::Person_jsonSchema
{

  contentType: 'application/json';
  modelIncludes: [
    model::Person
  ];
  modelToSchemaConfiguration: [
    useConstraints: false
  ];
}

CodeGeneration model::Person_python
{

  code: Python;
  modelIncludes: [
    model::Person
  ];
  modelToCodeConfiguration: [
    myPythonConfig: false
  ];
}


###Pure
Class model::Person
{
  fullName: String[1];
}

CVE-2020-11002 (High) detected in dropwizard-validation-1.3.5.jar - autoclosed

CVE-2020-11002 - High Severity Vulnerability

Vulnerable Library - dropwizard-validation-1.3.5.jar

Dropwizard is a Java framework for developing ops-friendly, high-performance, RESTful web applications.

Library home page: http://www.dropwizard.io/1.3.5

Path to dependency file: legend-engine/legend-engine-server/pom.xml

Path to vulnerable library: /home/wss-scanner/.m2/repository/io/dropwizard/dropwizard-validation/1.3.5/dropwizard-validation-1.3.5.jar

Dependency Hierarchy:

dropwizard-core-1.3.5.jar (Root Library)
- ❌ dropwizard-validation-1.3.5.jar (Vulnerable Library)

Found in HEAD commit: e1ef8a7daf4f0c08b2b72e8b800166776a25d6cb

Found in base branch: master

Vulnerability Details

dropwizard-validation before versions 2.0.3 and 1.3.21 has a remote code execution vulnerability. A server-side template injection was identified in the self-validating feature enabling attackers to inject arbitrary Java EL expressions, leading to Remote Code Execution (RCE) vulnerability. If you are using a self-validating bean an upgrade to Dropwizard 1.3.21/2.0.3 or later is strongly recommended. The changes introduced in Dropwizard 1.3.19 and 2.0.2 for CVE-2020-5245 unfortunately did not fix the underlying issue completely. The issue has been fixed in dropwizard-validation 1.3.21 and 2.0.3 or later. We strongly recommend upgrading to one of these versions.

Publish Date: 2020-04-10

URL: CVE-2020-11002

CVSS 3 Score Details (8.8)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: Low
- Privileges Required: Low
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: High
- Integrity Impact: High
- Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-8jpx-m2wh-2v34

Release Date: 2020-04-10

Fix Resolution: io.dropwizard:dropwizard-validation:2.0.3,1.3.21

Support Aws Persistence generation

Description of Problem:

Allow users to generate Aws-specific Persistence specifications

Potential Solutions:

Follow standard protocol / grammar / compiler pattern laid out in the system.

TestTemporaryFile.testGetTemporaryPathForFile fails on MacOSX

Bug Report

TestTemporaryFile.testGetTemporaryPathForFile fails on MacOSX. See finos/legend#309

Steps to Reproduce:

mvn clean install

Expected Result:

Test TestTemporaryFile should pass instead of failing

Actual Result:

See finos/legend#309

Environment:

Mac OSX 10.11.16
JDK 11.0.2

Additional Context:

TestTemporaryFile.java has OS specific code.

        if (SystemUtils.IS_OS_LINUX || SystemUtils.IS_OS_UNIX)
        {
            Assert.assertEquals(2, (int) tempFileWithoutEndingSlash.getTemporaryPathForFile().chars().filter(ch -> ch == '/').count());
            Assert.assertEquals(2, (int) tempFileWithEndingSlash.getTemporaryPathForFile().chars().filter(ch -> ch == '/').count());
        }
        else if (SystemUtils.IS_OS_MAC || SystemUtils.IS_OS_MAC_OSX)
        {
            Assert.assertEquals(1, (int) tempFileWithoutEndingSlash.getTemporaryPathForFile().chars().filter(ch -> ch == '/').count());
            Assert.assertEquals(1, (int) tempFileWithEndingSlash.getTemporaryPathForFile().chars().filter(ch -> ch == '/').count());
        }

But SystemUtils.IS_OS_UNIX evaluates to True on a Mac OSX box

public static final boolean IS_OS_UNIX = IS_OS_AIX || IS_OS_HP_UX || IS_OS_IRIX || IS_OS_LINUX || IS_OS_MAC_OSX
            || IS_OS_SOLARIS || IS_OS_SUN_OS || IS_OS_FREE_BSD || IS_OS_OPEN_BSD || IS_OS_NET_BSD;

causing the test to execute Linux code on MacOSX.

Build failure -- too many open files

Bug Report

I tried to build legend-engine with a fresh checkout from GitHub, following these instructions.

During its test of legend-engine-server it throws an error because too many sockets are open.

Do I need to increase the limit on the number of open sockets? I checked with ulimit, I get back unlimited....?)

Note that after this mvn install -DskipTests did succeed, for whatever this is worth. I was able to launch the server, but get a number of failures to load classes, and subsequently get an error when I actually hit the server. Not sure if those problems are related to this problem (I suspect not), but will add tickets for them and link them in.

I have to cut short error messages because the text is too long for this issue otherwise. I can upload a file with the entire log, if needed.

Steps to Reproduce:

gh repo clone finos/legend-engine
cd engine
man install

Expected Result:

I expect to see a successful build.

Instead I got:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  49:09 min
[INFO] Finished at: 2022-09-18T05:21:11-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on project legend-engine-server: There are test failures.
[ERROR] 
[ERROR] Please refer to /Users/vijaysaraswat/Documents/code/legend-engine/legend-engine-test-reports/surefire-reports-aggregate for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/vijaysaraswat/Documents/code/legend-engine/legend-engine-server && /Library/Java/JavaVirtualMachines/jdk-11.0.14.jdk/Contents/Home/bin/java '-javaagent:/Users/vijaysaraswat/.m2/repository/org/jacoco/org.jacoco.agent/0.8.5/org.jacoco.agent-0.8.5-runtime.jar=destfile=/Users/vijaysaraswat/Documents/code/legend-engine/legend-engine-server/target/jacoco.exec,excludes=org/finos/legend/engine/language/pure/grammar/from/antlr4/**/*' org.apache.maven.surefire.booter.ForkedBooter /Users/vijaysaraswat/Documents/code/legend-engine/legend-engine-server/target/surefire 2022-09-18T04-32-08_199-jvmRun1 surefire16215282550736567441tmp surefire_7712536291907575916478tmp
[ERROR] Process Exit Code: 0
[ERROR] Crashed tests:
[ERROR] org.finos.legend.engine.server.test.shared.PureTestHelper$PureTestCase
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/vijaysaraswat/Documents/code/legend-engine/legend-engine-server && /Library/Java/JavaVirtualMachines/jdk-11.0.14.jdk/Contents/Home/bin/java '-javaagent:/Users/vijaysaraswat/.m2/repository/org/jacoco/org.jacoco.agent/0.8.5/org.jacoco.agent-0.8.5-runtime.jar=destfile=/Users/vijaysaraswat/Documents/code/legend-engine/legend-engine-server/target/jacoco.exec,excludes=org/finos/legend/engine/language/pure/grammar/from/antlr4/**/*' org.apache.maven.surefire.booter.ForkedBooter /Users/vijaysaraswat/Documents/code/legend-engine/legend-engine-server/target/surefire 2022-09-18T04-32-08_199-jvmRun1 surefire16215282550736567441tmp surefire_7712536291907575916478tmp
[ERROR] Process Exit Code: 0
[ERROR] Crashed tests:
[ERROR] org.finos.legend.engine.server.test.shared.PureTestHelper$PureTestCase
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:282)
[ERROR] 	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:245)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
[ERROR] 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
[ERROR] 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
[ERROR] 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR] 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)

Actual Result:

Over 50K lines are printed out during the mvn install. Here is the relevant portion:

STOP
[ERROR] Tests run: 55, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 79.103 s <<< FAILURE! - in org.finos.legend.engine.server.test.pureClient.executionPlan.Test_ExecutionPlan_JavaPlatform_UsingPureClient
[ERROR] toString(org.finos.legend.engine.server.test.shared.PureTestHelper$PureTestCase)  Time elapsed: 0.028 s  <<< ERROR!
Execution error at (resource:/core/pure/protocol/vX_X_X/invocations/execution.pure lines:99c73-103c1), "Unexpected error executing function with params [Anonymous_NoCounter, 127.0.0.1, 49467, null]"
	at org.finos.legend.pure.runtime.java.compiled.metadata.JavaMethodWithParamsSharedPureFunction.execute(JavaMethodWithParamsSharedPureFunction.java:83)
[run_log.txt](https://github.com/finos/legend-engine/files/9593975/run_log.txt)

	at org.finos.legend.pure.runtime.java.compiled.generation.processors.support.Pure.evaluate(Pure.java:480)
	at org.finos.legend.pure.generated.CoreGen.evaluate(CoreGen.java:60)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_testUtils$13.value(core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.java:413)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_testUtils$13.execute(core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.java:406)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_testUtils$13.execute(core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.java:402)
	at org.finos.legend.pure.runtime.java.compiled.generation.processors.support.Pure.evaluate(Pure.java:460)
	at org.finos.legend.pure.runtime.java.compiled.generation.processors.support.Pure.legendTest(Pure.java:208)
	at org.finos.legend.pure.generated.CoreGen.legendTest(CoreGen.java:45)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.Root_meta_pure_executionPlan_engine_java_tests_executeTests_Project_1__Extension_MANY__DebugContext_1__Boolean_1_(core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.java:398)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.Root_meta_pure_executionPlan_engine_java_tests_runTests_JavaExpressionTest_MANY__Extension_MANY__DebugContext_1__Boolean_1_(core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.java:298)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.Root_meta_pure_executionPlan_engine_java_tests_runTests_JavaExpressionTestSet_1__Boolean_1_(core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.java:176)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_stringLibraryTests.Root_meta_pure_executionPlan_engine_java_tests_toString__Boolean_1_(core_pure_executionPlan_javaPlatform_planConventions_test_stringLibraryTests.java:1417)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.finos.legend.engine.server.test.shared.PureTestHelper$PureTestCase.runTest(PureTestHelper.java:414)
	at junit.framework.TestCase.runBare(TestCase.java:142)
	at junit.framework.TestResult$1.protect(TestResult.java:122)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.framework.TestResult.run(TestResult.java:125)
	at junit.framework.TestCase.run(TestCase.java:130)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
	at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.extensions.TestSetup.run(TestSetup.java:27)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:90)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.RuntimeException: Error: service call for URL: 'http://127.0.0.1:49467/api/pure/v1/execution/executePlan' failed: Too many open files
	at org.finos.legend.pure.runtime.java.shared.http.HttpRawHelper.executeRequest(HttpRawHelper.java:122)
	at org.finos.legend.pure.runtime.java.shared.http.HttpRawHelper.executeHttpService(HttpRawHelper.java:68)
	at org.finos.legend.pure.runtime.java.compiled.generation.processors.support.CompiledSupport.executeHttpRaw(CompiledSupport.java:3242)
	at org.finos.legend.pure.generated.core_pure_protocol_vX_X_X_invocations_execution.Root_meta_protocols_pure_vX_X_X_invocation_execution_execute_executePlan_String_1__String_1__Integer_1__String_1_(core_pure_protocol_vX_X_X_invocations_execution.java:750)
	at org.finos.legend.pure.generated.core_pure_protocol_vX_X_X_invocations_execution.Root_meta_protocols_pure_vX_X_X_invocation_execution_execute_executePlan_ExecutionPlan_1__String_1__Integer_1__Extension_MANY__String_1_(core_pure_protocol_vX_X_X_invocations_execution.java:745)
	at jdk.internal.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.finos.legend.pure.runtime.java.compiled.metadata.JavaMethodWithParamsSharedPureFunction.execute(JavaMethodWithParamsSharedPureFunction.java:52)
	... 41 more
Caused by: java.net.SocketException: Too many open files
	at java.base/java.net.Socket.createImpl(Socket.java:479)
	at java.base/java.net.Socket.getImpl(Socket.java:539)
	at java.base/java.net.Socket.setSoTimeout(Socket.java:1158)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:120)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at org.finos.legend.pure.runtime.java.shared.http.HttpRawHelper.executeRequest(HttpRawHelper.java:113)
	... 49 more

[ERROR] unitPureTests(org.finos.legend.engine.server.test.shared.PureTestHelper$PureTestCase)  Time elapsed: 0.164 s  <<< ERROR!
Execution error at (resource:/core/pure/protocol/vX_X_X/invocations/execution.pure lines:99c73-103c1), "Unexpected error executing function with params [Anonymous_NoCounter, 127.0.0.1, 49467, null]"
	at org.finos.legend.pure.runtime.java.compiled.metadata.JavaMethodWithParamsSharedPureFunction.execute(JavaMethodWithParamsSharedPureFunction.java:83)
	at org.finos.legend.pure.runtime.java.compiled.generation.processors.support.Pure.evaluate(Pure.java:480)
	at org.finos.legend.pure.generated.CoreGen.evaluate(CoreGen.java:60)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_testUtils$13.value(core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.java:413)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_testUtils$13.execute(core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.java:406)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_testUtils$13.execute(core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.java:402)
	at org.finos.legend.pure.runtime.java.compiled.generation.processors.support.Pure.evaluate(Pure.java:460)
	at org.finos.legend.pure.runtime.java.compiled.generation.processors.support.Pure.legendTest(Pure.java:208)
	at org.finos.legend.pure.generated.CoreGen.legendTest(CoreGen.java:45)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.Root_meta_pure_executionPlan_engine_java_tests_executeTests_Project_1__Extension_MANY__DebugContext_1__Boolean_1_(core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.java:398)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.Root_meta_pure_executionPlan_engine_java_tests_runTests_JavaPureTestSet_1__Extension_MANY__DebugContext_1__Boolean_1_(core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.java:385)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.Root_meta_pure_executionPlan_engine_java_tests_runTests_JavaPureTestSet_1__Boolean_1_(core_pure_executionPlan_javaPlatform_planConventions_test_testUtils.java:327)
	at org.finos.legend.pure.generated.core_pure_executionPlan_javaPlatform_planConventions_test_unitLibraryTests.Root_meta_pure_executionPlan_engine_java_tests_unitPureTests__Boolean_1_(core_pure_executionPlan_javaPlatform_planConventions_test_unitLibraryTests.java:29)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.finos.legend.engine.server.test.shared.PureTestHelper$PureTestCase.runTest(PureTestHelper.java:414)
	at junit.framework.TestCase.runBare(TestCase.java:142)
	at junit.framework.TestResult$1.protect(TestResult.java:122)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.framework.TestResult.run(TestResult.java:125)
	at junit.framework.TestCase.run(TestCase.java:130)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
	at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.extensions.TestSetup.run(TestSetup.java:27)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:90)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.RuntimeException: Error: service call for URL: 'http://127.0.0.1:49467/api/pure/v1/execution/executePlan' failed: Too many open files
	at org.finos.legend.pure.runtime.java.shared.http.HttpRawHelper.executeRequest(HttpRawHelper.java:122)
	at org.finos.legend.pure.runtime.java.shared.http.HttpRawHelper.executeHttpService(HttpRawHelper.java:68)
	at org.finos.legend.pure.runtime.java.compiled.generation.processors.support.CompiledSupport.executeHttpRaw(CompiledSupport.java:3242)
	at org.finos.legend.pure.generated.core_pure_protocol_vX_X_X_invocations_execution.Root_meta_protocols_pure_vX_X_X_invocation_execution_execute_executePlan_String_1__String_1__Integer_1__String_1_(core_pure_protocol_vX_X_X_invocations_execution.java:750)
	at org.finos.legend.pure.generated.core_pure_protocol_vX_X_X_invocations_execution.Root_meta_protocols_pure_vX_X_X_invocation_execution_execute_executePlan_ExecutionPlan_1__String_1__Integer_1__Extension_MANY__String_1_(core_pure_protocol_vX_X_X_invocations_execution.java:745)
	at jdk.internal.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.finos.legend.pure.runtime.java.compiled.metadata.JavaMethodWithParamsSharedPureFunction.execute(JavaMethodWithParamsSharedPureFunction.java:52)
	... 41 more
Caused by: java.net.SocketException: Too many open files
	at java.base/java.net.Socket.createImpl(Socket.java:479)
	at java.base/java.net.Socket.getImpl(Socket.java:539)
	at java.base/java.net.Socket.setSoTimeout(Socket.java:1158)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:120)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)

Environment:

Building on a MacBook Pro running macOS Monterey 12.5.1, 64 GB memory, 8-core intel i9.

Java:

java version "11.0.14" 2022-01-18 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.14+8-LTS-263)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.14+8-LTS-263, mixed mode)

maven:

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /Users/vijaysaraswat/Downloads/apache-maven-3.6.3
Java version: 11.0.14, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk-11.0.14.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "12.5.1", arch: "x86_64", family: "mac"

Identity, and Credential to be Serializable

Feature Request

Description of Problem:

Service execution jars leverage Identity (and Credential) as part of authentication and authorization flows. There could be places where we can benefit from this classes been Serializable. For example, Spark integration, where instances are distributed to Spark workers.

Potential Solutions:

Ensure the Identity classes are Java Serializable.

CVE-2018-10237 (Medium) detected in guava-24.0-jre.jar - autoclosed

CVE-2018-10237 - Medium Severity Vulnerability

Vulnerable Library - guava-24.0-jre.jar

Guava is a suite of core and expanded libraries that include utility classes, google's collections, io classes, and much much more.

Library home page: https://github.com/google/guava

Path to dependency file: legend-engine/legend-engine-executionPlan-execution/pom.xml

Path to vulnerable library: canner/.m2/repository/com/google/guava/guava/24.0-jre/guava-24.0-jre.jar,/home/wss-scanner/.m2/repository/com/google/guava/guava/24.0-jre/guava-24.0-jre.jar,/home/wss-scanner/.m2/repository/com/google/guava/guava/24.0-jre/guava-24.0-jre.jar,/home/wss-scanner/.m2/repository/com/google/guava/guava/24.0-jre/guava-24.0-jre.jar,/home/wss-scanner/.m2/repository/com/google/guava/guava/24.0-jre/guava-24.0-jre.jar

Dependency Hierarchy:

❌ guava-24.0-jre.jar (Vulnerable Library)

Found in HEAD commit: e1ef8a7daf4f0c08b2b72e8b800166776a25d6cb

Found in base branch: master

Vulnerability Details

Unbounded memory allocation in Google Guava 11.0 through 24.x before 24.1.1 allows remote attackers to conduct denial of service attacks against servers that depend on this library and deserialize attacker-provided data, because the AtomicDoubleArray class (when serialized with Java serialization) and the CompoundOrdering class (when serialized with GWT serialization) perform eager allocation without appropriate checks on what a client has sent and whether the data size is reasonable.

Publish Date: 2018-04-26

URL: CVE-2018-10237

CVSS 3 Score Details (5.9)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: High
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: None
- Integrity Impact: None
- Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nvd.nist.gov/vuln/detail/CVE-2018-10237

Release Date: 2018-04-26

Fix Resolution: 24.1.1-jre, 24.1.1-android

Check this box to open an automated fix PR

Image won't run by itself

Hi folks,
I am working on a Charmed Operator for Legend Server and I am trying to understand how the different parts of Legend interact during deployment.

Should I be able to simply pull and run the Engine image directly from dockerhub?

It seems like it's trying to access a volume that is not yet mounted. I know these images were probably design to work with Docker Compose mainly, but am I missing something obvious here (that is probably covered by the Maven install guide)?

Thanks :-)

$ docker pull finos/legend-engine-server:2.35.1
$ docker run --rm -it finos/legend-engine-server:2.35.1
15:26:30.336 [main] INFO Alloy Execution Server - {"duration":0.0,"eventType":"URL_FACTORY_REGISTERED","message":"Registered Alloy URL Factory","timeStamp":"2021-07-02 15:26:30.037","user":"_UNKNOWN_"}
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
java.io.FileNotFoundException: File /config/config.json not found
        at io.dropwizard.configuration.FileConfigurationSourceProvider.open(FileConfigurationSourceProvider.java:18)
        at io.dropwizard.configuration.BaseConfigurationFactory.build(BaseConfigurationFactory.java:80)
        at io.dropwizard.cli.ConfiguredCommand.parseConfiguration(ConfiguredCommand.java:126)
        at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:74)
        at io.dropwizard.cli.Cli.run(Cli.java:78)
        at io.dropwizard.Application.run(Application.java:93)
        at org.finos.legend.engine.server.Server.main(Server.java:91)

CVE-2020-15250 (Medium) detected in junit-4.13.jar - autoclosed

CVE-2020-15250 - Medium Severity Vulnerability

Vulnerable Library - junit-4.13.jar

JUnit is a unit testing framework for Java, created by Erich Gamma and Kent Beck.

Library home page: http://junit.org

Path to dependency file: legend-engine/legend-engine-language-pure-modelManager-sdlc/pom.xml

Path to vulnerable library: /home/wss-scanner/.m2/repository/junit/junit/4.13/junit-4.13.jar

Dependency Hierarchy:

legend-engine-language-pure-modelManager-0.0.1-SNAPSHOT.jar (Root Library)
- legend-engine-language-pure-compiler-0.0.1-SNAPSHOT.jar
  - legend-pure-runtime-java-engine-compiled-0.0.1-SNAPSHOT.jar
    - ❌ junit-4.13.jar (Vulnerable Library)

Found in HEAD commit: 34892ea74f0a3aa1a997772f184d05e1816d5102

Found in base branch: master

Vulnerability Details

In JUnit4 before version 4.13.1, the test rule TemporaryFolder contains a local information disclosure vulnerability. On Unix like systems, the system's temporary directory is shared between all users on that system. Because of this, when files and directories are written into this directory they are, by default, readable by other users on that same system. This vulnerability does not allow other users to overwrite the contents of these directories or files. This is purely an information disclosure vulnerability. This vulnerability impacts you if the JUnit tests write sensitive information, like API keys or passwords, into the temporary folder, and the JUnit tests execute in an environment where the OS has other untrusted users. Because certain JDK file system APIs were only added in JDK 1.7, this this fix is dependent upon the version of the JDK you are using. For Java 1.7 and higher users: this vulnerability is fixed in 4.13.1. For Java 1.6 and lower users: no patch is available, you must use the workaround below. If you are unable to patch, or are stuck running on Java 1.6, specifying the java.io.tmpdir system environment variable to a directory that is exclusively owned by the executing user will fix this vulnerability. For more information, including an example of vulnerable code, see the referenced GitHub Security Advisory.

Publish Date: 2020-07-21

URL: CVE-2020-15250

CVSS 3 Score Details (4.4)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Local
- Attack Complexity: High
- Privileges Required: Low
- User Interaction: Required
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: High
- Integrity Impact: None
- Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-269g-pwp5-87pp

Release Date: 2020-07-21

Fix Resolution: junit:junit:r4.13.1

Incorrect type inference

With two parameter lambdas, the return type is assumed to the same as that of the first parameter. This is valid for functions like sort and removeDuplicates, but not so for fold, where the return type is the type of the second parameter. The type inferencer is therefore attributing the wrong type, causing downstream problems in code generation. More details can be seen in the test case org.finos.legend.engine.query.pure.api.test.inMemory.TestM2MGrammarModel()

Support persistence trigger extensions

Feature Request

Description of Problem:

Allow other modules to extend persistence specifications with custom trigger definitions, e.g. for triggers specific to different persistence platforms OR for non open-sourced trigger mechanisms.

Potential Solutions:

Use standard pattern for protocol / grammar / compiler extensions with island syntax.

Vault is not available to library mode executions

Bug Report

The vault is not available to library mode executions.

This is because the default properties file vault implementation is only registered in the Dropwizard server code path [1].
[1]

legend-engine/legend-engine-server/src/main/java/org/finos/legend/engine/server/Server.java

Line 175 in 0856bb0

private void loadVaults(List<VaultConfiguration> vaultConfigurations)

Think the vault should be loaded in the PlanExecutor ?

Switch the default config file to use the GitlabClient instead of the KerberosClient

Feature Request

Switch the default config file to use the GitlabClient instead of the KerberosClient.

Description of Problem:

The userTestConfig.json[1] that we refer to in our install guides, uses the LocalKerberosClient. This is not useful to users on personal devices who do not have a local Kerberos installation.

[1] https://github.com/finos/legend-engine/blob/master/legend-engine-server/src/test/resources/org/finos/legend/engine/server/test/userTestConfig.json

Potential Solutions:

Include a config file that uses the GitlabClient instead of the KerberosClient.

Also, update the installer guides to use the new config file.

CVE-2020-9488 (Low) detected in log4j-1.2.17.jar - autoclosed

CVE-2020-9488 - Low Severity Vulnerability

Vulnerable Library - log4j-1.2.17.jar

Apache Log4j 1.2

Path to dependency file: legend-engine/legend-engine-server/pom.xml

Path to vulnerable library: canner/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar

Dependency Hierarchy:

❌ log4j-1.2.17.jar (Vulnerable Library)

Found in HEAD commit: e1ef8a7daf4f0c08b2b72e8b800166776a25d6cb

Found in base branch: master

Vulnerability Details

Improper validation of certificate with host mismatch in Apache Log4j SMTP appender. This could allow an SMTPS connection to be intercepted by a man-in-the-middle attack which could leak any log messages sent through that appender.

Publish Date: 2020-04-27

URL: CVE-2020-9488

CVSS 3 Score Details (3.7)

Base Score Metrics:

Exploitability Metrics:
- Attack Vector: Network
- Attack Complexity: High
- Privileges Required: None
- User Interaction: None
- Scope: Unchanged
Impact Metrics:
- Confidentiality Impact: Low
- Integrity Impact: None
- Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://issues.apache.org/jira/browse/LOG4J2-2819

Release Date: 2020-04-27

Fix Resolution: org.apache.logging.log4j:log4j-core:2.13.2

Check this box to open an automated fix PR

Wrong class leverage to validate partition fields

During multi flat target part validation, the partition fields are validated against the top level model class rather than against the flat part class (thru the modelProperty)

legend-engine/legend-engine-language-pure-dsl-persistence/src/main/java/org/finos/legend/engine/language/pure/dsl/persistence/compiler/toPureGraph/HelperPersistenceBuilder.java

Lines 387 to 398 in dfa4e98

    
           private Root_meta_pure_persistence_metamodel_persister_targetshape_MultiFlatTargetPart resolvePart(MultiFlatTargetPart part, Class<?> modelClass, CompileContext context) 
        
           { 
        
               Property<?, ?> property = validateAndResolveProperty(modelClass, part.modelProperty, part.sourceInformation, context); 
        
               Type targetType = property._genericType()._rawType(); 
        
               Assert.assertTrue(targetType instanceof Class, () -> String.format("Target shape modelProperty '%s' must refer to a class.", part.modelProperty), part.sourceInformation, EngineErrorType.COMPILATION); 
        
               return new Root_meta_pure_persistence_metamodel_persister_targetshape_MultiFlatTargetPart_Impl("") 
        
                       ._modelProperty(property) 
        
                       ._targetName(part.targetName) 
        
                       ._partitionFields(ListIterate.collect(part.partitionFields, p -> validateAndResolvePropertyName(modelClass, p, part.sourceInformation, context))) 
        
                       ._deduplicationStrategy(buildDeduplicationStrategy(part.deduplicationStrategy, modelClass, context)); 
        
           }

The class from targetType should have been pass instead of modelClass.

._partitionFields(ListIterate.collect(part.partitionFields, p -> validateAndResolvePropertyName((Class) targetType, p, part.sourceInformation, context)))

Update Eclipse Collections version in pom.xml

Update Eclipse Collections version from 10.2.0 -> 11.0.0. This will allow developers to make use of new features in EC.

Java Compilation fails for this function

Bug Report

Steps to Reproduce:

I have the following PURE implementation of a function:

function meta::nothing::inspect<T|m>(result: Result<T|m>[1], fn: FunctionDefinition<{T[m]->Any[*]}>[1]): Result<T|m>[1] {
  if($result.errors->isEmpty(), 
    {| $fn->eval($result.values->evaluateAndDeactivate()); $result;}, 
    | $result
  );
}

The class Result looks as follows:

Class meta::nothing::Result<T|n> {
  <<equality.Key>> values: T[n];
  <<equality.Key>> errors: Any[*];
}

The compiler generates the following Java Code for this function:

public static <T> org.finos.legend.pure.generated.Root_meta_nothing_Result<? extends T> Root_meta_nothing_inspect_Result_1__FunctionDefinition_1__Result_1_(final org.finos.legend.pure.generated.Root_meta_nothing_Result<? extends T> _result, final org.finos.legend.pure.m3.coreinstance.meta.pure.metamodel.function.FunctionDefinition<? extends java.lang.Object> _fn,final ExecutionSupport es)
{
    return ((org.finos.legend.pure.generated.Root_meta_nothing_Result<? extends T>)
            (CompiledSupport.isEmpty(_result._errors()) ?
                    new LambdaZero<org.finos.legend.pure.generated.Root_meta_nothing_Result<? extends T>>() {
                        public org.finos.legend.pure.generated.Root_meta_nothing_Result<? extends T> execute() {
                            ((RichIterable<? extends java.lang.Object>)(Object)CompiledSupport.toPureCollection(CoreGen.evaluate(es, (org.finos.legend.pure.m3.coreinstance.meta.pure.metamodel.function.Function)_fn, _result._values())));
                            return _result;
                        }
            }.execute():_result)
    );
}

However, the compiler complains that the inner statement

((RichIterable<? extends java.lang.Object>)(Object)CompiledSupport.toPureCollection(CoreGen.evaluate(es, (org.finos.legend.pure.m3.coreinstance.meta.pure.metamodel.function.Function)_fn, _result._values())));

"is not a statement".

Environment:

Tested using legend-engine-3.11.1-SNAPSHOT and openjdk 11.0.15 2022-04-19

Add endpoints for `format` type of `SchemaSet` and `contentType` of `Binding`

Feature Request

Add endpoints for format type of SchemaSet and contentType of Binding

Description of Problem:

Right now we are adding the new format types of SchemaSet and content types of Binding manually on studio. It is better to have an endpoint in the backend to fetch the data.

Compiler fails to find view filter in included store

Bug Report

Compiler fails to find view filter in included store

Steps to Reproduce:

Compile below grammar and compiler error will show

Model To Reproduce

###Relational
Database store::relational::tests::dbIncWithStoreFilterMain
(

  Table personTable
  (
    ID INTEGER PRIMARY KEY,
    FIRSTNAME VARCHAR(200),
    LASTNAME VARCHAR(200),
    AGE INTEGER,
    ADDRESSID INTEGER,
    FIRMID INTEGER,
    MANAGERID INTEGER
  )

)

Database store::relational::tests::dbIncWithStoreFilter
(
  include store::relational::tests::dbIncWithStoreFilterMain

  View PersonView
  (
    ~filter PersonFilter
    ID: personTable.ID PRIMARY KEY,
    FIRSTNAME: personTable.FIRSTNAME,
    LASTNAME: personTable.LASTNAME,
    AGE: personTable.AGE
  )
)

Expected Result:

Compiler should look for filter in included store and return success for above grammar.

Actual Result:

Compiler returns error: Compilation failed: Can't find filter 'PersonFilter' in database 'dbIncWithStoreFilter'

	private Root_meta_pure_persistence_metamodel_persister_targetshape_MultiFlatTargetPart resolvePart(MultiFlatTargetPart part, Class<?> modelClass, CompileContext context)
	{
	Property<?, ?> property = validateAndResolveProperty(modelClass, part.modelProperty, part.sourceInformation, context);
	Type targetType = property._genericType()._rawType();
	Assert.assertTrue(targetType instanceof Class, () -> String.format("Target shape modelProperty '%s' must refer to a class.", part.modelProperty), part.sourceInformation, EngineErrorType.COMPILATION);

	return new Root_meta_pure_persistence_metamodel_persister_targetshape_MultiFlatTargetPart_Impl("")
	._modelProperty(property)
	._targetName(part.targetName)
	._partitionFields(ListIterate.collect(part.partitionFields, p -> validateAndResolvePropertyName(modelClass, p, part.sourceInformation, context)))
	._deduplicationStrategy(buildDeduplicationStrategy(part.deduplicationStrategy, modelClass, context));
	}