finos / legend-studio

Legend Studio

legend-studio

The codebase and home of Legend applications: Legend Studio, Legend Query, etc.

Getting started

Studio relies Legend SDLC server and Legend Engine server. To quickly set these backend servers up, use our development Docker compose. If you need to debug and develop the backend, setup with Maven instead.

Last but not least, make sure you have Yarn installed. Run the following commands in order.

  yarn install
  yarn setup
  yarn dev

After setting up, visit http://localhost:9000/studio and the application should be up 🎉

If you get Unauthorized error, visit SDLC server at http://localhost:6100/api/auth/authorize in your browser, you will get redirected to the Gitlab login page or a Gitlab page asking you to authorize Legend OAuth application. After you completing these steps, you will be redirected back to SDLC. Now refresh Studio and the problem should be gone.

Documentation

You can find the usage documentation on the website.

Check out the Getting Started page for a quick overview. Also check out the Installation Guide to find out how to setup your Legend ecosystem.

You can improve it by sending pull requests to this repository.

For Legend Studio extension contributors, check out our API documentation.

Roadmap

Visit our roadmap to know more about the upcoming features.

Contributing

Please read our contributing guide.

Code of conduct

We adopted a Code of Conduct that we expect project participants to adhere to. Please read the full text so that you can understand what actions will and will not be tolerated.

Good first issues

We have a list of good first issues that contain bugs which have a relatively limited scope. This is a great place to get started, gain experience, and get familiar with our contribution process.

License

Copyright (c) 2020-present, Goldman Sachs

Distributed under the Apache License, Version 2.0.

SPDX-License-Identifier: Apache-2.0

legend-studio's Issues

CVE-2015-9251 (Medium) detected in jquery-2.1.4.min.js - autoclosed

CVE-2015-9251 - Medium Severity Vulnerability

Vulnerable Library - jquery-2.1.4.min.js

JavaScript library for DOM operations

Library home page: https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js

Path to dependency file: legend-studio/node_modules/deep-diff/test/tests.html

Path to vulnerable library: legend-studio/node_modules/deep-diff/test/tests.html

Dependency Hierarchy:

• ❌ jquery-2.1.4.min.js (Vulnerable Library)

Found in HEAD commit: 1de7f5505e98c70f04393f70b9fc52afccdfa463

Found in base branch: master

Vulnerability Details

jQuery before 3.0.0 is vulnerable to Cross-site Scripting (XSS) attacks when a cross-domain Ajax request is performed without the dataType option, causing text/javascript responses to be executed.

Publish Date: 2018-01-18

URL: CVE-2015-9251

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nvd.nist.gov/vuln/detail/CVE-2015-9251

Release Date: 2018-01-18

Fix Resolution: jQuery - v3.0.0

Bug: Uncompressed mapping and service test data can throw off generated grammar

🐛 Bug Report

If mapping and service test data or expected results are stored with newlines, the generated grammar shown when entering hacker-mode also shows these new lines as new lines and thus, causing parsing error.

Expected behavior

We would expect this to behave the same way that text element content are handled. We can type newlines in the text content, but upon entering text mode, the newlines are escaped properly. We need to investigate what is the difference between these 2 handlings.

Workaround

Right now to make this non-breaking until we can figure out the problem in the backend, we will minify the string data when it's saved, hash-computed and beautify when we display/read them

We will mark the area where we do this with marker /* @MARKER: Workaround for https://github.com/finos/legend-studio/issues/68 */ so we can come back to remove them later when we fix the backend. The ideal state we want to end up with is not to tamper with user input, we will leave it as they type it in.

---

@aziemchawdhary-gs @MauricioUyaguari let's find some time to look at the grammar handling in engine

---

Found a bug that might be related to this - #135

User ID data is missing error upon installation

Hi,

I have gone through the steps to install and run legend-studio on my computer (windows) by following the readme (npm install, npm run setup, npm start). Upon running npm start, my browser opens https://localhost:3000/studio, however I get an error on the screen that reads "Assertion Error: User ID data is missing".

Is there a file that I should use to configure a default user ID or should I have installed a separate legend component (I only installed studio)?

Feature request: Raw value specification protocol path resolution

Right now paths in lambda which shortened by import grammar are not properly processed by Studio since we swallow the import/SectionIndex.

Context and Motivation

When user uses import in text mode, they can refer to elements by their name instead of the full-qualified path (FQP), we store this info in the unique element SectionIndex which is currently not pushed to SDLC for storage. As such, users can use import statement but these will get swallowed up in Studio. In turns, we resolve the path to FQP when we can to avoid problems when the SectionIndex is nolonger available. However, we haven't done this for raw value specifications.

Ideally, if we can come up with a strategy in SDLC to store SectionIndex it would be great, but we are still in the process of discussing that: since we don't want the user to be able to directly change modify this entity, so we want to find a special path for it. So doing this in Studio really is trying to be nice until SDLC properly supports this.

Feature request: Service editor improvements

This is a compiled list of improvements for service editor, which is still in beta stage.

• Analytics Service Editor: warning for the mapping -> does not have class mapping for the target (needs some analytics here)
• Handle the switch between single/multiple executions - #943
• Support explorer for multi execution: support addition/deletion - #943
• Allow better handling for setting parameters for test - #942
• Allow better support for parameters coming from URL - #942
• Upon creation, add current user as the owner of service (if initial getCurrentUser fails, do not retry or warn. Take it as an empty array of owners) - #944
• MAYBE Do validation on the number of owners (at least 1) - #944
• Restyle service execution editor to have a better visual distinction between panels
• DISCUSS We should probably redesign this a little bit (it has a few panel being separated by thin border lines right now, perhaps, it's better to follow what we do in #659 with query builder)

Feature request: Execution plan viewer

Context and Motivation

The execution plan holds all the information required for the engine server (legend-engine) to execute a query. With this viewer, we plan to shed some light on the context of the execution plan and showcase to the user the steps taken to execute their queries to provide the data in the shape they want it.

Since the execution plan holds various types of nodes for different types of queries and data sources, it is not feasible to inaugurate a user interface covering all use cases. For this reason, this feature will be implemented in steps. We will cover basic relational, model-to-model cases in the beginning and lead up to more advanced use cases. For the nodes not supported we will default to a basic JSON string viewer.

Implementation

As mentioned above we will tackle the viewer through phases. When creating the viewer for each phase, the steps will be similar to adding support for a new packageable element except there is no need for transformation as the execution plan viewer is a readonly UI for now. Therefore to add support please follow the below steps.

• Add V1_ExecutionPlan models (#248)
• Add V1 serializer/de-serializer support to convert from/to raw JSON and V1_ExecutionPlan Models (#248)
• Add metamodel ExecutionPlan models (#248)
• Add Execution Plan builder converting V1_ExecutionPlan protocol to ExecutionPlan metamodel (and potentially vice versa so we can do more testing) (#248)
• Use the metamodel ExecutionPlan to create a viewer. Using the metamodel execution plan will provide you with the compiled metamodel graph to be able to quickly find referenced elements in the graph such as classes, mappings, and runtimes.
• MVP execution plan viewer interface (See #289)
• In plan viewer, classify the nodes in different buckets, and render them differently:
  • Flow control (sequence/conditional/loop/...)
  • Templating node (free-marker)

Phase 1 - Relational Mapping - Simple Projection with H2 Database

Execution Plan

See Attached Execution Plan Json
ExecutionPlan_Phase_1.txt

Pure Grammar

###Relational
Database model::Test
(
 Table FirmTable
 (
   id INTEGER PRIMARY KEY,
   Legal_name VARCHAR(200)
 )
 Table PersonTable
 (
   id INTEGER PRIMARY KEY,
   firm_id INTEGER,
   firstName VARCHAR(200),
   lastName VARCHAR(200)
 )

 Join FirmPerson(PersonTable.firm_id = FirmTable.id)
)


###Pure
Profile model::MyExtension
{
 stereotypes: [important];
 tags: [doc];
}

Enum model::IncType
{
 Corp,
 LLC
}

Class <<model::MyExtension.important>> {model::MyExtension.doc = 'This is a model of a firm'} model::Firm extends model::LegalEntity
[
 validName: $this.legalName->startsWith('_')
]
{
 employees: model::Person[1..*];
 incType: model::IncType[1];
 employeeSize() {$this.employees->count()}: Number[1];
}

Class model::LegalEntity
{
 legalName: String[1];
}

Class model::Person
{
 firstName: String[1];
 lastName: String[1];
}


###Mapping
Mapping model::NewMapping
(
 *model::Firm: Relational
 {
   ~primaryKey
   (
     [model::Test]FirmTable.id
   )
   ~mainTable [model::Test]FirmTable
   legalName: concat([model::Test]FirmTable.Legal_name, '_LTD'),
   employees[model_Person]: [model::Test]@FirmPerson
 }
 *model::Person: Relational
 {
   ~primaryKey
   (
     [model::Test]PersonTable.id
   )
   ~mainTable [model::Test]PersonTable
   firstName: [model::Test]PersonTable.firstName,
   lastName: [model::Test]PersonTable.lastName
 }

 MappingTests
 [
   test_1
   (
     query: |model::Firm.all()->project([x|$x.legalName, x|$x.employeeSize, x|$x.employees.firstName, x|$x.employees.lastName], ['Legal Name', 'Employee Size', 'Employees/First Name', 'Employees/Last Name']);
     data:
     [
       <Relational, SQL, model::Test,
         'Drop table if exists FirmTable;\n'+
         'Drop table if exists PersonTable;\n'+
         'Create Table FirmTable(id INT, Legal_Name VARCHAR(200));\n'+
         'Create Table PersonTable(id INT, firm_id INT, lastName VARCHAR(200), firstName VARCHAR(200));\n'+
         'Insert into FirmTable (id, Legal_Name) values (1, \'FirmX\');\n'+
         'Insert into PersonTable (id, firm_id, lastName, firstName) values (1, 1, \'John\', \'Doe\');\n'
       >
     ];
     assert: '{"columns":["Legal Name","Employee Size","Employees/First Name","Employees/Last Name"],"rows":[{"values":["FirmX_LTD",1,"Doe","John"]}]}';
   )
 ]
)

Bug: Memory-leak issue when working with large model in text-mode

🐛 Bug Report

When working with a large model in text mode. If we try to edit and compile multiple times, the memory usage will go up substantially, crashing the app. The fact that we only see this for large models might not be too accurate, this might happen on smaller models, but less noticeable.

We detected this bug a while ago before many changes happened to Studio, we should confirm if this still happens though.

Step to Reproduce

1. Turn on browser memory usage monitor
2. Try to work on a substantially large model
3. Go to text-mode, make some small change (i.e. add a dummy class)
4. Hit F9 and go back to form mode
5. In the memory tab, try to collect garbage
6. Keep repeating (watch for memory usage increase - if that happens and never goes down, meaning that memory leak has occurred somewhere - we seem to observe this when we delete elements in the text mode and add new ones - though this is not confirmed, we must check more)

Expected behavior

Memory usage should not go up substantially over time.

Reproduction Code or Repo

cdm.txt

Possible Solution

We have done one pass of checking for memory leak in the past - check for marker /* @MARKER: MEMORY-SENSITIVE */ in the codebase.
For another round, we should go back and fix updateApplicationAndGraph method in GraphState for editor states reprocessing, also we should add methods to reprocess tree using IDs, and methods to reprocess element editor state properly.

More linting checks to ensure better app flow

Right now the app flow that we try to enforce is:
components/stores -> protocols -> metamodels. We want folks who write their own DSL presets to understand and enforce this flow as well as this helps if in the future they decide to break down their presets like the way we break down Studio right now. For this to happen, the best way is to provide a set of linting rules.

• Disallow any imports that contains /src/ in its path, this will break Typescript in build mode.
• Disallow imports of form ../../.../.. as index.js files are really meant for consumers outside of the current package.
• Enforce VX_ prefixes for file in protocol vx
• Enforce VX_ prefixes for exports in protocol vx
• Disallow cross protocol version imports
• Enforce hierarchy of imports

Feature request: Query builder improvements

Following is a list of query builder improvements and features to be done. Ideally, we should create separate issues when we work on big chunks listed here (e.g. #253, #254)

Context and Motivation

General

• Support viewing element in read-only mode in query builder: maybe we can, and we let people open the element in a modal or so and we let them use the visit element button. The point is we don't require them to leave query builder to visit the elements - #955
• #631
• #319

Query parameter

• Allow specifying parameter for query (support service use case) (#508)
• #540

Subtype handling

• #630

Explorer tree:

• CONSIDER For allow DnD multiple selected items, we need to implement multi drag source in react-dnd - check react-dnd/react-dnd#14 (comment) AND react-dnd/react-dnd#1016 (comment)
• Qualified properties modal - #956
  • CONSIDER the explorer on the left, we can add a list/explorer???
  • Show full signature (with paren, types, ...)
  • Potentially introduce color-coding
  • Indent for the param inputs -> i.e. make it look like an actual function
• Property Search - #782

Preview data:

• #324

Result Panel

• Improve integration with ag-grid and interact with the grid to edit projection (sorting, column position, pivoting, etc.): we can have a check box Sync table with projection -> when this is on, we will modify the table and the projection at the same time. - #957

Query Options / Result Set Modifier

• DnD to re-arrange sort columns

Projection

• #626
• Allow deleting projection/graph-fetch node via right-clicking/context-menu
• #959

Filter - #958

• #627
• #628
• BUG squash node (when deleting) added to the end of list, not in place/ do not respect order
• Rearrange DnD algorithm: prevent drop to children
• Rearrange DnD algorithm: prevent move to itself
• Rearrange DnD algorithm: prevent drop to immediate parent
• Show background when drag is over tree node group
• Show color coding for group on the side of the group when selected (like indent guideline in code editor)
• FANCY while dragging to rearrange, hover over node to open it (~2 seconds) (see VSCode/Macintosh behavior)

Post Filter (TDS Filter)

• #839

Consider (features we haven't confirmed yet)

• Support fiscal calendar

Bug: Association Editor doesn't allow selection of property types

🐛 Bug Report

Association view doesn't let you select the property types.
Throws: 'can't find property from association in associated class

Step to Reproduce

Steps to reproduce the behavior:

• Create two classes in an empty project
• Create a new association.
• fill in the property name and select the corresponding property type from the drop down

Expected behavior

Association view should allow creation as well as updation from the view itself instead of text view

Possible Solution

add a check before removing other types to ensure it is populated before removal

Feature request: Enable native text navigation/keyboard shortcuts in Mac OS

🚀 Feature Request

In Mac OS Option + arrow keys allows user to navigate text, jumping from word to word. In Legend in Safari, this functionality is not available. In addition, Mac OS also allows user to highlight text using Option + Shift + arrow keys. Requesting this same functionality in the Legend text editor.

Motivation

Would like to work on this feature?

No

Is your feature request related to a problem?

Yes — I cannot navigate my code in the text editor as efficiency as I would like it.

Describe the solution you'd like

Enable Mac OS native text navigation in Legend text editor.

Feature request: File generation editor improvements

• DISCUSS Support using association as scope field element in file generation editor - we should also take into account what we do in #640 as it provides an extension mechanism for what which element type we can add per generation type
• Use PureModelContextText for generation - see #640
• Allow downloading generated files (and folders - using jszip - most likely)
• Allow uploading files/folders for model loader
• Inform users when an artifact generation type has no properties

RFC: Rework graph feedback

💬 Request for Comments

This is the initial discussion, we will revisit and add actionable items to this thread

Graph completeness

Ensure proper graph (sub) elements initialization (no hole): If we start with a complete graph and do not allow people to modify the graph in a bad way, the graph is complete in the end

• When create a new thing, packageable element -> ask the right information, the right ones ==> give user some guidance while modifying the graph
• When asking for the right information doesn't make sense or become too much of a burden (i.e. service creation), we should use default what we can, if it makes sense to choose the first thing, e.g. while adding a new runtime, we can automatically detect the first available mapping in the graph ==> helps ease the pain and improve ergonomics for user
• If default or first thing values do not make sense (e.g. mapping or query in Service execution), we will create a stub, this leaves the graph without hole, but here we must enter validation and inform users that there are problems (see more below) ==> avoid confusing users, boost speed
  • We should scan the current usage of createStub() and find places where default values can be used instead, e.g. for TaggedValue and Stereotype, we should pick default values from system.
• We should auto-generate (with increment, e.g. property_1, property_2, notice the underscore _) for values where we can such as class property name or constraint ==> boost ergonomics, reduce the use of placeholder and allow quick prototyping
• RARE For information input that requires immediate correctness upon input, such as no spacing or no duplication in value, we can have an editor state for the input, e.g. having a save button ==> avoid going into bad graph state
• Also, in lambda editor, we block people from escaping if they have parser error ==> disallow incomplete JSON for lambda, i.e. incomplete graph
• Have validation engine runs in the background to soft-check on graph veracity and correctness, we can also maintain a list of errors and warnings (e.g. seevscode Problems tab) ==> inform users of the bad state of the graph
• Make validation blocks critical operations that can drastically throw off the graph such as compilation or saving ==> make the user responsible for the bad state of the graph, or else they CANNOT proceed
• CONSIDER The previous type of validation is based on the element only, it is very simple, it does not do any smart validation that involves inter-element in the context of the whole graph (such that needs us to pass down pure model), that kind of work, we can consider taking up, or completely leave that to the compiler ==> simplify the flow of validation, AVOID making validation do too much
• Have a good enough shape for error/warning generated by validation (and potentially compilation) so that we can consume using method in each element editor state to navigate to the UI component ==> zenith of feedback as this can pinpoint where the problems are in form view (UI component level)
• CONSIDER Since we don't keep track of reverse-pointers, we used to not have a good way to validate elements when we delete an element from the graph. If we just maintain a isDeleted flag at the element level then pointers to sub-element models like tag or property also need to resolve that, causing the validation code to look convoluted. However, with the refactor in metamodel to have Reference, we are now able to better identify places, where we use references and convenient, have ownerReference where we can check if the owner/parent element is deleted or not using isDeleted flag (for secondary-sub element like database column, we can go up to table and then schema to verify if they are no-longer exists). Therefore, we can also do some analytics on parts of the graph using the deleted element/sub-element and put that as validation error, thus, potentially don't need to auto-compile post deletion. (Technically, we can already do this, but now we can encapsulate this logic nicely in the references instead of repeating this validation all over the places and pollute metamodels)

Graph compile correctness

Ensure giving user the right feedback about the bad state the graph/project is in: to ensure correctness, we give feedback and offer fixes -> but right now we can't really offer fixes/auto-heal

• All lambda editors should have local feedback, do not fall back to text mode ==> immediate feedback is more helpful for quick edits, text mode can be overwhelming and disruptive
• Feedback last compilation: We should/may have a flag for last time compilation, we can have an icon somehwere to tell the user about compilation status, if they change something, update the indicator to unknown state, also have an indicator for compilation in progress ==> user knows the project in its current state is compiled (hard check)

Graph Runtime correctness

Test to run

• Feedback build: we can setup a poll when we fetch the builds, i.e. right after something is pushed, we initiatate a call to fetch builds, then we can keep polling until there are no more pending builds, as we find new pending builds, we keep adding to the lsit of waiting builds and keeps polling, do proper timeout cancel when we leave, also show an indicator out in the UI or even having notification to warn user about build failure/success ==> user knows if antyhing beyond compilation is messed up
• Also care about the build status on master (we will build a scheduler for this, and notify people)

Simplify workflow

So everything is more predictable and less buggy

• When hard compilation occur, we should fall back to text mode when source information can not be located, or when we haven't worked on component-level navigation based on source value ==> text mode is a good place for people to see the whole graph and fix problems anyway ~ in many case, more efficient
• There are operations right now that might be really disruptive, such as delete, rename, move, we should call compilation after them as we haven't done back reference to heal the graph yet. i.e. avoid doing graph healing and smart thing for now ==> avoid bugs, maintainance, and make it easier for users

Feature request: Allow re-arranging editor tabs by drag-and-drop

🚀 Feature Request

Motivation

We should allow re-arranging tabs by drag-and-drop. First, we can start with the general editor tab bar, for the bonus, we can implement the same changes to the mapping editor.

Bug: Change detection reports false positive when creating a new mapping test

🐛 Bug Report

Step to Reproduce

Use the models from the attached file. Add a new mapping test similar to test_1 or test_2, try to save and we would see a 1 change detected on the mapping, this won't go away even when we try to save again.

Reproduction Code or Repo

models.txt

Possible Solution

Needs more debugging to find the root cause. But we might be related to #68
When we do not have to workaround, we should investigate this more

Feature request: Apply dark-mode consistently

Before we work on theming support (#264) and UI componentization, we should make the look and feel of the app more consistent: that is to make everything dark-themed. However, we should also make sure the panels are discernable visually.

Context and Motivation

• Dark-theme for mapping editor
• Dark theme for diagram
• Dark-theme UML editor
  • Class editor
  • Enumeration editor
  • Association editor
  • Profile editor
  • Function editor
• Dark-themed modals:
  • Service creation: componentize this - restyle this as well
  • Mapping element creation
  • Search element
  • Create element

RFC: Modularization (last mile)

Legend Studio core can still be broken down further; right now it still contains DSL-specifics and has the graph models, engine, and components all packed together. Following is a list of todos:

• Remove or move-out everything in legend-studio/src/utils (#410)
  • Move Logger to shared, keep CORE_LOG_EVENT in core
  • Eliminate usage of SourceInformation in TextEditorUtil
  • Move TextEditorUtil to legend-studio-components
  • Move logic inside of MappingResolutionUtil to stores or metamodels accordingly
• After elimitating legend-studio/src/utils/, break down legend-studio/src/models` into: (#422)
  • legend-server-sdlc
  • legend-server-depot
  • legend-graph
• Modularize legend-extension-dsl-diagram: this is currently complicated by the fact that we're using DiagramRenderer (which consumes Diagram) in ClassEditor (#439)
• #687
• Modularize PlanViewer - #717
• IMP Modularize AbstractPureGraphManager - #1266
• Modularize flat-data: legend-extension-store-flat-data - flat-data might get removed in the near future - if we finish #315 - maybe we could just remove all flat-data logic
• Modularize relational: legend-extension-store-relational
• After relational, move external format out of core potentially?
• Modularize service: legend-extension-dsl-service: We need to modularize how we start Legend Query
• Modularize generation: legend-extension-dsl-generation - This is considered part of core so we will likely not need to modularize it until much needed
• Modularize external-format: legend-extension-external-format (SchemaSet, Binding, etc.) - This will be motted as we really consider external format logic to be part of core, this will be consistent with engine
• Modularize mapping: legend-extension-dsl-mapping - be careful and really consider if we need to do this. This will break Studio down to a really really simple core though, which is a plus - This will be omitted as it's not really necessary and could cost us a lot to do, even https://github.com/finos/legend-engine is not doing this.

... and the last mile is often never easy to cross, but at least one does have it in their sight.

CVE-2019-11358 (Medium) detected in jquery-2.1.4.min.js - autoclosed

CVE-2019-11358 - Medium Severity Vulnerability

Vulnerable Library - jquery-2.1.4.min.js

JavaScript library for DOM operations

Library home page: https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js

Path to dependency file: legend-studio/node_modules/deep-diff/test/tests.html

Path to vulnerable library: legend-studio/node_modules/deep-diff/test/tests.html

Dependency Hierarchy:

• ❌ jquery-2.1.4.min.js (Vulnerable Library)

Found in HEAD commit: 1de7f5505e98c70f04393f70b9fc52afccdfa463

Found in base branch: master

Vulnerability Details

jQuery before 3.4.0, as used in Drupal, Backdrop CMS, and other products, mishandles jQuery.extend(true, {}, ...) because of Object.prototype pollution. If an unsanitized source object contained an enumerable proto property, it could extend the native Object.prototype.

Publish Date: 2019-04-20

URL: CVE-2019-11358

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-11358

Release Date: 2019-04-20

Fix Resolution: 3.4.0

Feature request: Alloy Drag and Drop on Associations for scope field in File Generation Editor

Motivation

An association is an acceptable type of scope field element in the file generation editor, therefore the drag and drop functionality for the explorer to the scope field should be enabled.

Feature request: Query builder: Rework graph fetch support

The graph fetch support MVP that we have right now doesn't really have the best UX, for one thing, it is inconsistent with the projection counterpart, which is DnD-based

Implementation Plan

• Use DnD for graph fetch construction: note that we don't need to support derived property parameters, alias, or sub-typing right now (#237)
• Support right-clicking on class property node in the explorer and add all simple property nodes to fetch structure (#237)
• Support right-clicking on a simple property node in explorer and add that property to the fetch structure (#237)
• Add context menu: right now the only option we can think of is Delete/Remove but it's still useful that way
• Support sub-typing (this might take time)
• Support qualified properties
• Support right-clicking on a node and add an alias/ or just change the label directly like what we do for projection
• Support rearranging node within an parent node
• Support moving nodes between nodes that are just aliases of one another
• Support contextual DnD, if I drag and drop a node from the explorer tree to a node on the graph fetch, we must try to build the graph tree from there if possible - this serves the case where there are 2 or more nodes for the same property but with different aliases

NOTE: When we support alias since we might need to support continuous edition of the alias, it's not wise to have the alias being factored in as ID for the node as this will require a complex algorithm to change the tree data on every keystroke while changing the alias. So when there are multiple nodes for the same property we should add a suffix to the node ID (e.g. car.wheels__1). The first property tree node will be labelled without the suffix, so when the user drag and drop a node to the fetch tree, we can use that as the default node to traverse and add the new node, we can also do an extra check on top: if there are multiple nodes with the same property, first check for the node that doesn't have an alias, if there is none, use the node where the ID does not have a suffix, then use this node to add the new node, in order to best fit user's intent.

Feature request: Connection editor improvements

• We should discuss how to improve the user experience when creating a new connection from scratch
• MAYBE Support XML connection: This falls under model connection so we should allow user to switch back and forth between JSONModelConnection and XMLModelConnection
• Allow editing post-processor in relational database connection editor - #945
• BUG for relational DB connection, we haven't modularized DatabaseType so in the dropdown, we are seeing all types. In fact, we should have a relational DB setup a matrix between types, authentication strategy, authentication strategy, and post-processor so when the user picks one, we can automatically pick the right preset for them (we shouldn't block them from using whatever combination they wish to though) - #946
• Improve the layout of connection editor, to no longer visually separate the panels by border lines, following #659 for what we do with query builder when showing multi-panel editor
• Rework database store generation flow (moved to - #268)

CVE-2020-11022 (Medium) detected in jquery-2.1.4.min.js - autoclosed

CVE-2020-11022 - Medium Severity Vulnerability

Vulnerable Library - jquery-2.1.4.min.js

JavaScript library for DOM operations

Library home page: https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js

Path to dependency file: legend-studio/node_modules/deep-diff/test/tests.html

Path to vulnerable library: legend-studio/node_modules/deep-diff/test/tests.html

Dependency Hierarchy:

• ❌ jquery-2.1.4.min.js (Vulnerable Library)

Found in HEAD commit: 1de7f5505e98c70f04393f70b9fc52afccdfa463

Found in base branch: master

Vulnerability Details

In jQuery versions greater than or equal to 1.2 and before 3.5.0, passing HTML from untrusted sources - even after sanitizing it - to one of jQuery's DOM manipulation methods (i.e. .html(), .append(), and others) may execute untrusted code. This problem is patched in jQuery 3.5.0.

Publish Date: 2020-04-29

URL: CVE-2020-11022

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://blog.jquery.com/2020/04/10/jquery-3-5-0-released/

Release Date: 2020-04-29

Fix Resolution: jQuery - 3.5.0

Feature request: Query builder: Support derivation in projection

Right now the only type of lambda that is supported in the query builder projection column is a property expression. We should allow any expression to be typed.

Context and Motivation

We want to support use cases like the following:

test::Person.all()
   ->project
    (
      [
        x|$x.lastName + $x.firstName
      ], 
      [
        'Full Name'
      ]
    )

Implementation Plan

We do a very strict check to ensure the only form of column lambda we support in form view is property expression, anything deviate from that would be considered derivation (or free form). To accommodate for this, we need to create a new wrapper type for ValueSpecification, we call this UnknownValue (this is in line with what we want to do in #315)

class UnknownValue extends ValueSpecification {
  content: object;
}

Steps

• Support compilation in form mode (F9)
• Support detecting derivation and converting it to UnknownValue
• Support roundtrip for processing unsupported column expressions in project() expression
• Support reveal compilation errors in form mode for derivation
• Support adding new property in form mode
  • Support + - default to x|'' in projection panel
  • Support DnD to re-arrange
• Support roundtrip for processing unsupported column expressions in groupBy() expression

Optional

• Support using hotkey F9 to compile in query builder
• Write a changeset for derivation feature
• Support custom aggregation - default to x|x->count() -> after discussion, this does not seem to make a lot of sense since we should only allow a fairly restricted sets of functions for aggregation
• Support custom filter - default to x|true -> This could make sense, but we would move it back to the main thread
  #256
• Support converting a normal property expression to a derivation (#328)

Bug: adding slash to workspace ID can cause unexpected 404 error

🐛 Bug Report

Issue

When attempting to create a workspace, if the user mistakenly uses some invalid character, for example workspace/something/something, this will then trigger a POST request to SDLC server with that as part of the URL, hence, causing a 404 - Not found.

Proposed solution

Right now, Studio relies on SDLC to handle the validation, but really the thing we need to do is to encodeURI all query parameters, which we should do in NetworkUtil

Feature request: Query builder: Support aggregation

Support using groupBy to do simple aggregation in Studio

Context and Motivation

The idea is to make switching between groupBy and project fast and convenient. This is due to user flow, where the user specifies the columns they are interested in, and then pick the column they want to groupBy. Upon this, we must turn project into groupBy.

In the future, we might potentially do more fancy operation right out from the result grid to support aggregation (e.g. user can right-click on a column and aggregate`

Implementation

The following is the query in Pure code

{|
model::Account.all()
   ->groupBy([x | $x.accountNumber,
      x | $x.id,
      x | $x.accountNumber],
      [agg(x | $x.active, y | $y->count())],
      ['Account Number',
      'ID',
      'Account Number1',
      'Active Count']
   );
}

For each type, the set of supported functions for each type is detailed below. We should use the same approach that we use with filter operators.

• Date/DateTime: Count count(), Distinct Count distinct()->count(), Distinct Value uniqueValueOnly(), Min min(), Max max()
• Integer: Count count(), Distinct Count distinct()->count(), Distinct Value uniqueValueOnly(), Sum sum(), Average average(), Min min(), Max max(), Std Dev (Population) stdDevPopulation(), Std Dev (Sample) stdDevSample()
• String: Count count(), Distinct Count distinct()->count(), Distinct Value uniqueValueOnly(), Concat joinStrings(';')
• Boolean:Count count(), Distinct Count distinct()->count(), Distinct Value uniqueValueOnly()
• EXTRA let people choose the delimiter for the concat operation

Steps:

• Show aggregation functions in dropdown based on types
• Build state based on input query using groupBy
• Build query lambda using groupBy based on state
• Work on an UI to switch between project and groupBy
• Take care of column order when switching between project and groupBy
  • For modularity of the app, we should create new states for aggregation/groupBy instead of reusing projection states
  • Note that projections right after promoted to aggregation columns are automatically moved to the end. When a pivot is switched back to become a normal projection, we should add it to the end of the list. Note that this behaviour is due to the limitation of groupBy (if we use olapGroupBy (in the future), this restriction might be lifted)
  • Support DnD to arrange columns within the aggregated columns

Bug: Typo in "Model Conneciton" in "Create a new connection" screen

🐛 Bug Report

The drop down list in the "Create a new connection" screen has the option "Model Connection" instead of "Model Connection"

Additional context

Also see typos in NewElementState.tsx. Search for CONNECITON

Bug: File Generation Editor does not reset when changing elements

🐛 Bug Report

Step to Reproduce

Steps to reproduce the behavior:
Create two classes (Class A, Class B).
Open the generated view for Class A to avro.
Open the generated view for Class B to avro.
File generation editor for class b will not auto open the avro generated file (it will for class A).

Expected behavior

When opening the generated view for Class B, the file generation editor should open the first avro generated file.

Feature request: Rework UX and workflow for handling multiple SDLC servers

Overview

In the past, we need to do a quick pass (#147) to allow Studio to work with multiple instances of SDLC servers. However, that quick attempt causes many loose ends. For example, right now whenever the user launches Studio, regardless of the URL route, they are being asked which SDLC server to choose in case where multiple servers are available in the config. This is inconvenient. Also, with the naive implementation we had, we force all sub-routes of Studio to be vocal about which SDLC instances it uses. This seriously limit the extensibility of our app in that conceptually, we cannot have any part of Studio that can work without caring too much about SDLC (see #617 for example)

Implementation plan

• Rework the approach done in #171 as it disallows us to have non-SDLC base URL pattern (i.e. part of Studio that might not need SDLC at all)
  • Refactor to accept only 2 types of SDLC-instance patterns: studio/-/... and studio/sdlc-:sdlcServerKey/...
  • Remove the decorator around extraApplicationPageRenderEntries URL patterns
  • Remove _currentSdlcServerOption in StudioConfig
• MAINTENANCE Be consistent and rename everything to SDLC
• Since we have the default SDLC option, we can remove the isConfigured flag in StudioConfig and the configuration screen altogether
• For SDLC setup, always use the default so we don't block the app
• In LegendStudioApplication, separate the concern:
  • We will scan the URL and predict the SDLCServerOption to pick: update the current scanner to distinguish between 3 cases:
    • When the URL matches SDLC-instance pattern: and the key is found -> standard flow
    • When the URL matches SDLC-instance pattern: and the key is not found -> redirect to use defaultSdlcServerOption
    • When the URL does not match SDLC-instance pattern: pass-through and just use the default SDLC, i.e. /-/
  • Update <Redirect> in <Router> of LegendStudioApplicationRoot to use defaultSdlcServerOption
  • Show the SDLC selection on the top right (like Location for shopping pages) - replace the URL with the new key and reload the page
• Adjust /-/view/gav/:gav url pattern to /view/:gav - see the work that has been done in #639 for #638

Additional context

No response

Feature request: More descriptive version info in About dialog

We should revamp the About dialog in the following ways:

• Change the creation of version.json, move away from using maven plugin (#147)
• Change how we show version info (only show commit/build-time but we can also include commit-time/branch/etc. in the version.json file) (#147)
• Instead show core version and plugins/presets versions
• Add a copy button that will copy the version info in the following format for example

Version: 1.57.1
Commit: 507ce72a4466fbb27b715c3722558bb15afa9f48
Date: 2021-06-17T13:28:32.912Z
Electron: 12.0.7
Chrome: 89.0.4389.128
Node.js: 14.16.0
V8: 8.9.255.25-electron.0
OS: Darwin x64 20.5.0

RFC: Self-hosting - Use generated protocol models and metamodels

💬 Request for Comments

We are a modelling tool, in fact, our protocols and metamodels are models we could build in Pure as well, and worse yet, we are not auto-generating these models but repeating their definition in 3 code bases legend-studio, legend-engine and legend-pure. We should model these in Pure and code-gen their definition into Typescript for consumption in Studio codebase.

Simplify metamodel

We need to simplify metamodels so that these can be auto-generated from models in Pure. After this, we're sure that the structure of metamodel more consistent and easier for code-generation

• Move the lambdaId to stores - #227
• Move PackageableElementSelectOption back into stores, don't leave those in graph and metamodels
• Remove rendering logic like label, e.g. MappingElementLabel - #1068
• For fields that we added to support better graph navigation (i.e. owner, parentClass, etc.) that are not in the actual metamodel, we should prefix them with _, such as _owner` - #1153
• Move helpers logic into corresponding classes MappingHelpers, PackageableElementHelpers like in finos/legend-pure - #1159
• IMPORTANT Add field integrity verification to hashCode (basically check if properties annotated with ! are actually non-null) - Detailed in #288 but no longer deem needed
• Consider to remove unnecessary interfaces (these make it hard for tools like VSCode to know where the interface method being called):
  • Stubable interface -> Have a factory to create stub and check for stub maybe? -> move the logic out for stubable to a factory or some helper methods to initialize new instances - #1159
  • Hashable interface, we don't need models to implement this interface anymore. - see below
• Move validation logic to top-down as well - we will most likely do this later in #1168
• For certain @computed cases (e.g. allSuperClasses, allSubClassses) we might not need that as computed value at all - #1159
• MAYBE Turn ALL metadata fields (fields like Property's owner, uuid etc.) to readonly and leave the access modifier as public - ONLY during initialization do we allow to change these readonly property using the Writable improved mapped type modifiers trick in Typescript, write a good doc there to explain why this abuse should only be treated as a hack used ONLY during graph initialization. Or we can simply use Object.assign syntax... - #1153

type Writable<T> = { -readonly [K in keyof T]: T[K] };

NOTE: there are fields that the UI does not and perhaps should not allow changing like class to a ClassView, but that's an UI decision, this field is not metadata, and it should not be made readonly

• Remove all constructor parameters. Then remove all initialization logic from constructor, in fact, we could remove/make private the constructor all together and replace them by static creator methods
  • Do not initialize any value, as these should be taken care of by the static creator method
  • Use ! and ? in the property definition
  • Remove constructor
• Move hashing logic out of metamodels:
  • After we move hashing and validation logic, revisit spots where we should use isStubbed_... method and use them, right now to avoid bad dependencies between models and the helpers method, we don't do this
  • Move remaining code-logic in DSL Diagram classes out to helpers.
  • Consider moving logic in Packageable Element out
  • Use the following technique to move hashing out. Remember that this is useful for derivations, which are the hardest things left to move. Hashing is one example of derivations which is useful to us and we should keep around. Also, if we don't do hashing at protocol level, I think doing hashing in the reactive manner is still the best, but we will see

// ModelExtension.ts
declare module './models/metamodels/pure/packageableElements/PackageableElement' {
  interface PackageableElement {
    get path(): string;
    get hashCode(): string;
  }
}

...

export {}; // export this so `Typescript` --isolatedModule check does not complain

// --------------------------------------------------------------

// Some file that we call once when we load the app
export const decorateMetamodel = (): void => {
  // e.g. Constraint
  Object.defineProperty(Constraint.prototype, 'hashCode', {
    get: function (this: Constraint) {
      return hashArray([
        HASH_STRUCTURE.CONSTRAINT,
        this.name,
      ]);
    }
  });
};

Other considerations

• CONSIDER We probably would need to take care of visitors as part of code generation. JS does not support overloading like Java so we can't really have 2 accept() methods, we would need to have them differentiated, e.g. accept_SetImplementationVisitor() and accept_PropertyMappingVisitor() for embedded property mappings.
• CONSIDER How do we support Typescript union type? Maybe we can use Any in Pure and then use a tagged-value to annotate union type?

Class/Interface swap:

Since Pure support multiple inheritance, we might need to do what legend-pure does by turning metamodels into interfaces to support multiple-inheritance (with C3 linearization support?), something like the following

• Create CoreInstance interface and ConcreteCoreInstance class:

interface CoreInstance {
  /**
   * Can be used to uniquely identify the instance in a collection (used for UI majorly)
   */
  _UUID?: string;
}

class ConcreteCoreInstance implements CoreInstance {
  _UUID?: string;
}

• We should turn each class into _Impl (that extends ConcreteCoreInstance) and have everything as interface that extends CoreInstance: i.e. Class vs ClassImpl.
• MAYBE We should have optional UUID to each CoreInstance, add a method to allow setting this UUID. NOTE that if we make this non-optional, we're adding a lot of data to the graph
• At this point, we lose the ability to do instanceof, to recover, for each interface, we can do the following:

// Class.ts
export interface Class {
  ...
}

export const Class_InheritanceSignature = 'Class';

// Class_Impl.ts
class Class_Impl implements Class {
  ...
}
Class_Impl.inheritanceSignature = Class_InheritanceSignature;
Class_Impl.prototype.superTypes = [PackageableElement_InheritanceSignature, CoreInstance_InheritanceSignature];

const instanceOf = <T>(val: unknown, inheritanceSignature: string): val is T => val.protoype.superTypes.includes(inheritanceSignature);

interface A {
  type: "1";
}

interface B {
  type: "2";
}

interface C extends A {}

class D implements C {
  type: "1" = "1";
}

class E implements B {
  type: "2" = "2";
}

const instanceOfA = (val: Record<PropertyKey, any>): val is A => {
  return "type" in val && val.type === "1";
};

const invD = (d: D): void => {
  console.log(d.type);
};

// NOTE: this line is perfectly fine in Javascript, but during compile time, Typescript will complain
invD(new E()); // Argument of type 'E' is not assignable to parameter of type 'D'

Feature request: Create an npx executable to auto-generate the Legend coding environment and web app

🚀 Feature Request

Create an npx program (something like an yeoman generator to code-generate Legend application coding environment (like this repo itself) including an webapp, like how docusaurus and create-react-app does it

Motivation

Right now we publish both libraries (to NPM) and web application (i.e. @finos/legend-studio-app to Dockerhub) from this repository. For folks who need to build their tailored version of Studio app, they would need to copy almost exactly @finos/legend-studio-app, and pick the plugins they need as well as include the plugins they build themselves - this is what we do inside of GS as well. So it would be a big time saver to have a script to just code-generate this module.

Implementation

Use cli to do so - https://dev.to/iulianoctavianpreda/run-your-npx-script-directly-from-github-create-your-own-cli-commands-and-other-stories-4pn3

• Studio how npx and https://yarnpkg.com/cli/dlx work
• Create a command-line prompt interface to ask questions about (1) the name of the project, (2) scope of the package, etc. to generate the package.json: probably we could use enquirer here
• We need to create a simplified package.json - here we can look at what we do for https://github.com/finos/legend-studio/blob/master/package.json
  • Add relevant dependencies for things like eslint, babel, stylelint, prettier, etc. Here we need to trace the latest version for these, we could use yarn workspaces list --json
• Copy over files/dirs:
  • .vscode
  • .editorconfig
  • .eslintignore
  • .eslintrc.cjs
  • .gitattributes
  • .gitignore
  • .prettierignore
  • .prettierrc.cjs
  • .stylelintignore
  • .stylelintrc.json
  • babel.config.cjs
• Create a README.md file
• Create a simplified tsconfig.json
• Create a dummy yarn.lock
• Create /packages
• Create /packages/legend-studio-deployment
  • package.json: use (2) scope of the package for generating the package name, simplify from packages/legend-studio-deployment/package.json
  • Copy the rest of the content of packages/legend-studio-deployment - we could let the user customize this even more based on what we got in the first step when asking for user inputs
  • Here we need to trace the latest version for these, we could use yarn workspaces list --json
• Create a script that
  • Install Yarn with latest version yarn set version stable https://yarnpkg.com/getting-started/install
  • Setting other attributes for Yarn - see what we do for our own .yarnrc.yml - https://yarnpkg.com/configuration/yarnrc - one can set an attribute by running a command like https://yarnpkg.com/cli/config/set
  • Do an yarn install for initializing the yarn.lock file
  • Do a git add . and git commit -m "Setting up Legend application working environment"
  • echo Welcome! and some kind of getting started instruction

Feature request: Support preview of Markdown for text element editor

🚀 Feature Request

Motivation

For text element editor, if the user select the type to be markdown, it would be nice to have a side panel to preview the markdown text. react-markdown seems like a good library to use in this case.

Dependencies Watchman

Dependency upgrades/bug-fixes to track

This list keeps track of dependencies that are either unstable or blockers to development in our project. Since unstable dependencies are usually marked with tag @next, or currently in alpha, beta channels, which can be ignored by dependabot. For now, we want to have this pinned issue as the watchman, eventually, we should split it into smaller issues

• @pmmmwh/react-refresh-webpack-plugin: Using beta version. Part of webpack@5 upgrade. (Resolved in #495)
• webpack-dev-server Using beta version. Part of webpack@5 upgrade (Resolved in #422)
• babel: In babel@8, a lot of configurations are to be set to default, we should keep track of these to slim down our babel preset.
• jest: Jest does not fully support ESM - See jestjs/jest#9430
  • #502
  • Jest@27 will use jest-circus by default so we don't need to manually specify this like right now.
  • Upgrade to Jest@27 is currently being blocked due to @types/jest not being upgraded. This is causing a weird error: The inferred type of ... cannot be named without a reference to 'jest/node_modules/pretty-format/build/types'. Also, we should wait out a little bit as we need to confirm Jest@27 works well with testing-library
• yarn/berry:
  • #501
  • yarn workspaces foreach does not support --skip-missing flag, so if a workspace does not have a script with name specified, it will error out when we try to run such script on all workspaces (unlike lerna). See yarnpkg/berry#62
• #500
• eslint-plugin-import Wait for the fix for no-extraneous-dependencies in import-js/eslint-plugin-import#2121 to be published and enable that rule
• husky: husky@5 forces us to change our license. We have to wait till the early-access phase is over.

Dependency features to try out

This list keeps track of nice dependency features that are already supported in the versions we have, but we haven't quite had the time and effort to apply.

• yarn/berry: PnP is nice to try out but currently does not support ESM so we have to use node_modules as node linker mechanism. See yarnpkg/berry#638
• mobx: Use override for computed properties where appropriate (i.e. when both super class and sub class have concrete methods/getters) - this will be enforced and checked by mobx automatically. The last thing we need to do is to stop using super. in @computed
• mobx: Use flow annotation with makeObservable and flowResult instead of wrapper flow(function * ...) - Check out doc
• typescript: [email protected] supports for the override syntax. This is currently blocked by prettier (#244)

Feature request: Improve validation feedbacks for runtime editor

• Analytics: Show warning if a store does not belong to a specified mapping (not warning, just indicator)
• Analytics: Show warning if a store in a mapping does not have a connection
• Analytics: Show warning for the runtime -> does not have a connection for the store (just based on the mapping)
• Analytics: in mapping editor, we can show covering runtime in the mapping (maybe not opened by default as it takes screen real-estate but as a collapsed tab in the explorer)

Bug: Execution of mapping results in "IndexOutOfBoundsException: Index: 0 Size: 0"

How to reproduce

###Relational
Database model::DemoDataBase
(
  Table FirmTable
  (
    id INTEGER,
    Legal_Name VARCHAR(200)
  )
)


###Diagram
Diagram model::MyDiagram
{
  classView 1380bb9c-d585-436a-808a-dc5594f8d7ac
  {
    class: model::LegalEntity;
    position: (459.0,110.0);
    rectangle: (141.03759765625,44.0);
  }
  classView 9257b5fd-0852-49e0-9529-9024b0938a0e
  {
    class: model::Firm;
    position: (428.0,250.0);
    rectangle: (201.54296875,100.0);
  }
  classView d0c6bbde-fb0c-4c60-baaa-a4b86deab7d0
  {
    class: model::Person;
    position: (732.0,263.0);
    rectangle: (162.43603515625,72.0);
  }
  propertyView
  {
    property: model::Firm.employees;
    source: 9257b5fd-0852-49e0-9529-9024b0938a0e;
    target: d0c6bbde-fb0c-4c60-baaa-a4b86deab7d0;
    points: [(528.771484375,300.0),(813.218017578125,299.0)];
  }
  generalizationView
  {
    source: 9257b5fd-0852-49e0-9529-9024b0938a0e;
    target: 1380bb9c-d585-436a-808a-dc5594f8d7ac;
    points: [(528.771484375,300.0),(529.518798828125,132.0)];
  }
}


###Text
Text model::DemoGoals
{
  type: plainText;
  content: 'Create a Data Model';
}


###Pure
Profile model::MyProfile
{
  stereotypes: [important, legacy];
  tags: [alias];
}

Enum model::IncType
{
  Corporation,
  LLC
}

Class <<model::MyProfile.important>> {model::MyProfile.alias = 'Corporation', meta::pure::profiles::doc.doc = 'This is a model of a firm'} model::Firm extends model::LegalEntity
{
  incType: model::IncType[1];
  employees: model::Person[1..*];
  NumberOfEmployees() {$this.employees->count()}: Number[1];
}

Class model::Person
{
  firstName: String[1];
  lastName: String[1];
  age: Number[1];
}

Class model::LegalEntity
[
  LegalNameStartsWith: $this.legalName->startsWith('L')
]
{
  legalName: String[0..1];
}


###Mapping
Mapping model::MyMapping
(
  *model::Firm: Relational
  {
    ~mainTable [model::DemoDataBase]FirmTable
  }

  model::IncType: EnumerationMapping
  {
  }
)

Mapping model::MyMapping2
(
  *model::Firm: Relational
  {
    ~mainTable [model::DemoDataBase]FirmTable
  }
)

Feature request: Allow exporting/downloading diagram as an image (PNG/SVG)

🚀 Feature Request

Motivation

Right now diagrams are already rendered in a canvas, would be nice to be able to convert this to PNG/SVG if possible so people can download and use this elsewhere.

Bug: Mapping transform lambda does not respect section import

Bug Report

Steps to Reproduce:

Try to use the following text in hackermode

Class test::units::dest::ClassWithPound
{
  weight: unit::Mass~Pound[1];
}

Class test::units::dest::DecomposedHealthProfile
{
  fullName: String[1];
  weightUnit: String[1];
  weightValue: Number[1];
}

Class test::units::dest::HealthProfile
{
  fullName: String[1];
  weight: unit::Mass~Kilogram[1];
}

Class test::units::src::_ClassWithKilogram
{
  weight: unit::Mass~Kilogram[1];
}

Class test::units::src::_DecomposedHealthProfile
{
  fullName: String[1];
  weightUnit: String[1];
  weightValue: Number[1];
}

Class test::units::src::_HealthProfile
{
  fullName: String[1];
  weight: unit::Mass~Kilogram[1];
}

Measure unit::Mass
{
  *Gram: x -> $x;
  Kilogram: x -> $x * 1000;
  Pound: x -> $x * 453.59;
}


###Mapping
import unit::*;

Mapping test::alloy::units::unitComposeMapping
(
  test::units::dest::HealthProfile: Pure
  {
    ~src test::units::src::_DecomposedHealthProfile
    fullName: $src.fullName,
    weight: Mass~Kilogram->newUnit($src.weightValue)->cast(@unit::Mass~Kilogram)
  }
)

Compilation should be fine. Exit text mode and re-enter. Try to compile again, there's a problem with unknown type Mass~Kilogram

Explanation

Right now we don't poke into process transform lambda, so when we parse the text for that lambda, we got the partial path for Mass~Kilogram, but when we internalize the model in Studio, we currently swallow the import as we don't support it. As such, we end up with no import info and lambda with partial path, causing compilation error.

This can be solved when we properly process the lambda, or when we start storing the section import info

Feature request: Provide extension to send additional elements to execute query function

🚀 Feature Request

Right now we prune our graph for specific elements when executing a query. There is no way for a preset to send additional elements when executing. This may be relevant as execution may require additional model elements to properly execute.

Describe the solution you'd like

Add an extension function to PureProtocolProcessorPlugin to provide the ability to send additional execution elements.

Feature: Support for sub-typing handling

Right now we do keep track of subtypes and supertypes in the graph, but for a lot of our checks, we don't handle covariance and contravariance properly.

• Covariant for union operation class mapping type scanning (e.g. Worker extends Person, so if the target class for the op class mapping is Person, any class mapping for Worker should be listed as well)
• Expected type label in the lambda editor now does not check for subtype, only strict type equivalent.

Feature request: Editor support for relational (MVP)

We will start building forms to edit relational-related entities (i.e. stores, mappings, connections, diagrams, etc.). This is not meant to be exhaustive but just the MVP, which lists out items to be done in order of priority. We will keep this around until we're happy with the MVP and move the remaining outstanding items to another thread.

Implementation Plan

• Relational mapping source selector (#207)
  • API expose an engine API to transform (grammar <-> JSON) relational operation element (finos/legend-engine#232)
  • Leave relational operation element raw in relational property mapping
  • Operation element editor
  • Source tree need to support joins
  • Support DnD for joins from source-tree
• Relational mapping execution panel: note that we will use H2 for this
  • Use query builder
  • Auto-generate test data (currently blocked by engine) - #947
• Relational mapping test panel
  • Create an API that takes in mapping, model, source/target class -> generate test data: this API will accommodate for M2M use case as well (and potentially flat-data), if there is no connection specified -> use H2, do some dummy data generation (if it gets too complicated, throw?), if there is a connection specified -> use the API that we haven't released yet
  • Create a protocol to hold relational mapping test and make it work with test runner (finos/legend-engine#238)
• Basic support for database editor
  • Be able to generate store from connection
  • Basic join editor (need more investigation, we can take a look at Microsofts Access) - #949
• More advanced lambda builder for relational property mapping: supports function compose, group operators, filter, groupBy, etc.
• Support identifying/interpreting Primary Key (PK) in relational mapping
• Support embedded relational property mapping - #948

Feature request: Avoid trapping users in bad editor state due to change detection

There are a few problems with change detection right now that we should address to unblock users: i.e. to allow users to save their models in those case at least. In particular, we should do the followings:

TODOs

• Allow change detection to be failed by hash computation error, right now we do nothing on hash error
• CHECK Block normal syncing while change detection is running as this might cause SDLC to throw entity already exists/created error
• Support hard-save functionality that will just save all entities (i.e. like model loader) without computing the entity changes
  • Add this button to local change panel
  • MAYBE If the quick-save call fails, show a dialog or something to prompt for disabling change detection
• When there's a problem with change detection, disable it (turn if off).
  • Show this in status bar (refer to how VSCode shows ESLint disable)
  • Show some icon in local change panel
  • In this mode, syncing is replaced by hard-save and we should put a blocking modal to confirm user's intention, something along the line of: change detection is disabled, so we cannot identify the changes you made locally in order to optimize save, this save action may take a long time, so be patient.
  • Make sure if change detection is disabled, the refresh action, quick-save action on the panels are disabled too, and the diffs are not shown (also can show a warning placeholder in the panel to say change detection is not running)

Improvement

• In case change detection is broken or reporting false positive, we can technically get the models from SDLC, and compare JSON-to-JSON with the entities being serialized from Studio graph to do a one-off change detection to help quick-save; this would be more efficient than hard-save but a little more complicated to implement so we will handle this later.
• We should re-evaluate and check what this would mean for the standalone review page (there, if diff is not showing, we need to call comparison API from SDLC instead).
• We should also re-evaluate what this would mean for conflict resolution: the fact that change detection is failable might means we need to block conflict resolution accept operation and modify SDLC API

Draft: Studio codebase improvements post OSS

This lists out tasks to do post OSS of Studio and release of relational capabilities. From this point on, Studio is encroaching version 1.0 🎉 . But there is much stabilization work to be done. Below, we just list out several that we know about. Keep in mind that we would definitely need to come back to this and reorganize, as well as, break this down into multiple RFCs and issues. But this will be the home for ideas for now.

Setup end-to-end test suite

Currently, we have a set of end-to-end (e2e) tests implemented with Cypress. Since these are written by many people at different stage in the projects, they could lack consistency and organization; also modularization somewhat messed up this suite, so we need to cleanup, and restructure this and most importantly automate this in Github actions

Discussion

The goal of E2E test is to mimic users' behaviour and test the whole app as if a user was using it.
That's why we prefer to search for elements in the UI using something a user can see like visible text or button title, input placeholder, etc.

Implementation

Automating e2e test suite

• Setup bypass authentication for Gitlab (see discussion in the next section) - #755
• Publish snapshot Docker images for engine and sdlc - finos/legend-engine#466, finos/legend-sdlc#334
• Make sure we can authenticate SDLC with private access token - #755
• CHECK Write a simple application depending on sdlc and engine, then write a simple e2e test pointing at that, then turn this whole thing into docker-compose and start it in a pipeline
• Port this simple example to legend-studio and setup the pipeline
• Create a package @finos/legend-internal-testing-e2e
• Target projects from dedicated Gitlab project - https://gitlab.com/legend-integration-testing/basic-m2m
• Setting the environment like this https://github.com/akphi/test-app using a readonly PAT token

Test writing

• Remove the current end-to-end test suite using Cypress to lessen the bloat and use playwright with Jest
• Prefer using input names, button titles, roles, etc. way rather than using TEST_ID but still if it is possible, try to search a button by the text it contains, input by the placeholder a user can see, etc.
• Consider following the Page object pattern - https://www.selenium.dev/documentation/test_practices/encouraged/page_object_models/#:~:text=Page%20Object%20is%20a%20Design,a%20page%20of%20your%20AUT.

Additional context

Setting up Gitlab

When automating the suite, the biggest challenge is to consider how we should setup Gitlab. Here are a few options we considered as well as its pros and cons

• (1) Run the whole thing in Github runner - see what SDLC did, but we can use gitlab-ce: Gitlab image is heavy and takes 7-30 mins to run depending on how much resource available. This won't scale.
• (2) Point at gitlab.com: This is light-weight and probably the most straight-forward --> We go with this
• (3) Mock SDLC server: Use msw: this is not necessarily a bad but mocking in general defeats the purpose of this test suite
• (4) Mock SDLC backend: work on InMemoryAPI: this takes too much time and though it has some benefit in help with SDLC testing in general, it takes time to keep this up with Gitlab.

When we point at gitlab.com we accept the risk carried with it, test can be flaky sometimes. Also, we create/delete project on the flight so we need to be careful with that to accommodate for multiple pipelines running in parallel, the projects/workspaces being created must have some kind of ID/hash and we must do proper cleanups
When we first pointed at gitlab.com we got blocked at authentication step due to rate-limitter (DDoS protection from Gitlab.com) so what we did is to setup authentication bypass using private-access-token (see #755)

More on Playwright vs. Cypress

It differs to cypress in the followings way:

• It use web protocol for event instead of dispatching DOM event
• It's similar to Puppeteer in coding style, it uses async/await unlike cypress's jQuery/fluentAPI style
• It allows us to use our own test runner - i.e. Jest -> this could mean it's probably lighter

Read more at:

• https://blog.logrocket.com/playwright-vs-puppeteer/
• https://blog.checklyhq.com/cypress-vs-selenium-vs-playwright-vs-puppeteer-speed-comparison/
• https://medium.com/uk-hydrographic-office/why-we-chose-playwright-over-cypress-db4770cf5204
• https://medium.com/sparebank1-digital/playwright-vs-cypress-1e127d9157bd
• https://www.testim.io/blog/puppeteer-selenium-playwright-cypress-how-to-choose/

Also read a bit on:

• https://playwright.tech/blog/record-your-browser-tests-with-playwright
• https://playwright.dev/docs/inspector/

Feature request: Show stereotype and tagged value for properties in diagram

🚀 Feature Request

Motivation

In diagram view, right now we only show the tagged values and stereotypes for a class, we don't show these for properties in class view.

Note that because properties can be expressed as edges, we should be careful not to pollute the diagram view. Maybe if it gets too crammed, we might not show the tagged values and stereotypes.

Introduce semantic release for automated releases on npmjs.org

Assuming that the Node logic that part of legend-studio must be consumed from GS as NPM package, we need to define a release process.

By introducing semantic-release, it will be possible to instrument release cycles and automated version updates based on git comment conventions; checkout https://github.com/semantic-release/semantic-release . We used it in the past for other hosted projects and it's quite powerful, but relies on precise git comments.

I'm happy to support this activity and integrate it into GitHub Actions.

[DRAFT] Query builder MVP

Closing as this is meant only as a draft and the issue has been splitter into multiple ones

Feature request: Theming support

Studio follows VSCode look and feel, which is naturally dark-themed. We should also support light-themed. In general, we need to support theming.

Implementation Plan

• Ensure styling consistency: Right now, parts of the apps are still in light modes. We should fix that (#265)
• Have a color registry - Look at VSCode for inspiration, have a color registry (have all color names (editorColor, etc.)); just a small point, we can use library like color to manipulate light/dark/opacity etc. to be consistent in certain area: such as when we want to make a darker/lighter color, we can use the library to do so instead of hard-coding some color.
• Create theme mechanism. There are 2 ways to do this: using CSS variables or inline CSS (CSS in JS): we will pick the former because the latter requires some sort of JS reactivity when we change theme, where for the former, it's natural
• CONSIDER we need to consider the impact of theming on icon color because icon's color is cross-theme

// CSS variables
    --color-header: black;
    --color-background: white;
... and when we switch theme, we set these manually

// or we can use inline CSS
<div className="..." style={{ backgroundColor: colorRegistry.get('background') }} />

Feature request: Add filter support for Relational Mapping

Support a filter lambda as part of relational mapping. This feature gives users the ability to filter through a lambda. The filter will be applied to the source which in this case is a relational mapping.

Example:

Class my::Person
{
   firstName : String[0..1];
   lastName : String[0..1];
}

###Relational
Database my::mainDb
(
   Table PersonTable(firstName VARCHAR(200), lastName VARCHAR(200))
   Filter PersonFilter(PersonTable.firstName = 'Utkarsh')
)

###Mapping
Mapping my::mainMap
(
   my::Person : Relational
   {
      ~filter [my::mainDb] PersonFilter
      firstName : [my::mainDb]PersonTable.firstName,
      lastName : [my::mainDb]PersonTable.lastName
   }
)

Setup grammar roundtrip test - serialization rework

Serialization right now leaks out some inference we do during graph building which makes it not 1-1 with the grammar and the input JSON. We need to dumbify the serializer, this will help with change detection as especially if in the future SDLC starts supporting entities store as grammar text.

Context and Motivation

The basic flow of the app data is like this: protocol JSON -- [deserialize]--> protocol -- [graph build] --> metamodel -- [transform] --> protocol -- [serialize] --> protocol JSON

In [graph build], we have been trying to do some level of inference to fill the gaps in the protocol JSON. This is needed since sometimes if the user input is grammar text, the parser itself cannot derive enough contextual information to fill the protocol JSON, note that this is also the intended behaviour of the parser to keep the grammar and the protocol 1-1. The problem with this is during serialization, we don't want to leak these graph-building inference to the protocol JSON. Otherwise, we will see problems with change detection, especially when we start to get entities from SDLC which are stored as grammar text.

Implementation Plan

Since there is no easy way to keep track of all the issues we might face, the best way to manage this is to file separate issues for each problem found and refer them back to this thread for better context. Each of the issues will need to include a roundtrip test case which:

• Takes grammar text as input
• Converts the grammar text into JSON by calling engine backend. Store this JSON as A
• Build the graph with A and serialize the graph to obtain JSON B
• Make sure A matches B
• Make sure we can compute hash using A because when SDLC server starts storing the graph models in grammar text, when they send us back the entities, those will be used for hash computation

Test runner

• Automate the test in the build pipeline: use maven to build and run the latest version of finos/legend-engine (#293)

Sample Issue tracker

• Basic M2M (#188)
• Embedded relational mapping (#189)

CVE-2020-11023 (Medium) detected in jquery-2.1.4.min.js - autoclosed

CVE-2020-11023 - Medium Severity Vulnerability

Vulnerable Library - jquery-2.1.4.min.js

JavaScript library for DOM operations

Library home page: https://cdnjs.cloudflare.com/ajax/libs/jquery/2.1.4/jquery.min.js

Path to dependency file: legend-studio/node_modules/deep-diff/test/tests.html

Path to vulnerable library: legend-studio/node_modules/deep-diff/test/tests.html

Dependency Hierarchy:

• ❌ jquery-2.1.4.min.js (Vulnerable Library)

Found in HEAD commit: 1de7f5505e98c70f04393f70b9fc52afccdfa463

Found in base branch: master

Vulnerability Details

In jQuery versions greater than or equal to 1.0.3 and before 3.5.0, passing HTML containing elements from untrusted sources - even after sanitizing it - to one of jQuery's DOM manipulation methods (i.e. .html(), .append(), and others) may execute untrusted code. This problem is patched in jQuery 3.5.0.

Publish Date: 2020-04-29

URL: CVE-2020-11023

CVSS 3 Score Details (6.1)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: Required
  • Scope: Changed
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: Low
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-11023

Release Date: 2020-04-29

Fix Resolution: jquery - 3.5.0