Coder Social home page Coder Social logo

webgme / webgme-taxonomy Goto Github PK

View Code? Open in Web Editor NEW
0.0 9.0 1.0 14.82 MB

A design studio for creating taxonomies which can then be used to tag various resources.

License: MIT License

JavaScript 59.64% EJS 0.06% CSS 0.33% SCSS 0.45% HTML 0.42% Svelte 7.25% TypeScript 31.49% Dockerfile 0.03% Rust 0.34%

webgme-taxonomy's Introduction

webgme-taxonomy

Installation

First, install the webgme-taxonomy following:

Second, start mongodb locally by running the mongod executable in your mongodb installation (you may need to create a data directory or set --dbpath). Alternatively use docker:

docker run --name webgme-mongo -d -p 27017:27017 mongo:5

Then, run webgme start from the project root to start . Finally, navigate to http://localhost:8080 to start using webgme-taxonomy!

Creating a Taxonomy

The easiest way to create a taxonomy is to simply import a CSV file. Here are the steps:

  • First, create a taxonomy in a spreadsheet where properties have (text), (int), (bool), (enum), (set), etc, appended to the given property name. Each child term/property should be on a line below the parent and indented by 1 cell. Next, export the page as a CSV. An example as a CSV is shown below:
    parentTerm,,,
    ,childTerm,,,
    ,,name (text),
    ,,age (int),
    ,,color (enum),
    ,,,red
    ,,,blue
    ,,,green
    
  • Next, use ./bin/taxonomy-from-csv <path-to-csv> > taxonomy.json to generate a JSON representation that can be imported into the environment.
  • Create a new project in the design studio and import the taxonomy seed as a library.
  • Create a new Taxonomy node in the root node.
  • Enable the SetStateFromJSON plugin for this new node.
  • Run SetStateFromJSON on this new Taxonomy node with the taxonomy.json file generated in the second step.
  • All done!

Development

This project uses rollup to build and bundle the Typescript files (and copy over the client files for the routers). At a high level, client files are independent svelte projects served up by routers, all WebGME entrypoints (e.g. routers, plugins) are bundled with their dependencies (e.g., src/common/ files). Information about a few common use-cases can be found below.

Manually testing

After making changes to any of the server-side source files, run npm run build. For client-side changes, run npm run prepare.

Unit testing

Since rollup bundles all dependencies with the WebGME entrypoints, modules in src/common will not be available in build/common unless building for testing explicitly.

When unit testing non-client code, set NODE_ENV=test before running npm run build as below.

NODE_ENV=test npm run build

As this builds all modules for testing by default, it is recommended to use the TEST_TARGETS environment variable to set the targets to build for testing. TEST_TARGETS does not need to be an exact match and simply builds the files with the given string in the file path.

For example, if we want to test the JSONSchemaExporter, we can build only these files using:

NODE_ENV=test TEST_TARGETS=JSONSchemaExporter npm run build

Adding new WebGME components

Adding WebGME components requires a couple extra steps since this project is using Typescript. Specifically, the WebGME entrypoints for components is in build/ instead of src/ for plugins and routers. Seeds are stored in seeds/ instead of src/seeds. For more details, see below.

Adding new plugins

  1. Create the plugin with webgme new plugin <name>.
  2. Open webgme-setup.json and change the src value in <plugin name> to build. For example, if the plugin is called TestPlugin set src to build/plugins/TestPlugin.
  3. Copy an existing Typescript plugin source file, say OpenTagForm, to the plugin source directory and name it <plugin name>.ts. Rename OpenTagForm to the plugin name and start writing your plugin!

Adding new routers

  1. Create the router with webgme new router <name>.
  2. Open webgme-setup.json and change the src value in <router name> to build. For example, if the router is called TestRouter set src to build/routers/TestRouter.
  3. If the router includes any static assets, such as client files, add an assets field to the entry in the webgme-setup.json file.
  4. Copy an existing Typescript router source file, say Insights, to the router source directory and name it <router name>.ts. Start writing your router!

Adding new seeds

  1. Create the seed with webgme new seed <name>.
  2. Open webgme-setup.json and change src in <seed name> to seeds/<seed name>.
  3. Move the webgmex file from src/seeds/<seed name>/<seed name>.webgmex to seeds/<seed name>/<seed name>.webgmex.

Integrated Tools

There are a few integrated tools in the design studio which are automatically configured using projects in the design studio. However, if you want the following links to work, you will first need to create the example project with the following command:

npm run import -- seeds/test/test.webgmex -p TaxonomyDemo

The main integrated tools are listed below.

  • Search Dashboard: This is a web-based dashboard for viewing data associated with terms in the taxonomy. Currently, only Microsoft Premonition Data Platform is supported to store artifacts but this shouldn't be hard to generalize. (Generalization has been the hopes :).)
  • Taxonomy Term Creator (Form): This is a web-based form for selecting terms given the taxonomy defined in the studio.

Making a Release

The process of making a release is pretty straight-forward.

  1. make a new branch (so CI will run on the new release before it is confirmed) labeled bump-vX.Y.Z.
  2. bump the version in package.json and update the lockfile
  3. commit the version bump with commit message bump to vX.Y.Z
  4. make a PR and merge once all checks have passed
  5. create a tag vX.Y.Z at main and push it
  6. make a GH release (using autogenerated notes) for the new tag. This will trigger a new docker image to be published to DockerHub.

webgme-taxonomy's People

Contributors

brollb avatar dax-westerman avatar romancow avatar pmeijer avatar kecso avatar yogeshvu avatar

Watchers

John McSwain avatar Laszlo Juracz avatar Zsolt Lattmann avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

johnmcswain

webgme-taxonomy's Issues

Tag sets to 'preselect' items in the dashboard

It would be nice if there were an option to send some partial taxonomy to preselect filtering terms (maybe even values) and therefore provide a smaller initial list of items to 'select'.
The motivation behind is that when the user uses the dashboard to select the source of a certain input of a CWL step, it might have some attached taxonomy that clarifies the type of input that is expected. If such taxonomy is available, it would be nice if the list would already reflect that in terms of preselected tags.

Implement PDP storage adapter methods

This consists of a few specific action items:

  • listing artifacts
  • downloading artifacts (a bit hacky atm)
  • dataset/artifact creation
  • appending data to existing datasets

Add support for human-readable tags/terms

Tags files are currently a bit cryptic (in exchange for being less brittle) but it would be useful to have an option that is more brittle and less cryptic, too. Specifically, they currently use UUIDs for nodes and properties but would benefit from having an optional format that uses names instead.

Stream file uploads

Uploading files (appending an observation to a dataset) should stream instead of load the file in memory then upload. It should be as simple as setting body: file.stream() and duplex: 'half' in the call to this._fetch here.

Add search dashboard to Search router

This has a couple steps:

  • Host the search dashboard file on the Search router
  • add endpoint for retrieving the taxonomy form data
  • Create an adapter to connect to the storage backend for retrieving data (eg, PDP)

WebGME Library Repository?

Could this be used as a repository for WebGME libraries? If so, this would be useful to DeepForge among others.

It seems that the biggest thing would be to simply extend the dashboard storage ability to support mongodb/blob

open end point for tag transformation

currently the tag transformation between human<-->guid is sitting behind the auth. Would be good to open so that CLI apps can call this transformer.

Artifacts in search dashboard should be filtered by the taxonomy version used

Artifacts in search dashboard should be filtered by the tag used (This is relevant to #47). The versioning should be standard - perhaps major and minor number tracking breaking (deletions) vs other changes.

When the dashboard is loaded with a given version, say v3.4, the content items should be loaded which used taxonomy w/ version 3.0 - 3.4 (inclusive).

Download tags using human-readable format

The forms should be able to download the tags in human-readable format (probably by default). Maybe it should have the option for the permanent UUID format, too...

AzureActiveDirectory missing

When browsing the data dashboard, we encounter following error on the server side:
image

and following error on the browser side:
image

Show complete taxonomy info

Once the user filtered the data sets enough, it would be nice if we could allow a check complete metadata functionality.
It would bring up the form editor in a read-only mode, allowing the user the make a final look before starting the actual download.

Unable to list artifacts: Unauthorized error

After using the dashboard locally for a bit, I received the error:

Unable_ to list artifacts: Unauthorized

Looking at the output of the debug process, I see the following error:

FetchError: invalid json response body at https://leappremonitiondev.azurewebsites.net/v2/Process/ListProcesses?permission=read reason: Unexpected end of JSON input

Logging out in webgme, then back in and reloading the dashboard fixes the issue. But clearly would be better to not have to do this workaround. If it's some sort of session expiration, a message/visual notification of that would be nice.

tagformatter should resolve the taxonomy version to use

Given that the Taxonomy will evolve, we need a way to specify which taxonomy file to use for the current dataset.
One potential way is to store that information in the index 0 of the dataset(perhaps the tax vocab, webgmex..)

Else we could end up with the following tag not found error

image

Use GUIDs for attributes

attributes should use GUIDs (like tags) so they also don't need a migration for renames (or localization).

Missing upload progress status

Currently, when we are uploading a new file, there is no upload status("x percentage of file upload complete", "x amount of time remaining" for instance) indicator. This is problematic, as the user uploads could consists of large file(maybe in 500mb) and there is no way to know the progress activity of the upload(other than perhaps file uploaded successfully at the end).

Evolve datatype definition in taxonomy

There would be nice to have a dedicated button (for those users that have access to the main taxonomy project), that would take the user to the taxonomy editor in the context of the given datatype (defined by the selected data item on the dashboard).
It would allow a quicker and smoother way to update data type definitions.

Input parameter for behavior of Dahsboard

Either we create multiple endpoint or version for the dashboard to be able to reflect the different tasks it has to cover, or we model somehow certain details about the page. A few items that should be 'configurable':

  • action on items (by default it can be 'download all' or 'download latest', but it might be 'select item' or 'start new project' from
  • title / tooltip texts
  • default click behavior on the item (currently it does not do anything, but it might open up a detailed listing of the data items in the data set or all versions of a workflow, etc.)

instanceof is not object error

If a user is login to the dashboard without any datasets provisioned to the user, the users sees following screen

image

Server log: Uncaught exception:

 2022-09-12T18:20:22.825Z - error: uncaughtException: Right-hand side of 'instanceof' is not an object 

image

Set/list properties for terms

Along with integer, text, enum, etc, properties we should support set properties (corresponding to an "anyOf" list in JSON schema).

Re-use taxonomy information

When the user is about to upload a new data item, she/he might do a search for an older item and try to reuse its metadata/taxonomy information so that she/he do not have to refill every item once again, just change (or maybe even add new tags).
The same scenario can happen when a user just wants to enhance the meta information attached to a data item (do the data does not change but only the taxonomy information)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.