microsoft / datashaper Goto Github PK

This may be possible with other verb combinations, but an antijoin operation is useful in some data manipulation situations.

Join T1 to T2 on KEY, deleting entries in T1 that had join values.

DataShaper verbs are essentially transformation functions. Our processing graph provides some dynamic workflow utility, but as a library consumer who wants a coding-first entrypoint, it would be valuable to have the raw verb functions available for my DataFrame-manipulating code.

With the sample datasets (in react/tests/public/*.csv):

• Select the stocks table
• Binarize a numeric column
• Select the "Bin" operation on that numeric column
• Observe render errors in console

Currently, if a verb does not receive a default input, the workflow engine will throw an error. This works most of the time, but in some cases it may be preferable to allow verbs to declare their own named input invariants.

When the steps are topologically sorted, some verbs may end up being runnable at the same time. We will need to update our status-updating callbacks to account for multiple inflight steps executing at once.

e.g. fail-fast, fail-last, custom function

Workflows should specify named outputs that can be referenced downstream. These outputs should have a name, and a specification on which nodes they are derived from:

e.g.

{
  "output": {
     "nodes": { "source": "graph-node", "output': "nodes" },
     "edges": { "source": "graph-node", "output": "edges" } 
}

After upgrading to @data-wrangling-components/core 4.2.1 and @data-wrangling-components/react 3.0.0 I get the following error during build and bundling

 YN0000: [webapp]: ✘ [ERROR] No matching export in "../../node_modules/@data-wrangling-components/core/dist/index.js" for import "ParseType"
➤ YN0000: [webapp]:
➤ YN0000: [webapp]:     ../../node_modules/@data-wrangling-components/react/dist/verbs/Convert/Convert.js:2:9:
➤ YN0000: [webapp]:       2 │ import { ParseType } from '@data-wrangling-components/core';
➤ YN0000: [webapp]:         ╵          ~~~~~~~~~
➤ YN0000: [webapp]:
➤ YN0000: [webapp]: ✘ [ERROR] No matching export in "../../node_modules/@data-wrangling-components/core/dist/index.js" for import "ParseType"
➤ YN0000: [webapp]:
➤ YN0000: [webapp]:     ../../node_modules/@data-wrangling-components/react/dist/verbs/Convert/ConvertDescription.js:2:9:
➤ YN0000: [webapp]:       2 │ import { ParseType } from '@data-wrangling-components/core';
➤ YN0000: [webapp]:         ╵          ~~~~~~~~~

I think the latest version of @dwc/react in npm is depending on features of @dwc/core that have not made it to npm yet.

It appears ParseType is in the latest API of @dwc/core in https://github.com/microsoft/data-wrangling-components/blob/main/javascript/core/docs/core.api.md but ParseType is not present in the released npm module (https://cdn.jsdelivr.net/npm/@data-wrangling-components/[email protected]/dist/types/enums.d.ts).

Steps have if-conditionals. When conditional steps are used as inputs to other steps, and are disabled, we should either do one of the following:

• Use a null table
• Augment the input schema with a conditional "if" field.

this may be python only, but the source table must be explicitly identified

Update Cache implementation to be restorable between runs, redrive from the first uncached point.

In order to enable debugging and complex output scenarios, we can include a pin verb configuration that will flush the verb parquet to disk

e.g.

{
  "verb": "binarize",
  "args": {...},
  "pin": {
     "enabled": true,
     "name": "binarized-data-table",
     "format": "parquet"
  }

Explore using the melt operator in pandas as opposed to our custom impl.

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.melt.html

After upgrading to the latest version of @data-wrangling-components/react, ^3.0.0, I get the following error when running yarn install.

YN0000: [webapp]: ✘ [ERROR] Could not resolve "@essex/themed-components"
➤ YN0000: [webapp]:
➤ YN0000: [webapp]:     ../../node_modules/@data-wrangling-components/react/dist/Steps/ManageSteps/ManageSteps.js:2:30:
➤ YN0000: [webapp]:       2 │ import { DialogConfirm } from '@essex/themed-components';
➤ YN0000: [webapp]:         ╵                               ~~~~~~~~~~~~~~~~~~~~~~~~~~

I believe this is because @essex/themed-components is listed as a devDependency but used as an actual runtime dependency.

It's a bit onerous as a client to acquire a required named input. e.g.

 named_inputs = input.named
    if named_inputs is None:
        raise ValueError(_NAMED_INPUTS_REQUIRED)
    nodes = named_inputs.get("nodes")
    if nodes is None:
        raise ValueError(_NODES_INPUT_REQUIRED)

We can streamline this in datashaper as:

nodes = input.get_required_table("nodes") # throws if not present or configured
# or
nodes = input.get_table("nodes", required=True)

e.g.

{
   "verb": "genid",
  "if": [
      // array to support manual injection
     foo_bar,
     // parsed condition strings
     "!has_field('id')"
   ]
}

new FileWithPath(content, filename, path) will only save files into a path if the path is expressed in filename. However, the path argument is required. We should either respect the path argument or specify that the filename is fully qualified.

e.g.

verb: "select",
args: {
   columns: [
      "*_embedding",
  ]
}

Convert camelCase verb argument fields into snake_case for idiomatic Python.