Coder Social home page Coder Social logo

cwise's Introduction

cwise

This library can be used to generate cache efficient map/reduce operations for ndarrays.

build status

Examples

For brevity, we will assume the following precedes each example:

//Import libraries
var cwise = require("cwise")
  , ndarray = require("ndarray")

Adding two arrays

The array equivalent of +=:

//Create operation
var addeq = cwise({
    args: ["array", "array"],
    body: function(a, b) {
      a += b
    }
  })

//Create two 2D arrays
var X = ndarray(new Float32Array(128*128), [128,128])
var Y = ndarray(new Float32Array(128*128), [128,128])

//Add them together
addeq(X, Y)

Formally, you can think of addeq(X,Y) as being something like the following for-loop, except optimized with respect to the dimension and order of X and Y:

for(var i=0; i<X.shape[0]; ++i) {
  for(var j=0; j<X.shape[1]; ++j) {
    X.set(i,j, X.get(i,j) + Y.get(i,j))
  }
}

Multiply an array with a scalar

var muls = cwise({
  args: ["array", "scalar"],
  body: function(a, s) {
    a *= s
  }
})

//Example usage:
muls(array, 2.0)

Initialize an array with a grid with the first index

var mgrid = cwise({
  args: ["index", "array"],
  body: function(i, a) {
    a = i[0]
  }
})

//Example usage:
var X = mgrid(ndarray(new Float32Array(128)))

Compute 2D vector norms using blocks

var norm2D = cwise({
  args: ["array", {blockIndices: -1}],
  body: function(o, i) {
    o = Math.sqrt(i[0]*i[0] + i[1]*i[1])
  }
})

//Example usage:
var o = ndarray([0, 0, 0], [3])
norm2D(o, ndarray([1, 2, 3, 4, 5, 6], [3,2]))
// o.data == [ 2.23606797749979, 5, 7.810249675906654 ]

Note that in the above, i is not an actual Array, the indexing notation is just syntactic sugar.

Apply a stencil to an array

var laplacian = cwise({
  args:["array", "array", {offset:[0,1], array:1}, {offset:[0,-1], array:1}, {offset:[1,0], array:1}, {offset:[-1,0], array:1}],
  body:function(a, c, n, s, e, w) {
    a = 0.25 * (n + s + e + w) - c
  }
})

laplacian(next, prev)

Compute the sum of all the elements in an array

var sum = cwise({
  args: ["array"],
  pre: function() {
    this.sum = 0
  },
  body: function(a) {
    this.sum += a
  },
  post: function() {
    return this.sum
  }
})
  
//Usage:
s = sum(array)

Note that variables stored in this are common to all three code blocks. Also note that one should not treat this as an actual object (for example, one should not attempt to return this).

Check if any element is set

var any = cwise({
  args: ["array"],
  body: function(a) {
    if(a) {
      return true
    }
  },
  post: function() {
    return false
  }
})

//Usage
if(any(array)) {
  // ...
}

Compute the index of the maximum element of an array:

var argmin = cwise({
  args: ["index", "array"],
  pre: function(index) {
    this.min_v = Number.POSITIVE_INFINITY
    this.min_index = index.slice(0)
  },
  body: function(index, a) {
    if(a < this.min_v) {
      this.min_v = a
      for(var i=0; i<index.length; ++i) {
        this.min_index[i] = index[i]
      }
    }
  },
  post: function() {
    return this.min_index
  }
})

//Usage:
argmin(X)

Install

Install using npm:

npm install cwise

API

require("cwise")(user_args)

To use the library, you pass it an object with the following fields:

  • args: (Required) An array describing the type of the arguments passed to the body. These may be one of the following:
    • "array": An ndarray-type argument
    • "scalar": A globally broadcasted scalar argument
    • "index": (Hidden) An array representing the current index of the element being processed. Initially [0,0,...] in the pre block and set to some undefined value in the post block.
    • "shape": (Hidden) An array representing the shape of the arrays being processed
    • An object representing a "blocked" array (for example a colour image, an array of matrices, etc.):
      • blockIndices The number of indices (from the front of the array shape) to expose in the body (rather than iterating over them). Negative integers take indices from the back of the array shape.
    • (Hidden) An object containing two properties representing an offset pointer from an array argument. Note that cwise does not implement any boundary conditions.
      • offset An array representing the relative offset of the object
      • array The index of an array parameter
  • pre: A function to be executed before starting the loop
  • body: (Required) A function that gets applied to each element of the input arrays
  • post: Executed when loop completes
  • printCode: If this flag is set, then log all generated code
  • blockSize: The size of a block (default 32)
  • funcName: The name to give to the generated procedure for debugging/profiling purposes. (Default is body.name||"cwise")

The result is a procedure that you can call which executes these methods along the following lines:

function(a0, a1, ...) {
  pre()
  for(var i=0; i<a0.shape[0]; ++i) {
    for(var j=0; j<a0.shape[1]; ++j) {
      ...
      
          body(a0[i,j,...], a1[i,j,...], ... )
    }
  }
  post()
}

Notes

  • To pass variables between the pre/body/post, use this.*
  • The order in which variables get visited depends on the stride ordering if the input arrays. In general it is not safe to assume that elements get visited (co)lexicographically.
  • If no return statement is specified, the first ndarray argument is returned
  • All input arrays must have the same shape. If not, then the library will throw an error

As a browserify transform

If bundle size is an issue for you, it is possible to use cwise as a browserify transform, thus avoiding the potentially large parser dependencies. To do this, add the following lines to your package.json:

//Contents of package.json
{
    // ...

    "browserify": {
      "transform": [ "cwise" ]
    }

    // ...
}

Then when you use the module with browserify, only the cwise-compile submodule will get loaded into your script instead of all of esprima. Note that this step is optional and the library will still work in the browser even if you don't use a transform.

FAQ

Is it fast?

Yes

How does it work?

You can think of cwise as a type of macro language on top of JavaScript. Internally, cwise uses node-falafel to parse the functions you give it and sanitize their arguments. At run time, code for each array operation is generated lazily depending on the ordering and stride of the input arrays so that you get optimal cache performance. These compiled functions are then memoized for future calls to the same function. As a result, you should reuse array operations as much as possible to avoid wasting time and memory regenerating common functions.

License

(c) 2013 Mikola Lysenko. MIT License

cwise's People

Contributors

etpinard avatar jaspervdg avatar mikolalysenko avatar rreusser avatar tgabi333 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cwise's Issues

Broadcast support

Right now, the library throws Uncaught Error: cwise: Arrays do not all have the same shape! when I try to add, for example, an array with shape [1, 1, 50] with another one with shape [20, 30, 50].

Is it possible to implement broadcasting in cwise?

CVE in dependencies

Old version of static-module is used which is using old version of quote-stream which is using old version of minimist with a CVE

potential security vulnerability via an outdated version of [email protected] > [email protected]

Hi,
this is perhaps not the place to report it, please feel free to close the issue, but the version of static-module specified in the package.json is affected by this security vulnerability:
https://nodesecurity.io/advisories/548
[email protected] > [email protected] > [email protected]

I have tried to update static-module to version ^2.0.0 which fixes the issue:
browserify/static-module#34

...but the tests are failing. I do no know this code enough to fix it, any help is welcome.

This is part of making plotly.js pass security tests:
plotly/plotly.js#2386

Would also be good to have a security badge with:
snyk: https://github.com/snyk/snyk#badge
or
nsp: see https://github.com/dwyl/repo-badges
Thx
Alex

cwise docs out of data?

In some of the examples, a cwise function can be called these two ways:

let somearray = cwiseFunc(arr1, arr2)
cwiseFunc(somearray, arr1, arr2)

I could only get the second to work, so I wonder if the docs aren't up to date in this and perhaps other ways. Also, is this expected to work without the browserify transform or is that just to get size down? I'm in webpack.

In the mean time, any links to repos making use of cwise would get great to check out.

bug with Safari ?

I'm trying to track down a nasty bug with Safari which I think is related to cwise. The trouble is, it only crops up when uglify is set to compress mode. And the Safari debugger sucks, so there is a chance it's giving me a totally wrong line number in the minified bundle (i.e. might lie in a different module). Gonna try to get source maps working to figure this out.

Can be reproduced with:
http://mattdesl.github.io/frontend-npm-goodies/dist/webgl-bunny.html

global cwise transform fails if regl is a dependency

A weird interaction, but figured I'd ask since it lives in the same ecosystem. A global cwise transform is useful since most cwise-based modules are not cwise-transformed. However, browserify -g cwise fails if regl is a dependency.

To reproduce:

$ echo "require('regl')" > index.js
$ npm i cwise
$ browserify -g cwise index.js

The resulting output is:

SyntaxError: Unexpected token (2:22) while parsing file: /Users/rreusser/test8/node_modules/regl/lib/constants/arraytypes.json
    at Parser.pp.raise (/Users/rreusser/test8/node_modules/cwise/node_modules/static-module/node_modules/falafel/node_modules/acorn/dist/acorn.js:1745:13)
    at Parser.pp.unexpected (/Users/rreusser/test8/node_modules/cwise/node_modules/static-module/node_modules/falafel/node_modules/acorn/dist/acorn.js:2264:8)
    at Parser.pp.semicolon (/Users/rreusser/test8/node_modules/cwise/node_modules/static-module/node_modules/falafel/node_modules/acorn/dist/acorn.js:2243:59)
    at Parser.pp.parseExpressionStatement (/Users/rreusser/test8/node_modules/cwise/node_modules/static-module/node_modules/falafel/node_modules/acorn/dist/acorn.js:2677:8)
    at Parser.pp.parseStatement (/Users/rreusser/test8/node_modules/cwise/node_modules/static-module/node_modules/falafel/node_modules/acorn/dist/acorn.js:2462:160)
    at Parser.pp.parseBlock (/Users/rreusser/test8/node_modules/cwise/node_modules/static-module/node_modules/falafel/node_modules/acorn/dist/acorn.js:2692:21)
    at Parser.pp.parseStatement (/Users/rreusser/test8/node_modules/cwise/node_modules/static-module/node_modules/falafel/node_modules/acorn/dist/acorn.js:2443:19)
    at Parser.pp.parseTopLevel (/Users/rreusser/test8/node_modules/cwise/node_modules/static-module/node_modules/falafel/node_modules/acorn/dist/acorn.js:2379:21)
    at parse (/Users/rreusser/test8/node_modules/cwise/node_modules/static-module/node_modules/falafel/node_modules/acorn/dist/acorn.js:101:12)
    at module.exports (/Users/rreusser/test8/node_modules/cwise/node_modules/static-module/node_modules/falafel/index.js:22:15)

It's just a small json file, so it's not immediately apparent what there is to dislike about it. I'm wondering if perhaps it's just uglify-js that needs a 2.6 -> 2.7 update or something of the sort (acorn maybe?).

Pass arbitrary data to pre/body/post

It's possible to pass a scalar to the body function, but is there any way of passing arbitrary data to the functions? (arg type=object?) I've encountered a couple cases where this would be convenient and, I believe, valid:

  • Passing a function to cwise. Think of an image processing function. Unless body implements the function or the function is defined in the containing scope (which isn't always possible if you don't know the function ahead of time, right?), it's not immediately clear how to pass this.
  • More questionable uses. I'm currently trying to write a axis-wise function that computes the mean across a number of axes. This can be accomplished with transpose + blockIndices + map-reduce, but it's not clear how to pass the data in/out.

Long story short, here are the two possibilities:

  • add an object type for passing arbitrary stuff that's useful
  • this isn't really what cwise is for and would defeat the purpose of the useful optimizations it provides

Tensor contraction?

Following up here on my ill-advised rreusser/ndarray-awise-prototype#1 since this is starting to seem more like a extension or fork of cwise than a use of cwise.

Was trying to figure out how to express reduce operations in a general manner but fell way short. Your response "there are probably better ways to do this but hey, more than one way to skin a cat" is way too kind. 😝

I'm currently thinking about what it'd mean to reorder dimensions within cwise. Expressing this seems the most immediately challenging part while the rest is an exercise for the implementer…

What about actually allowing index notation? For example:

contraction over one index: In other words, matrix multiplication. The k loop goes on the inside with its pre/post just outside it. The scope of this would be only over a k loop, but you could use c directly. Would have to think about a syntax that would actually allow gemm-style block-wise multiplications. Represented as:

{
  args: [ 'array(i,j,k)', 'array(k,l,m)', 'array(i,j,l,m)'],
  contractIndices: ['k'],
  beforeContract: function(a, b, c) { c = 0 },
  body: function(a, b, c) { c += a * b },
  afterContract: function(a, b, c) { }
}

contraction over two indices: Similar. beforeContract and afterContract go just outside the innermost contraction loops:

{
  args: [ 'array(i,j,k)', 'array(j,k,l)', 'array(i,l)' ],
  contractIndices: ['j','k'],
  beforeContract: function(a, b, c) { c = 0 },
  body: function(a, b, c) { c += a * b },
  afterContract: function(a, b, c) { }
}

Average over two of three dimensions: Here the notation gets unwieldy and it starts to feel like it's all falling apart.

{
  args: [ 'array(j)', 'array(i,j,k)', 'size' ],
  contractIndices: ['i', 'k'],
  pre: function( size ) { this.factor = 1 / size.i / size.k }
  beforeContract: function(a, b) { a = 0 },
  body: function(a, b) { a += b },
  afterContract: function(a, b) { b *= this.factor }
}

I think this is logically consistent and can be optimized within reason (or at least reduces to cwise if unused). It seems possible but probably too invasive to work into cwise which seems more oriented toward image processing. Features like blockIndices and offset are great, but it might add factorial complexity to get all combinations of cases to play well together.

It sounded like you've thought about this before. I think what I've described is a thing that could exist; I just don't know if it's a thing that should exist…

Then again, I might be aiming for something more complicated than the benefit derived from it…

matrix multiplication

Hi,

Could you share an example for computing matrix multiplication of 2 arrays using cwise (something like numpy.dot)?

Best regards (and thanks for the great work!)

Cumulative sum with cwise?

If I understand it correctly, to compute a cumulative sum along a dimension, I would need support for #9, right?

Can't access RGB triplet with blockIndices

I'm trying to process an image and need to access the full RGB triplet on each loop.

var myop = cwise({
  args: [{ blockIndices: -1 }, 'scalar'],
  body: function(rgb, s) {
      console.log(rgb[0], rgb[1], rgb[2]); // <= this works
      console.log(rgb); // <= this doesn't
    }
  }
});

The error I get is Uncaught ReferenceError: _inline_1_arg0_ is not defined.

ping @jaspervdg

Summation over single axis

I'm trying to sum the ndarray from get-pixels over a single axis. I can only seem to either get it to just flatten the array or sum everything. Could you help me sum just a particular axis?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.