ml5js / ml5-library Goto Github PK

Simple example using sketch-rnn plus p5.js could be integrated as part of this project, or at least linked to! (cc @hardmaru yet again!)

As mentioned by @genekogan, doing a t-SNE port would be interesting

Could be based on this: https://github.com/karpathy/tsnejs

Related to #26 #24 #21 #15, there are going to be many examples working with images. There are some awkward things now that perhaps we can handle internally and/or with an "image utility" class for the library that handles the images <--> NDArray stuff.

1. Should all functions that receive an image should work with DOM elements (<img>, <video>, <canvas>, p5.Image objects, and pixels arrays?
2. I've noticed our examples only work with certain square dimensions 127x127, 227x227, etc. There are likely performance considerations but ideally the library could handle internally any image resizing necessary.

What else?

How to handle if a property is not passed into generate():

• seed: pick a random sequence of characters from source text as seed?
• temperature: 0.5 ?
• length: 140 ?

@nsthorat and @cvalenzuela have an e-mail thread about this, but adding here to track going forward.

At the moment the LSTM example uses a model trained with this script from deeplearn.js. Eventually we want to train the LSTM in browser, but before that I thought it might be simpler to demonstrate the training with a keras model using this example from my class last year.

@cvalenzuela attempted to use this script to convert the model from my example but the output isn't working just yet.

Shall we do more work to get the keras trained example to be compatible or point students towards train.py that's in deeplearn.js if they want to train their own model?

Did I get this right?

This project recently came out:

http://propelml.org/

Since it uses the Tensorflow C API, it might be interesting to port all training examples to use Javascript. Then we would not need to set up environments in python!

I noticed during the workshop last week that p5.js canvas frame rate drops significantly while running video classification. Here's an example output from this code with console.log(frameRate()) in draw().

44.2477876172749
62.11180126747634
58.139534876795196
60.240963845789416
4.321521175375056
14.684287814305357
59.52380948074774
61.72839506855966
58.479532172979155
59.171597624902525
54.34782608448077
65.78947370859808
60.240963845789416
59.8802395465879
60.97560972744388
7.41839762612301
64.93506491876796
60.975609781548265
56.17977527634279
63.29113926550177
62.499999987267074
56.17977527634279
60.606060613757485
60.97560972744388
57.803468233998
65.78947364561368
56.497175151796085
63.29113926550177
60.606060613757485
58.47953212321371
58.479532172979155
57.14285713715517
63.69426756050796
62.8930817669005
59.171597624902525

imagenet.predict(...) happens asynchronously but I think deeplearn.js is using GPU, which is blocking canvas rendering. Is this a technical limitation, or can we somehow continue updating the canvas while classification job is running?

There is interest in my A2Z class re: image captioning. I thought I would try to create a simple example modeled after the deeplearn.js imagenet that provides captions for a p5.Image as well as real-time captions of a recorded or live video. @cvalenzuela, just wanted to check whether I'd be duplicating anything you have already done? Will leave this thread open to discuss / plan.

@hardmaru, what was the data set you mentioned when you visited ITP that's a nice replacement for digits?

Let's make a datasets page!

I'm porting pix2pix model.
Here is a demo: https://affinelayer.com/pixsrv/

deeplearn.js is Apache License 2.0:
https://github.com/PAIR-code/deeplearnjs/blob/master/LICENSE

p5.js is GNU Lesser General Public License v2.1
https://github.com/processing/p5.js/blob/master/license.txt

Certainly we want something permissive.

Opinions?

As briefly discussed with @shiffman, this project should at some point have a testing framework and CI

and link from here

Hi there. Hopefully, git issues is an acceptable means of communication. I would like to take part in the Google Summer of Code 2018 and it would be great to make some meaningful contributions to this library. I'll need to write a proposal and was wondering if there are features/examples that are being sought?

I can see that Shiffman would like hardmaru's sketch-rnn to potentially be integrated in #11, and I can see that pix2pix is being ported in #43. It may be that pix2pix will be difficult due to the depth of the model, but potentially a shallower one could be trained?

In summary, it would be great for the devs and any users browsing to let me know what would be helpful and I'll do my best to add that to the proposal.

how should we call it? I'm using HighML just to start developing the library and refactor the code.

In examples/fast_style_transfer_mirror/sketch.js, I get 404 on 'assets/models/udnie' and 'assets/img/udnie.jpg'. On my fork I've corrected this to the local files. I can submit a PR if this is correct. But I also noticed you have these files in docs/assets/ so I thought maybe you meant the global ones or that I made a mistake in setup. let me know if you'd like a PR.

This is related to #27 as well as pull requests #69 and #66. I'm reposting a comment I made on @dariusk's amazing recommendations document! Perhaps we can discuss at our meeting on Thursday.

Regarding the p5 examples, I would love to have a discussion about how to manage this. One of the reasons things skewed towards p5 is that I hope to integrate it into my beginner programming classes with p5, but this is probably not helpful for the larger web audience and adds unnecessary extra stuff where simple vanilla JS makes the most sense. I see a few options here:

1. Maintain a separate repo with a p5 + ml5 examples.
2. Maintain two sets of examples here with ml5 itself.
3. Use p5 only where p5 shines (i.e. if we're making heavy use of canvas drawing, webcam capture, loading a CSV, etc.) but otherwise stick with vanilla JS.

Thoughts?

Not an issue, I just wanted to thank you for generating your examples in both ES6 and ES5 (plainJS) formats. Not being used to Typescript, I really appreciate it.

P.S. How do you use TSC to generate the plainJS folder? I would really like deeplearnjs to do it just not sure what to suggest.

• Original project
• deeplearn.js issue on this
• Darknet port to tf

I don't see data.temperature being used at the moment in the two LSTM examples. @cvalenzuela did I miss it somewhere?

How best to handle source text? For this quick example it's a JS file with all of hamlet stored into a variable. To be more flexible / beginner friendly, we should probably switch to using loadStrings()? Or there may be a more appropriate workflow once we have an example where the training uses the same source text in JS?

https://github.com/PAIR-code/deeplearnjs/releases/tag/v0.5.0

https://beta.observablehq.com/

I think deeplearnjs Sessions and Graphs are on their way out. Although no one seems to be saying if this change is coming soon or going to take a while.

Is p5-deeplearn-js ready for the changes? Or is this going to cause a major break? I often make my web pages point to a specific build

<script src="https://unpkg.com/[email protected]/dist/deeplearn.js"> </script>

instead of

<script src="https://unpkg.com/deeplearn"></script>

Any opinions?

I think it would be useful for demonstrating LSTM's to have the two basic examples running with pre-trained models. Maybe we start by doing a large shakespeare corpus from Project Gutenberg?

https://itpnyu.github.io/ml5-js/
please help

I'm thinking about adding a demo about fast-style-transfer.
Fast-style-transfer in Tensorflow
fast-style-transfer demo from deeplearn.js

I am working on some examples to cluster data sets. Here are some algorithms I imagine eventually having in this library:

• kmeans
• principal component analysis (PCA)
• TSNe

I committed a kmeans example (uses random vectors) as a start.

https://github.com/ITPNYU/p5-deeplearn-js/tree/master/examples/clustering/kmeans

Some next steps are:

• Is the algorithm correct? (I whipped it up quickly and needs double-checking)
• Use an actual sample dataset
• Visualize clusters
• Think through clustering ITP and re-factors into library using deeplearn.js for vector math

There are some interesting possibilities with combining clustering algorithms with word vectors.

Continuing off of #63, we might consider providing some friendly error messages if users attempt to call train() before a model is loaded. Not sure how many situations this would apply to.

I get this error while predicting the class:

ml5.js:11350 Uncaught (in promise) Error: No data found for NDArray with data id 2441. Use dl.ENV.math instead of constructing your own NDArrayMath. If you need to construct your own math, make sure this array is allocated after the math construction
    at MathBackendWebGL.throwIfNoData (ml5.js:11350)
    at MathBackendWebGL.readSync (ml5.js:10824)
    at MathBackendWebGL.<anonymous> (ml5.js:10866)
    at step (ml5.js:10698)
    at Object.next (ml5.js:10679)
    at fulfilled (ml5.js:10670)
    at <anonymous>
MathBackendWebGL.throwIfNoData @ ml5.js:11350
MathBackendWebGL.readSync @ ml5.js:10824
(anonymous) @ ml5.js:10866
step @ ml5.js:10698
(anonymous) @ ml5.js:10679
fulfilled @ ml5.js:10670
Promise.then (async)
BackendEngine.scope @ ml5.js:18075
NDArrayMath.scope @ ml5.js:3387
_callee4$ @ ml5.js:26126
tryCatch @ ml5.js:17140
invoke @ ml5.js:17378
prototype.(anonymous function) @ ml5.js:17192
step @ ml5.js:26037
(anonymous) @ ml5.js:26037
(anonymous) @ ml5.js:26037
predict @ ml5.js:26159
predict @ knn_saturday.js:48

the script is:

let video;
let knn;
let buttonA, buttonB, buttonStart;

function preload () {
  // callback, num classes, k-value (just find one neighbor)
  knn = new ml5.KNNImageClassifier(modelLoaded, 2, 1);
}

function setup () {
  createCanvas(320, 240);

  video = createCapture(VIDEO);
  video.attribute('width', 227);
  video.attribute('height', 227);
  video.hide();

  buttonA = createButton('train A');
  buttonB = createButton('train B');
  buttonStart = createButton('start guessing');
  // buttonA.mousePressed(() => {
  //   train(1, 'A');
  // });
  // buttonB.mousePressed(() => {
  //   train(2, 'B');
  // });
  buttonA.mousePressed(train.bind(null, 1, 'A'));
  buttonB.mousePressed(train.bind(null, 2, 'B'));
  buttonStart.mousePressed(predict);
}

function draw () {
  background(0);

  tint(0, 223, 254);
  image(video, 0, 0, width, height);
}

function modelLoaded () {
  console.log('model loaded');
}

function train (category, type) {
  knn.addImage(video.elt, category);
}

function predict () {
  knn.predict(video.elt, gotResult);
}

function gotResult (res) {
  console.log('result: ', res);
}

The KNNImageClassifier has a function to import a model (loadModel(model)). Is there any way to export the trained model so I can import it later?

(Yes, eventually we want to train models in JavaScript in the browser too, but this might be useful now and later.)

https://research.google.com/colaboratory/unregistered.html

Currently the npm package for this library is empty.

Should we start publishing it? Maybe as a pre-release
It would be nice to do:

npm install ml5 

no info about the callback

This is a proposal for the LSTM library from #2.

I'm imagining something like:

const lstm = new LSTMGenerator('/path/to/model/', 'path/to/variables.json');
let txt = lstm.generate(len, seed, temperature);

I am not at all sure about the naming. I am also wondering if variables.json should be integrated into the model files directory and not treated separately?

I am getting error as mentioned in the screen-shot. I need to create prototype similar to teachable machine as there in your KNN image classifier example, before that I tried to build following ImageNet Model but ml.js seems to have some issues. Could you please provide some help. Thanks in advance :)

As of now, the examples depend on models, js files, and other elements via relative paths in this full repo. We should come up with a way to "distribute" stand-alone examples where everything is contained within a single directory.

(Related to this, testing and releasing examples in the p5 web editor could also be useful.)

Teachable Machine: Teachable Machine

The code is here: https://github.com/googlecreativelab/teachable-machine

I'd like to make a very simple version of it in p5.

I'm working on a planjs version of the LSTM port. @cvalenzuela I'm running into this error: Uncaught ReferenceError: NDArrayMathGPU is not defined. I might have missed something in the "unbundling" process? I'm comparing to your MNIST plainjs example, but haven't caught my mistake yet. I'll report back if/when I do, but in the meantime if you have a minute to take a look!

https://github.com/ITPNYU/p5-deeplearn-js/tree/master/examples/plainjs/lstm

Since this project is making use of modules and other ES6 features, not to mention yarn and/or webpack-dev, we should make a getting started developer guide.

Making a note here to discuss style transfer example / library object.

https://github.com/reiinakano/fast-style-transfer-deeplearnjs
https://reiinakano.github.io/fast-style-transfer-deeplearnjs/

I noticed in my new LSTM example that I am able to call generate() before the model is loaded if I'm quick to press the button (or certainly if I call it directly in setup()) Perhaps this is something for later once we're further along in making a "library" but we should offer some sort of optional callback to know when the model is loaded?

In class this week, to compare with my markov chain example, I ran through training an LSTM model using train.py with itp.txt. I've updated the code in this repo to reflect this. The steps I took are:

1. Run train.py
2. Run json_checkpoints_var.py
3. Switch out hamlet.js in example with a new itp.js.

I realize this is tiny tiny data with little training so getting good results isn't really possible. However, I got nonsense results with characters not in the original data set. I'm imagining this has something to do with char_indices and indices_char. Is there a way to auto-generate these during the training process? (They would be different depending on the characters used in the training data, yes?)

Right now the lstm.js file is included in both of my examples 1 and 2. Long-term this is a bigger discussion: in theory all of this code would likely live in either deeplearn.js or a new p5-deeplearn add-on/bridge.

However, for now, should I refactor it out now into libraries?

The lstm() method should allow for options arguments:

• No seed text provided, use random seed text from source text (how many chars?)
• default temperature
• default return length (or should this be required?)

Hi there,

Training a custom LSTM model using the supplied script causes an error when assigning a model name if the folder is not more than two directories deep.

python3 train.py --data_dir data/my_data_folder

Will cause the following error:

Traceback (most recent call last):
  File "train.py", line 171, in <module>
    main()
  File "train.py", line 69, in main
    train(args)
  File "train.py", line 78, in train
    model_name = args.data_dir.split("/")[2]
IndexError: list index out of range

It's a very easy fix, args.data_dir.split("/")[2] should be changed to something like args.data_dir.split("/")[-1]. I'm happy to submit a pull request if that is easier - just let me know.

All the best!

Update instructions to use char-rnn-tensorflow