Coder Social home page Coder Social logo

node-realsense's Introduction

Intel® RealSense™ technology with JavaScript API for Linux OS

Build Status

[ License license-image ] license

This project is offering Node.js modules of Intel® RealSense™ technology including Librealsense, Person Tracaking, Object Recognization, and SLAM capabilities. The target is Intel® Joule™ and ZR300 camera on top of Ubuntu 16.04 DT.

These modules have been registered as NPM packages, listed as below.

For example, Person tracking module can be installed by following command.

npm install --save node-person

[Note] The target platform of these modules is Intel® Joule™ and ZR300 camera on top of Ubuntu 16.04 DT, please prepare the hardware devices and setup proper environment before install and use these modules.

Besides the API Specification and NPM package site, JavaScript samples are also good examples demonstrating how to use these modules.

Documents

  • API Specification - This specification describes the JavaScript API provided by the Node.js C++ add-ons of this repository.
  • Getting Started - A guide on how to setup build environment, how to build one or many Node.js C++ add-on(s) residing in this repository and/or how to use, debug & change API definition to the C++ add-on(s).
  • Contribution - A guide on how to contribute code to this repository.
  • BKMs - Listing Best Known Methods (BKMs) for common issues.

License

The MIT License (MIT)

Copyright (c) 2016 Intel Corporation. All rights reserved.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

node-realsense's People

Contributors

donnawudongxia avatar haoyunfeix avatar kenny-y avatar lisa0314 avatar tingshao avatar wanghongjuan avatar zhaoming0 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

lisa0314 rockets-cn kenny-y tingshao haoyunfeix halton xiaolongx minggangw kenchris shening enjoygeek zhaoming0 guozanhua wanghongjuan cigraphics roy2an gitwangshuo rcadecaro isabella232

node-realsense's Issues

[PT] Example code should be unit test

halton:

src/person-tracking/example should be unit test and move into src/person-tracking/test

tingshao:

UT cases have been added for newly added features under src/person-tracking/test and those files under example also seem useful now, can we keep these old example files

Support file input in Instance.start()

Provides a way to catch data from file instead of the camera.
This feature could enable some scenario without camera, like Travis CI, APIs automation check....
Usage could be: instance.start('/path/to/test.file').then(function() {console.log('Start camera');});

[SLAM] Support control interfaces - reset

It seems the reset interface, should restore all the configurations.
[OPEN] Should its target state is "ready", so need to invoke the "start" interface later.
Or, we combine all the steps.

  1. stop SLAM
  2. restore all the configurations
  3. start SLAM

Is there still restrictions of dir structure

Q:

This was found in Beta1, but changed in Beta2, not sure RC(s).

We need to do a test and see if there is problems, if any, Matt, arch omf MW could be a contact.

A:

do you mean below files?

PT files under /usr/share/librealsense/pt/data/person_tracking/*
OR files under /usr/share/librealsense/object_recognition/*.classifier
Is this still valid, we're using the debian package, still a concern?

Unable to locate package realsense-meta-dev

tingshao:

When install beta3 release, using "sudo tools/install_realsense_pkgs.sh beta3", this error occurs, seems that package is not found.

halton:

Turn out the apt server is broken now, if we try sudo apt-get librealsense-persontracking-dev, the dependency is broken.

Dictionary arguments without required members must be optional in bikeshed parsing

This issue is found when using bikeshed to process our idls. We may need to change our idl.

[
Constructor,
Constructor(optional InstanceOptions options) //Here, optional is a must
]
interface Instance {
    Promise<void> setInstanceOptions(optional InstanceOptions instanceOptions); //Here, optional is also a must
...

The error msg:
FATAL ERROR: IDL ERROR LINE: 15 - Dictionary argument "instanceOptions" without required members must be marked optional

For setInstanceOptions, the parameter is obviously not optional. But it is semantic consideration, seems
spec definition cares most about syntax definition.

Please refer to the follow guy's experience, Finally, he used documentation to explain this, and optional is added:

w3c/mediacapture-main#366
w3c/mediacapture-main@5b1fe0e

As discussed with Halton and lisha, the widl is better not to be affected by documentation work recently.

ppa-installed binaries are not compatible with most recent internal realsense_samples repo, what's our strategy?

It's not possible to completely follow native realsense_samples to write Node.js add-on due the the inconsistent version issue (rolling back to earlier version might hit bugs that they have already addressed).

I believe we need to work out a strategy: e.g. stick with a working impl for pre-alpha, and then migrate to a consistent version (say RC1) in the future?

e.g. compilation error as following:

object_recognition_utils.hpp:227:79: error: ‘struct rs::core::video_module_interface::supported_image_stream_config’ has no member named ‘size’
             if(supported_config.image_streams_configs[(int)rs::stream::color].size.width == 640

The installed RSSDK version is

#define SDK_VERSION_STRING static volatile char version_id[] = "VERSION: 0.2.10.0";

The definition in video_module_control_interface.h of that version is:

            struct supported_image_stream_config
            {
                sizeI32        min_size;           /* minimum size */
                sizeI32        ideal_size;         /* maximum size */
                float          minimal_frame_rate; /* minimal operetional frame rate */
                float          ideal_frame_rate;   /* frame rate for ideal operation */
                sample_flags   flags;              /* optional samples flags */
                preset_type    preset;             /* stream preset */
                bool           is_enabled;         /* is the indexed stream is enabled, defaults to 0 = false */
            };

halton:
the beta3 script is now using SDK version as 0.4.0. Is this issue still valid? BTW, the internal APT server is broken now #51 is tracking on this.

[PT] Lose precise of gestureStartTimeStamp 

interface PointingInfo {
    readonly attribute PointingData3D worldPointingData;
    readonly attribute PointingData2D colorPointingData;
    readonly attribute long confidence;
    readonly attribute long gestureStartTimeStamp;
};

According to native API, it should be readonly attribute long long gestureStartTimeStamp;.

When converting to JavaScript number, please follow the https://heycam.github.io/webidl/#es-long-long.

There is no PixelFormat member in ImageStreamConfig{}, does this mean that we don't allow user to specify stream format?

https://github.com/01org/node-realsense/blob/master/src/idl/common/realsense-camera.widl#L87

dictionary ImageStreamConfig {
  long width;
  long height;
  float frameRate;
  Intrinsics intrinsics;
  Extrinsics extrinsics;
  boolean isEnabled;
};

If RealSense MiddleWares can work on a set of pixel formats, we might need to enable user to choose from one of them to adapt their requirement to image format (for display/save/etc.)

kenny-y:
DonnaWuDongxia Another one that is asking for your input... thanks 👍

halton:
How is the native side?

[SLAM] Support for Camera Config

To support setCameraOptions(), getCameraOptions() API.
We need to use the common CameraOptions definition.
Test cases: set first, get back, compare the results.

[SLAM]Should get reject promise when invoke loadOccupancyMap()/loadRelocalizationMap() without parameter

According spec:

Promise<void> loadOccupancyMap(String mapFileName);
Promise<void> loadRelocalizationMap(String mapFileName);

Method should have a mandatory parameter input, when missing it, method should return reject promise

Test Code:

let argv = '/tmp/loadOccupancyMap.map';      
     slamAddon.createInstance().then((Instance) => {
        slamInstance = Instance;
        return slamInstance.start();
      }).then(() => {
        return slamInstance.saveOccupancyMap(argv);
      }).then(() => {
        return slamInstance.loadOccupancyMap();
      }).then((Data) => {
        reject('Promise should reject, but got resolved');
      }).catch((err) => {
        resolve();
      });

Result:
Error: the string "Promise should reject, but got resolved" was thrown, throw an Error :)

Change of ArrayBuffer impl requires change of node-realsense code that are using it

When WIDL-NAN 0.1.16 (estimated ver) is published (intel/widl-nan#92), ArrayBuffer type in IDL definition will be implemented via v8::ArrayBuffer (was: Node.js' Buffer).

With this change, we can explicitly specify whether an ArrayBuffer can actually own a raw buffer or not. The advantage is: if an ArrayBuffer doesn't own a raw buffer, it won't delete the buffer when it's being destroyed, which brings up the ability to reuse the same raw buffer and to avoid large amount of buffer copying.

kenny:

Mark this as a bug because code won't compile after this change.

[SLAM] Throws TypeError when invoke getOccupancyMap() method

Test Code:

        slamAddon.createInstance().then((Instance) => {
          slamInstance = Instance;
          return slamInstance.start();
        }).then(() => {
          if (options == null) {
            return slamInstance.getOccupancyMap();
          }else{
            return slamInstance.getOccupancyMap(options);
          }   
        }).then((Data) => {
         ......
         ......
          resolve();
        }).catch((err) => {
          reject(err);
        });

Result:
TypeError: Cannot convert undefined or null to object

[SLAM] saveRelocalizationMap() should return string but got a number

According spec:
Promise<String> saveRelocalizationMap(optional String mapFileName);

Should got string but return a number

Test Code:

     let options = '/tmp/saveRelocalizationMap.map';
        slamAddon.createInstance().then((Instance) => {
          slamInstance = Instance;
          return slamInstance.start();
        }).then(() => {
            if (options == null ) { 
              return slamInstance.saveRelocalizationMap();
            }else{
              return slamInstance.saveRelocalizationMap(options);
            }   
        }).then((Data) => {
          assert.equal(typeof(Data), 'string');
          resolve();
        }).catch((err) => {
          reject(err);
        });

Result:

AssertionError: 'number' == 'string'

[Sample][Multi] Implement the JavaScript version of or_pt_tutorial_1

or_pt_tutorial_1

Description:

This console app illustrates the use of libRealsense, OR and PT libraries using the ZR300 camera identify objects and persons. The name of the object along with the confidence value will be printed on the console. Person information like person id and the 2D box coordinates will be printed on the console.

[SLAM] Method getRelocalizationPose returns a number not array

According to spec Promise<sequence<float>> getRelocalizationPose();
This method should get array but got a number

Test Code:


      slamAddon.createInstance().then((Instance) => {
        slamInstance = Instance;
        return slamInstance.start();
      }).then(() => {
          return slamInstance.getRelocalizationPose();
      }).then((Data) => {
        assert.equal(typeof(Data), 'object');
        assert.equal(typeof(Data[0], 'number'));
        resolve();
      }).catch((err) => {
        reject(err);
      });

Result:

      AssertionError: 'number' == 'object'
      + expected - actual

[SLAM] Automation test with playback

SLAM need to support automated testing w/o Realsense camera plugged in. The pre-condition is to use CameraOption in common, which is already used in OR module.

[SLAM] Support control interfaces - resume & pause

To support resume(), puase() API.
Test cases:

  1. pause w/o start, or not in TRACKING state
    Expection: error, haven't start
  2. resume w/o pause, or not in PAUSED state
    Expection: error, haven't pause
  3. start --> pause
    Expection: SLAM stop working
  4. start --> pause --> resume
    Expection: SLAM continually works
  5. start --> pause --> resume --> pause (aims to check pause&resume interfaces are all restorable)
    Expection: SLAM stop working

[OR] OR instance got undefined after several times of start/stop on joule board

OR instance got undefined after several times of start/stop on joule board, this issue already commented

const addon = require('bindings')('realsense_object_recognition');

const EventEmitter = require('events').EventEmitter;
function inherits(target, source) {
  // eslint-disable-next-line guard-for-in
  for (let k in source.prototype) {
    target.prototype[k] = source.prototype[k];
  }
}
inherits(addon.ObjectRecognizer, EventEmitter);

let or = undefined;

const orOptionsPositive = { 
  maxReturnObjectCount: [1, 2, 3], 
  computeEngine: ['CPU'],
  enableObjectCenterEstimation: [false],
  mode: ['single-recognition', 'tracking', 'localization'],
  confidenceThreshold: [0.15, 0.75],
  enableSegmentation: [true, false],
};

function testPositive(orOptions) {
  let key;
  let mode = 'single-recognition';
  // eslint-disable-next-line guard-for-in
  for (let k in orOptions) {
    key = k;
    if(k === 'mode') {
      mode = orOptions[k];
    }   
  }
describe('Object Recognition Test Suite - or - Positive - ' + key, function() {
    after(function() {
      this.timeout(15000);
      return new Promise((resolve, reject) => {
        or.stop().then(function() {
          console.log('Stop Camera');
          resolve();
        }).catch(() => {
          reject();
        });
      }); 
    }); 

    it('Method start() - Positive - ' + mode + ' - ' + key + ' - ' + orOptions[key], function() {
      this.timeout(60000);
      return new Promise((resolve, reject) => {
        addon.createObjectRecognizer().then((objectRecognizer) => {
          or = objectRecognizer;
          return or.setObjectRecognitionOptions(orOptions);
        }).then(() => {
          return or.start();
        }).then(() => {
          resolve();
        }).catch((e) => {
          console.log('error message:' + e); 
          reject(e);
        });
      });
    });
  });
}

// eslint-disable-next-line guard-for-in
for (let k in orOptionsPositive) {
  for (let i = 0; i < orOptionsPositive[k].length; i++) {
    let orOptions = {};
    orOptions[k] = orOptionsPositive[k][i];
    testPositive(orOptions);
  }
}

Result:

 Object Recognition Test Suite - or - Positive - maxReturnObjectCount
    ✓ Method start() - Positive - single-recognition - maxReturnObjectCount - 1 (4131ms)
Stop Camera

  Object Recognition Test Suite - or - Positive - maxReturnObjectCount
    ✓ Method start() - Positive - single-recognition - maxReturnObjectCount - 2 (2492ms)
Stop Camera

  Object Recognition Test Suite - or - Positive - maxReturnObjectCount
    ✓ Method start() - Positive - single-recognition - maxReturnObjectCount - 3 (2499ms)
Stop Camera

  Object Recognition Test Suite - or - Positive - computeEngine
    ✓ Method start() - Positive - single-recognition - computeEngine - CPU (2491ms)
Stop Camera

  Object Recognition Test Suite - or - Positive - enableObjectCenterEstimation
    ✓ Method start() - Positive - single-recognition - enableObjectCenterEstimation - false (2514ms)
Stop Camera

  Object Recognition Test Suite - or - Positive - mode
    ✓ Method start() - Positive - single-recognition - mode - single-recognition (2498ms)
Stop Camera

  Object Recognition Test Suite - or - Positive - mode
    ✓ Method start() - Positive - tracking - mode - tracking (2239ms)
Stop Camera

  Object Recognition Test Suite - or - Positive - mode
    ✓ Method start() - Positive - localization - mode - localization (9439ms)
Stop Camera

  Object Recognition Test Suite - or - Positive - confidenceThreshold
    ✓ Method start() - Positive - single-recognition - confidenceThreshold - 0.15 (4539ms)
Stop Camera

  Object Recognition Test Suite - or - Positive - confidenceThreshold
error message:undefined
    1) Method start() - Positive - single-recognition - confidenceThreshold - 0.75
Stop Camera

  Object Recognition Test Suite - or - Positive - enableSegmentation
error message:undefined
    2) Method start() - Positive - single-recognition - enableSegmentation - true
Stop Camera

  Object Recognition Test Suite - or - Positive - enableSegmentation
error message:undefined
    3) Method start() - Positive - single-recognition - enableSegmentation - false
Stop Camera


  9 passing (48s)
  3 failing

  1) Object Recognition Test Suite - or - Positive - confidenceThreshold Method start() - Positive - single-recognition - confidenceThreshold - 0.75:
     Error: Promise rejected with no or falsy reason
  

  2) Object Recognition Test Suite - or - Positive - enableSegmentation Method start() - Positive - single-recognition - enableSegmentation - true:
     Error: Promise rejected with no or falsy reason
  

  3) Object Recognition Test Suite - or - Positive - enableSegmentation Method start() - Positive - single-recognition - enableSegmentation - false:
     Error: Promise rejected with no or falsy reason

Whether 'frameprocessed' and 'persontracked' event should be merged.

tingshao:

This topic is triggered when discussing how to embed the rgb framedata in result.
‘frameprocessed’ is currently send per frame no matter whether there is actual person tracked.
‘persontracked’ is only sent when some person is tracked.
The event data of these two events are the same format, with exception that the data of ‘frameprocessed’ can be empty.

Choice 1: use options to config the results, using only one event:
add options in config to choose whether rgb data is needed and whether the event should be per frame triggered.

Choice2: this is PT’s current implementation: use two events, to let user dynamically add/remove event listener to make addon aware which event to send. While whether rgb data is carried is also configed in options.

Which choice do you prefer? or new choices?

huningxin:

I prefer to multiple events, it is straightforward for use case and flexible to compose.

The question here is it seems redundant to carry same data in multiple kinds of events, correct?

Could we introduce corresponding getters to query the data? The the events could be just notification for data ready instead of carrying any actual data. It would also benefit for performance due to less objects constructions for different events.

For example:

An app wants to display each frame and to be notified by person tracked:

> var frameData = new Unit8Array(size);
> instance.on('frameprocessed', function() {
>    instance.getFrameData(frameData);
>    ui.displayFrame(frameData);
> });
> 
> instance.on('persontracked', function() {
>   var trackedPersons = instance.getPersonData();
>   ui.displayPersons(trackedPersons);
> });

These getters could be synchronized as there would be no I/O operations. Implementation could make the data ready after emitting an event. So there will only be a object construction or a memory copy.

Here, getFrameData accepts an existing typed array object and fill the contents instead of returning a new object. It improves the performance by less object constructions (memory chunk allocation and free). getPersonData doesn't save much if using the filling pattern.

I am thinking whether we need to define persondetected and personnotdetected events, as PT has detecting and tracking states. If we have them, the pt_tutorial_1 would only listen to the two events instead of listen to the frameprocessed event. I will open another issue for discussion.

DonnaWuDongxia:
In this solution, do we have the requirement to match the frameData with the persontracked data? The handler is developer owned, we can't garantee that sync API can keep the allience as we always keep the latest sample and processed result in native world, right

huningxin:
Module can attach time stamp to data so JS code can match them if needed.

tingshao:
is it ok to getResult() which returns both framedata and its associated persontracking result?

huningxin:
For some use cases, like console samples, it doesn't need the frame data.

[Sample][Multi] Implement the JavaScript version of slam_or_pt_tutorial_1

Description:

This console app illustrates the use of libRealsense, SLAM, OR and PT libraries using the ZR300 camera to print out the current camera module position and , identified objects and identified persons. The name of the object along with the confidence value will be printed on the console for identified objects. Person information like person id and the 2D box coordinates will be printed on the console.

[PT] [PnP]CPU utilization of sample PT bigger >15% than native

Test with latest realsense build: v0.8.0, detail info as below table.

SampleCategoryKPIsv0.8.0
pt_tutorial_1 node resultpt_tutorial_1 native resultNode VS. Native
PTCPUCPU_Utilization_No_Person (%)10.18.8214.51%
CPU_Utilization_One_Person_Tracking (%)23.8220.1917.98%

getFrameData() is not Promisified in PT

kenny-y:
When I was learning how to use the class FrameDataAssembler, I found this

interface PersonTracker {
    // ...
    FrameInfo getFrameData();

And one more thing: shouldn't it be named getFrameInfo() since we had adopted the review result to rename the interface from FrameData to FrameInfo ?

tingshao:
The getFrameData is synchronous is discussed in #54

I often got conflict naming suggestions, It seems we need to make a final decision. We did agreed with the review result to use Info instead of Data. I also got review comments to be aligned with native API side, like the PR 188 (I used PointingData instead of PointingInfo). Seems we need a final decision.

kenny-y:
I agree that we should consistent naming across all 3 add-ons, and also: the sync-vs-async style need to be consistent across all 3 add-ons.

huningxin:
getFrameData would be synchronous API as 1) the getFrameData implementation should just memory operations other than any I/O; 2) simplify the consumer of this API.

For naming, I support the native API alignment, so use Data instead of Info. BTW, the Info sounds like some meta info besides the real data.

tingshao:
Let me explain what PT implements the frameData, in the worker thread, each captured frame was copied and saved for retrieval by main thread. So, when getFrameData was called in main thread, the already saved result would be returned with no further work. So, if we use promise to encapsulate it, it seems a bit strange. I also noticed the frame data retrieval way of SLAM is different from both PT and OR, so, it seems we already run into different ways for this API. Shall we keep this difference in public release, otherwise, we may need reconsider the implementation.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.