vast-community-hub / tensorflow-vast

TensorFlow binding library for VA Smalltalk

TensorFlow for VAST Platform (VA Smalltalk)

This is a TensorFlow wrapper for VA Smalltalk
Report a defect | Request feature

TensorFlow is a Google open source machine learning library for research and production. And this is a wrapper to be used from a higher level language like VA Smalltalk.

License

• The code is licensed under MIT.
• The documentation is licensed under CC BY-SA 4.0.

Supported platforms and versions

Currently we tested this wrapper on Linux and Windows, both on x86 and x64. In addition, we tested on ARM (Raspberry Pi 3B+ and Raspbian Buster), ARM64 (Rock64 and Armbian Buster) and ARM64 with GPU support (Nvidia Jetson Nano).

VA Smalltalk needed version is 9.2.2 or above and we have only tested on TensorFlow versions 1.13.x and 1.14.x.

Installation

• Download the 9.2.2 from Instantiations. If any of the following steps cannot be achieved, it might be due to last minute changes in the TensorFlow configuration maps and/or improvements on the VAST VM or the base library. Please contact us for an up-to-date download.
• Install TensorFlow for C for your operating system (download one of the tested versions).
• For Windows installations, make sure Microsoft Visual C++ Redistributable for Visual Studio 2017 is installed. If you still have troubles trying to load TensorFlow lib in VAST, read this and this blog post to help debugging the situation.
• Ensure tensorflow shared library (.so or .dll) is findable by OS lookup procedure or reference full path in VAST ini file.
• Add TENSORFLOW_LIB key/value under [PlatformLibrary Name Mappings] section in abt.ini file. Some examples:

TENSORFLOW_LIB=tensorflow
TENSORFLOW_LIB=/usr/local/lib/libtensorflow_framework.so.1.14.0
TENSORFLOW_LIB=/home/mpeck/Instantiations/TensorFlow/libtensorflow-cpu-linux-x86_64-1.14.0/lib/libtensorflow.so.1.14.0
TENSORFLOW_LIB=c:\Users\mpeck\Documents\Instantiations\tensorflow.dll
TENSORFLOW_LIB=z:\Instantiations\TensorFlow\libtensorflow-cpu-windows-x86_64-1.13.1\lib\tensorflow.dll

• Install Tonel support in your development image following this guide.
• Clone this repository.
• The easiest and recommended approach is to install TensorFlow for VASAT is via a script:

| loader path |
path := (CfsPath named: '<insert path to root tensorflow-vast local repo here>').
loader := TonelLoader readFromPath: path.
loader
	beUnattended; "do not prompt and use all defaults"
	useGitVersion.
loader loadAllMapsWithRequiredMaps.

Or you can load the Configuration Map TensorFlow from the context menu of the Configuration Maps Browser: "Import" -> "Load Configuration Maps from Tonel repository..." -> select path to root tensorflow-vast local repo. This will open a dialog and will use convenient defaults for the load. Refer to its documentation for more details.

• Optionally run the SUnit tests included in the map TensorFlow to ensure correct installation. One easy way is to right-click on the TensorFlow map name in the Name pane (as opposed to version pane) and then select Test Loaded Applications.

Examples

We will be submitting more and more examples in TensorFlowExamplesApp. So far the only working example is LabelImage which loads a pre-trained TensorFlow network and use it to recognize objects in images. You can read its class comments for details, instructions and possible uses.

There is also a full detailed blog post about this example.

You can also run a Inception V3 like what is described here:

We also have a more advanced Object Detection example with bounding boxes, labels and scores. This blog post goes over the glory details of this example:

Running TensorFlow and VA Smalltalk on ARM

We believe that running TensorFlow and VA Smalltalk on ARM-based boards is really interesting. From limited devices such as a Raspberry Pi to a Nvidia Jetson.

We have compiled TensorFlow C library for ARM and ARM 64 and have been playing with different operating systems and boards such as Raspberry Pi 3B+ (Raspbian), Rock64 (Armbian) and soon Nvidia Jetson Nano and XT2.

Building TensorFlow from scratch on ARM is a bit complicated so we try to document this process on blog posts as we learn. For the moment, you can checkout these:

• Challenge Accepted: Build TensorFlow C Binding for Raspberry Pi in 2019
• The battle continues: Build TensorFlow C Binding for Raspberry Pi in 2019

Blog Posts

• Getting Started with Nvidia Jetson Nano, TensorFlow and Smalltalk
• Recognizing objects in images with TensorFlow and Smalltalk
• Object Detection with TensorFlow and Smalltalk
• TensorFlow — The first week
• Challenge Accepted: Build TensorFlow C Binding for Raspberry Pi in 2019
• The battle continues: Build TensorFlow C Binding for Raspberry Pi in 2019

Acknowledgments

• gera for all his work on the original code for Cuis Smalltalk as well as for the port to VA Smalltalk.
• Github repository layout was generated with Ba-St Github-setup project.

Contributing

Check the Contribution Guidelines

tensorflow-vast's Issues

Uploaded our generated shared libraries for ARM or document where to get them

We want to be able to have ARM shared libraries for the Raspberry Pi, Nvidia jetson Nano, etc... we are in the process of deciding whether to compile ourselves and upload here or document exactly how and where to get them.

More examples: INVALID_ARGUMENT: In[1] is not a matrix. Instead it has shape [2]

(Case reproduced in this commit

I imported the rest of the PolyMathOrg examples , and adapted all the tests in TensorFlowExamplesTest them for VAST but 6 of them are failing with this same error:

INVALID_ARGUMENT: In[1] is not a matrix. Instead it has shape [2]
[[{{node MatMul_4}}]]

Probably there is an error in the code I adapted while writing it for VAST but can't find the error

[Image Rendering] Make the label of the boxes to be filled with color

I would like the labels area to be FILLED with the given color and just let the text as black/white (depending on the background color). See this example. The orange and green are filled as background of the text, while the text remains just black. I would do that same but also auto select white text if the background is dark.

Update the documentation after having move from ENVY to Tonel

The documentation at https://github.com/vast-community-hub/tensorflow-vast#installation is outdated and says

"From the configuration map browser, import all versions of the TensorFlow map from envy/TensorFlow.dat. Then "Load With Required Maps" the latest version of it.
"

But now TensorFlow is not on ENVY anymore but on Tonel.

Add a simple image recognizon example

I would love to have an example like this one: https://blog.insightdatascience.com/tensorflow-image-recognition-on-a-raspberry-pi-3645a1634c66

Check CgImageSupport app for our images support.

Check TFBufferTest >> #testNewFree

It was failing in some platforms because new was implemented to allocate a ByteArray to host the TFBuffer instead of an external pointer, and then delete is called on the buffer. I believe the best solution is to unify creation protocol (new?) and make it very clear when it's necessary to call delete (never?) and when it's done by finalization

[Image Rendering] Automatically have a table of colors that maps to labels/classes

Right now all boxes and labels are display with the same color (whatever that is). It would be great if we could have a list of colors (those that normally have a nice contrast with most pictures) and that each "class" (Person, Dog, etc) is associated to a given color. That way, all boxes of the same class will have same color and each class will have a different one.

See for example this picture. It has one color for kite and another one for person.

[Image Rendering] Scale fonts in labels

Would be nice to have HiDPI scaling to the text used for labels (it looks pixeled now). See:

CgPixmap>>#isAutoScalable
    ^false

You may try to create subclass and override this method (but I'm not sure about the result).
Second way is apply for current gc a scaled font. Something like:

DPIScaler autoScaleObject: aCurrentFont for: aCurrentAutoScaleContext.

Update master with a commit made from latest Tonel

Unexpected Gradient error (No gradient defined for op: OnesLike)

I intend to write the second example in the GradientTape tutorial

x = tf.constant(3.0)
with tf.GradientTape() as g:
  g.watch(x)
  with tf.GradientTape() as gg:
    gg.watch(x)
    y = x * x
  dy_dx = gg.gradient(y, x)     # Will compute to 6.0
d2y_dx2 = g.gradient(dy_dx, x)  # Will compute to 2.0

in TensorFlowCAPITest>>#testNestedGradient but the got the following error

NOT_FOUND: No gradient defined for op: OnesLike. Please see https://www.tensorflow.org/code/tensorflow/cc/gradients/README.md for instructions on how to add C++ gradients.

It seems gradientsOf:respectTo: in VAST is not doing the same as the gradient in python.

• gradientsOf:respectTo: ends up calling the C++ api TF_AddGradients
• while gradient in python calls tensorflow.python.eager.imperative_grad.imperative_grad, which calls tensorflow.python.pywrap_tensorflow.TFE_Py_TapeGradient, which is a C++ function calling ComputeGradient.

(Case reproduced in this commit)

envy subfolder missing

The README refers to a .dat configuration map that is missing in this repo.
VAST-Tonel can' be loaded into Unix environments currently.
Could you please reintroduce the envy folder temporarily?

loading into ECAP fails

Hi!

I am trying to load tensorflow-vast into my ECAP image that is running on a Jetson Nano.
I successfully loaded Tonel 1.74 into this image. The v1.3 Tonel version can't be loaded because my ECAP image does not have the newest required SUnit maps available.
Loading tensorflow-vast into this image via the application manager fails due to a missing subordinate related to TFInput during the loading process.

Loading the code via the following script also fails because the SUnit prerequisite 'z.ST: SUnit V 9.2.2 [002]'
is missing in my image

| loader path |
path := (CfsPath named: '').
loader := TonelLoader readFromPath: path.
loader
beUnattended; "do not prompt and use all defaults"
useGitVersion.
loader loadAllMapsWithRequiredMaps.

Is there a newer ECAP release available for the Jetson Nano?
If not, how can I load an older version of this repo that works with the 'ECAP VAST 9.0.0' release on a nvidia jetson?

Thank you!

[Image Rendering] Maximizing size of opened VA Window for image

When I open the image on VA, the default opened size/extent of the windows is not the full size of the pic and so I must extend it by hand. See both, vertical and horizontal scrolls on below example:

I want that by default, the windows fix the picture (if possible). Below is how I would like that example (expanded by hand by me):

Add a simple object detection example

Now that we got image recognition to work, we shoud try "Object Detection". Something like this or this or this.

I wonder if we can download pre-trained models with that and then display the images from Smalltalk. Did you ever try? @SergeStinckwich ?

[Image Rendering] Smart location of the label

If the box is too high, the label is now outside the image..should be printed BELOW the box instead. See below example:

See the label of the laptop? No, you probably can't :) Because it was outside of the picture...

Add support for TensorFlow Lite C API

It looks like @freedomtan did a fork of Pharo port and added support for TensorFlow Lite C API. We would love to have that integrated in Cuis/VAST/Pharo. I checked the differences and doesn't seem to be that much.

TFOutput>>#operation has graph in nil

Case reproduced in this commit

I updated TFOutputTest>>#testOperation to show the case: while TFOutput>>#operation and TFOutput>>#operationOn: are apparently the same TFOperation, the first one has the instance variable graph in nil and can't be used in any example defined in LibTensorFlowExamplesApp.

Would be nice them to be the same, or only having one way to ask for the operation and avoiding any confusion

[Image Rendering] Be able to change the color of boxes and labels

Right now, they are black. We experimented with processResultsAndDrawBoxes and saw that adding something like:

gc setForeground: (image palette nearestPixelValue: (Color red)).

seemed to work. However, the resulting display color is not similar to the one we specified. This is likely related to the palette we are using to send nearestPixelValue:.

Check what the color palette is for it. If it’s really limited, then perhaps it is the nearest color..

Maybe we need to enable full VA RGB palette by reading the rgb.txt? Check private email for this.