runwayml / processing-library Goto Github PK

Will there be support for hosted models from runway in the processing library itself?
Right now it just supports local instances over http.

The python code example with requests lib works smoothly.
Although needed a bit of a cleanup, in the response section to, in order to parse the data for saving the image:

[python code example snippet for re-saving the response as an image]

# Get the interested part from the response, containing base64 image data
img_str = output["image"]

# To remove "data:image/png;base64,..." from the response
print("\nClean Output:")
clean_img_str = img_str[img_str.index(',')+1:]

# For saving the image file
img_data = base64.b64decode(clean_img_str)
filename = 'your image.png'
with open(filename, 'wb') as f:
    f.write(img_data)

BTW, Great work. 👍

I was trying to use processing http.requests library to send a post request to query with a base64 string data of an image (as the api only support data inputs as base64 string).

Here's my snippet:

import javax.xml.bind.DatatypeConverter;
import http.requests.*;
String model_end_point = "<API_ENDPOINT>/query";
String api_key = "<API_KEY>";

void setup() {
  size(1020, 480);
  background(25);

  // convert image to byte array
  byte imageByte[] = loadBytes("3.jpg");
 
  // prepare the base64 string from the byte array of the image 
  String b64 = DatatypeConverter.printBase64Binary(imageByte);
  // println("Original image base64 data:");
  // println(b64);
  // println("\n");

  PostRequest post = new PostRequest(model_end_point);
  post.addHeader("Accept", "application/json");
  post.addHeader("Content-Type", "application/json");
  post.addHeader("Authorization", api_key);
  post.addData("image", b64);
  post.send();

  println("\nResponse:");
  try{
    println(post.getContent());
  }catch (Exception e){
    println(e);
  }
  exit();
}

To this I always get an error response:
{"error":"Internal server error"}

I do not know what I'm doing wrong. 🤔
Note: I was referencing the python code example from the runway itself.

Any lead would help here.

Once we have our first release, we should be able to submit the library to the Processing library website and directory: https://processing.org/reference/libraries/

Is more recent updates Runway has changed the output text data.getString() to be "generated_text" not "text". This makes the Processing example work on versions around 0.10.10.
However, in the more recent versions 0.10.27 this example fails to generate the output text in either Runway or Processing. It receives the input String fine which makes me think it isn't a port or network issue. Perhaps another message structure was changed that is not visible in the code but part of the library file?
Other examples continue to work such as AttGAN.

I get the following error message when Processing connects to Runway:
"couldn't find runwayErrorEvent callback in sketch, ignoring data"

{"outputs":[{"defaultOutputFormat":"JPEG","channels":3,"name":"generated_output","description":null,"type":"image"}],

"inputs":[{"samplingMean":0,"default":null,"length":128,"name":"z","description":null,"type":"vector","samplingStd":2},{"default":"tench, Tinca tinca","oneOf":["tench, Tinca tinca","goldfish, Carassius auratus","great white shark, white shark, man-eater, man-eating shark, ...

"Error, disabling runwayInfoEvent()"

I am getting the error listed below when I attempt to run all of the provided examples in the HTTP folder. System setup: OSX 10.12, Processing 3.4, Runway 0.9.16. Not a bug per se I suppose, more a case of missing install instructions?

java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at processing.core.PApplet.createInputRaw(PApplet.java:6780)
	at processing.core.PApplet.createInput(PApplet.java:6738)
	at processing.core.PApplet.createReader(PApplet.java:6549)
	at processing.core.PApplet.loadJSONObject(PApplet.java:5901)
	at com.runwayml.RunwayHTTP.setup(Unknown Source)
	at com.runwayml.RunwayHTTP.<init>(Unknown Source)
	at CycleGAN.setup(CycleGAN.java:72)
	at processing.core.PApplet.handleDraw(PApplet.java:2387)
	at processing.awt.PSurfaceAWT$12.callDraw(PSurfaceAWT.java:1540)
	at processing.core.PSurfaceNone$AnimationThread.run(PSurfaceNone.java:316)
The file "http://127.0.0.1:8000/info" is missing or inaccessible, make sure the URL is valid or that the file has been added to your sketch and is readable.
error parsing JSON from /info HTTP route
java.lang.NullPointerException
	at processing.data.JSONTokener.<init>(JSONTokener.java:58)
	at processing.data.JSONObject.<init>(JSONObject.java:226)
	at processing.core.PApplet.loadJSONObject(PApplet.java:5901)
	at com.runwayml.RunwayHTTP.setup(Unknown Source)
	at com.runwayml.RunwayHTTP.<init>(Unknown Source)
	at CycleGAN.setup(CycleGAN.java:72)
	at processing.core.PApplet.handleDraw(PApplet.java:2387)
	at processing.awt.PSurfaceAWT$12.callDraw(PSurfaceAWT.java:1540)
	at processing.core.PSurfaceNone$AnimationThread.run(PSurfaceNone.java:316)

The processing-library-template comes with a web version of the docs that's autogenerated. We should host this in GitHub pages and create a friendly domain to host it.

Hi, Thank you for your great work! And I would like to know that is the computation(NN forward inference) done in remote? And how about the speed (espacially for face landmark and human pose estimation)?

I appreciate Runway ML Lab isn't as supported now however I'm hoping I might be able to find a solution for teaching purposes.

I am running the StyleGAN example in processing 4.01A (August 9th 2022 release). The example communicates via HTTP successfully. I can see the Runway input interface and preview window update. However, the image is not returned to processing. The error is
Error, disabling runwayDataEvent() null

I have printed the infoEvent details and they are as follows.
{ "outputs": [{ "defaultOutputFormat": "JPEG", "channels": 3, "name": "image", "description": null, "type": "image" }], "inputs": [ { "samplingMean": 0, "default": null, "name": "z", "length": 512, "description": null, "type": "vector", "samplingStd": 0.5 }, { "default": 0.8, "min": 0, "max": 1, "name": "truncation", "description": null, "step": 0.01, "type": "number" } ] }

The example works perfectly in processing 3.5.4 so i don't think it is a message string issue or change on Runway's side. I have tried giving Processing 4 permission access to all files and folders but no change. Other examples run without issue in Processing 4.01 A (eg:GPT2 example that is text based) therefore it is maybe a Base64 issue? Any pointers greatly appreciated as I hope to use this in a teaching example and would rather not request students mix versions of Processing.

It would be nice to be able to encode PGraphics object to a base64 image to be send to runway too (as it is already possible for PImage).

To be more clear here is what I have in mind :

`
PGraphics pg;
void setup(){
pg = createGraphics(600,400);
}

void draw(){
pg.beginDraw();
pg.fill(255);
pg.ellipse(mouseX,mouseY,25,25);
pg.endDraw();
}

void keyPressed(){
JSONObject input = new JSONObject();
// set style image
input.setString("style_image",ModelUtils.toBase64(styleImage));
// set content image
input.setString("content_image",ModelUtils.toBase64(pg));
// set interpolation weight
input.setFloat("interpolation_weight",interpolationWeight);
// query Runway
runway.query(input.format(-1));
}
`