Coder Social home page Coder Social logo

joewalnes / websocketd Goto Github PK

View Code? Open in Web Editor NEW
17.1K 367.0 1.0K 1.05 MB

Turn any program that uses STDIN/STDOUT into a WebSocket server. Like inetd, but for WebSockets.

Home Page: http://websocketd.com/

License: BSD 2-Clause "Simplified" License

Makefile 11.02% Go 84.40% Roff 4.58%
websocket-server websockets proxy

websocketd's Introduction

websocketd

websocketd is a small command-line tool that will wrap an existing command-line interface program, and allow it to be accessed via a WebSocket.

WebSocket-capable applications can now be built very easily. As long as you can write an executable program that reads STDIN and writes to STDOUT, you can build a WebSocket server. Do it in Python, Ruby, Perl, Bash, .NET, C, Go, PHP, Java, Clojure, Scala, Groovy, Expect, Awk, VBScript, Haskell, Lua, R, whatever! No networking libraries necessary.

-@joewalnes

Details

Upon startup, websocketd will start a WebSocket server on a specified port, and listen for connections.

Upon a connection, it will fork the appropriate process, and disconnect the process when the WebSocket connection closes (and vice-versa).

Any message sent from the WebSocket client will be piped to the process's STDIN stream, followed by a \n newline.

Any text printed by the process to STDOUT shall be sent as a WebSocket message whenever a \n newline is encountered.

Download

If you're on a Mac, you can install websocketd using Homebrew. Just run brew install websocketd. For other operating systems, or if you don't want to use Homebrew, check out the link below.

Download for Linux, OS X and Windows

Quickstart

To get started, we'll create a WebSocket endpoint that will accept connections, then send back messages, counting to 10 with 1 second pause between each one, before disconnecting.

To show how simple it is, let's do it in Bash!

count.sh:

#!/bin/bash
for ((COUNT = 1; COUNT <= 10; COUNT++)); do
  echo $COUNT
  sleep 1
done

Before turning it into a WebSocket server, let's test it from the command line. The beauty of websocketd is that servers work equally well in the command line, or in shell scripts, as they do in the server - with no modifications required.

$ chmod +x count.sh
$ ./count.sh
1
2
3
4
5
6
7
8
9
10

Now let's turn it into a WebSocket server:

$ websocketd --port=8080 ./count.sh

Finally, let's create a web-page to test it.

count.html:

<!DOCTYPE html>
<pre id="log"></pre>
<script>
  // helper function: log message to screen
  function log(msg) {
    document.getElementById('log').textContent += msg + '\n';
  }

  // setup websocket with callbacks
  var ws = new WebSocket('ws://localhost:8080/');
  ws.onopen = function() {
    log('CONNECT');
  };
  ws.onclose = function() {
    log('DISCONNECT');
  };
  ws.onmessage = function(event) {
    log('MESSAGE: ' + event.data);
  };
</script>

Open this page in your web-browser. It will even work if you open it directly from disk using a file:// URL.

More Features

  • Very simple install. Just download the single executable for Linux, Mac or Windows and run it. Minimal dependencies, no installers, no package managers, no external libraries. Suitable for development and production servers.
  • Server side scripts can access details about the WebSocket HTTP request (e.g. remote host, query parameters, cookies, path, etc) via standard CGI environment variables.
  • As well as serving websocket daemons it also includes a static file server and classic CGI server for convenience.
  • Command line help available via websocketd --help.
  • Includes WebSocket developer console to make it easy to test your scripts before you've built a JavaScript frontend.
  • Examples in many programming languages are available to help you getting started.

User Manual

More documentation in the user manual

Example Projects

Got more examples? Open a pull request.

My Other Projects

And follow @joewalnes!

websocketd's People

Contributors

andreydiveev avatar asergeyev avatar azjezz avatar bfx81 avatar coderofsalvation avatar cosmo avatar da440dil avatar danielbodart avatar franklinchen avatar ideal avatar jeroenjanssens avatar joewalnes avatar kfl avatar maciej-gurban avatar markushx avatar patrickjs avatar pborreli avatar pnasrat avatar rowanthorpe avatar ruediger avatar sai-manoj-kumar avatar samansjukur avatar samyakbhuta avatar sl4mmy avatar theburningmonk avatar thypon avatar volkovmqx avatar warpcoil avatar zackexplosion avatar zlim avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

websocketd's Issues

Look over wiki pages

Wiki was not updated for a while. It might use another pass on all pages and pursue couple additional goals for the project:

  • mention collaboration, how one could help this project (by trying out experimental versions, create binary installable packages for variety of OS, publicly sharing experience with websocketd, etc)
  • actively asking for features that are missing for particular use case, it's fine that many ideas would not fit but just having a place to share them for people might be good.

Multiplexing/Demultiplexing mode

In certain situation, (like vmstat over websocket) there is no reason to have data generating scrip to be run new time for each client. More obvious choice is to run it once and mux all input to it, and demultiplex the output to each connected client.

What do you folks think about having such mode available via option? Sure it'll confuse some people and should be off by default.

Connection timeout in less than 2 minutes

My open websocket dies after 1:30/2:00 minutes. I've created a keepalive mechanism with javascript (periodicaly send something).

Is it the default by specs ?

Thanks!

Should websocketd support closed STDOUT

In rare case when stdout is closed but input is still processed by an application (and regardless of stderr) websocketd would not behave properly. Should this use be properly handled?

could not launch process

hi everything was running fine, but a stop and start of websocket --port=port ./myscript.sh
now starts but when trying to access via the webpage gives the following error
Could not launch process ./myscript.sh (fork/exec ./myscript.sh: exec format error)
any idea what this could be.

Handle "no such file or directory", don't just close

If I start websocketd with a ref. to a non-existing application, e.g. if I am in the wrong directory, I think it would be useful if websocketd could give the client some info about the problem, maybe an "error". With the curren version (0.2.7), the client just receives a "close" with code=1000.

Needs throughout testing on IPv6 vs IPv4.

Some things in go web code rely on colons for port numbers and other things. I doubt there are bugs but it would be good to hear from someone who can check things methodically on IPv6.

Associate executables with extensions

At the moment, websocketd will map urls to files and then attempt to execute the file itself.

It would also be useful to allow the file to be executed by another program. This mapping will be done by file extension.

This removes the need to have a #!/usr/bin/blah header at the top of each script.

Examples of when this is useful:

  • PHP: Typically PHP files do not include the #! header and are identified by a .php extension.
  • Windows scripting host: Avoids the needs for a .cmd file for each script.

Implementation notes:

  • Should be possible to define executors for more than one file extension.
  • Should be possible to specify additional cmd line args that can be passes to executable.
  • Special care needed on case insensitive file systems to ensure correct executable still gets invoked (e.g. foo.PhP)

Release as deb/rpm/pkg

jordansissel/fpm allows you to easily create packages in multiple formats such as *.deb, *.rpm and *.pkg for Mac (and even tarballs for those who do not want them).
bernd/fpm-cookery also helps creating packages by allowing to write small scripts that use fpm.
With those two projects, it should be fairly easy to build packages in multiple formats.

Thinking about --basepath option. Suggesting to allow multiple "dir" with different bases.

What was --basedir made for? Anyone uses it as it is implemented now?

IMO there is no clarity on what should happening if someone says one of the three:

--basepath=dir
--basepath=/dir
--basepath=/dir/

IMO first one should be prohibited and second means that you need to use prefixed URI ws://host:port/dirdump-env.sh to reach the script. I think it could be useful.

But I also feel that It would be better to have ability to set multiple --dir options with different bases e.g

--dir "/usr/local/tools /article-data/" --dir "/usr/local/tools/monitoring /monitoring/"

Same could be applied to staticdir and cgidir.

document go get

Package is go gettable but not documented as such.
Consider updating the install section with the same.

Origin checks break count.html in Ten minute tutorial via file://

Summary

  • Expected behaviour: Load count.html from Ten minute tutorial via file://, and get the count displayed
  • Actual behaviour: Blank page, with rejected null origin in websocketd console.

From reading #20 comments and https://github.com/joewalnes/websocketd/blob/master/help.go, I expected the origin check to be disabled by default.

Detail

I'm working through the Ten minute tutorial.

I've created and tested count.py, and am running it via

websocketd-0.2.9-windows_amd64\websocketd.exe --port 8080 --devconsole python count.py

The test of loading via http://localhost:8080/ works.

However, when I create count.html, copied directly from the tutorial, I get a blank page. (JavaScript is not disabled)

The console output from websocketd says:

Tue, 15 Jul 2014 21:04:59 +0100 | ACCESS | session    | url:'http://localhost:80
80/' id:'1405454699281408400' remote:'PCBlah' command:'C:\Python27\python.exe'
 | rejected null origin

Document how to build the code on Windows

In order to test the fix for #75, I tried building the latest code (on Windows 7) to test the fix, by following Working-on-the-code.

I installed go 1.3 for Windows 64-bit, and was then able to download the code using these commands (in a CMD window with git.exe in the path)

cd    %GOPATH%
mkdir %GOPATH%\src\github.com\joewalnes
cd    %GOPATH%\src\github.com\joewalnes
git clone https://github.com/joewalnes/websocketd
cd websocketd

Then I realised that the instruction to run make wasn't going to run on Windows, and from looking at the Makefile, I see it's (understandably) really unix-focussed.

As a complete Go novice, I stopped at this point.

It would be really helpful if a new Windows section could be added to Working-on-the-code, please.

Two DNS lookups of remoteAddr

websocketd handler does lookup to perform logging. Same time there is another lookup in env module. It's not a big deal but should be fixed.

QUERY_STRING not returning expected data using windows 64 bit executable

Only tested against 64 bit websocketd

Create a .cmd file with following:

@echo off
setlocal EnableDelayedExpansion
@echo %PATH%
@echo %QUERY_STRING%
@echo %SERVER_NAME%
@echo %GATEWAY_INTERFACE%

Service with websocketd, use count.html example to connect and display the result.
Use url like for example: file:///C:/temp/websocketd/count.html?asfds=foo&bar=loo

Expected:
CONNECT
MESSAGE: c:...; ...
MESSAGE: asfds=foo&bar=loo
MESSAGE: localhost
MESSAGE: websocketd-CGI/0.1
DISCONNECT

Got:
CONNECT
MESSAGE: c:...; ...
MESSAGE: ECHO is off.
MESSAGE: localhost
MESSAGE: websocketd-CGI/0.1
DISCONNECT

Empty Sec-WebSocket-Protocol: in response

If the request has no Sec-WebSocket-Protocol: field the response returns with a Sec-WebSocket-Protocol: field that is present but has no value. This causes chrome to give an error of "failed: Invalid UTF-8 sequence in header value"

Interestingly, the binary versions available for download don't do this.

A workaround browser-side is just to specify a protocol

var ws = new WebSocket(uri,"dummyprotocol");

Why not use npm?

use npm -g install websocketd to install is much better than present workflow

Lazy process

I'm not sure if i'm using this correctly, but if i launch websocketd with a python script, iot doesn't actually get executed until someone connects to the websocket. Is there any way to start the process immediately?

32-bit executable for OS X

Hey Joe,
we're using websocketd at http://codebender.cc and I absolutely love it! we're having issues with some older macs though, and it turns out, it's because they are 32-bit only.

any chance we could get a 32-bit binary, or even better, a cross-arch one?

thanks!

Publish 0.2.9 binaries

First I'll put them as non-default ones and later swap 0.2.8 to these experimental ones.

Long Help

While working on verifying directories I run into realization that help message is crazy long and only will get longer. How about ShortHelp function that's called by default and full help only generated by calling --help?

    short = `
    {{binary}} ({{version}})

    Usage:

      Export a single executable program a WebSocket server:
        {{binary}} [options] COMMAND [command args]

      Or, export an entire directory of executables as WebSocket endpoints:
        {{binary}} [options] --dir=SOMEDIR

      Or, show extended help message:
        {{binary}} --help
    `

Raw protocol support (for binary messages)

I propose an option to allow websocketd to be run in "raw mode". In this mode, processes will send/receive message using the WebSocket data framing protocol as defined in RFC6455 (as opposed to newline delimited text).
http://tools.ietf.org/html/rfc6455#section-5

When you'd want this:

  • For text messages that contain "\n"
  • For binary messages
  • For lower level control of ping/pong messages

Of course this adds complexity to programs as they now need to speak to the data framing protocol, which is more complicated than line delimited text, so programs should only use this when they need it.

In raw mode, websocketd will continue to act as a socket server, handle HTTP upgrades, perform WebSocket handshakes and route to the correct process. It is only the actual message framing that apps need to worry about.

Need a logo

Need a logo for websocketd. A big one for the website, and a favicon.

A way to support PATH_INFO calls maybe

When someone calls /somepath/script.sh/test/ok in Apache, and the script.sh present in somepath directory it will be called with PATH_INFO set to /test/ok.

I'm not sure if that would be good feature to support in websocketd or no.

Document how to run websocketd behind a frontend

In some production setups, it's desirable to run websocketd behind another HTTP frontend such as Nginx or HAProxy. Benefits include:

  • Easily fit into existing web infrastructure setup
  • Load balancing, throttling, etc
  • SSL offloading
  • Sharing the same host/port with other services

Add to the wiki configuration example of how to do this.

opensnoop (OSX) doesn't get a SIGINT when browser does socket.close()

First I do this (on Mac OS X 10.6.8 and 10.8.5) to enter my password:
sudo opensnoop -v

I then do a Ctrl-C, and start for real:
websocketd --port=8080 sudo opensnoop -v

I can connect from my browser (Chrome), but when my browser does socket.close(), 'opensnoop' doesn't seem to terminate. I connect/disconnect a few times. When I in the end do a Ctrl-C to terminate websocketd, then my browser gets all the missing 'close' events.

wrapped processes must exit when websoket.close occurs

Joe;

First of all, thanks for creating an incredibly useful bit of code. It works great and does just what it's supposed to do, simply, elegantly and reliably.

When websocketd wraps a process and the wrapped process behaves like a daemon the wrapped process needs a mechanism to terminate itself on a sigint / sigterm. This may be obvious to some but it wasn't obvious to me.

It might be worthwhile to mention this somewhere in the documentation. i.e. "programs running under websocketd that persist must handle signals instructing them to exit (i.e. a sigterm handler on a UNIX-like operating system). Unless the wrapped program exits, the websocketd sub-process that is running it will not ever exit.

Here's some code that illustrates this.

https://gist.github.com/Royhb/ce3972ef07cc3a619b23

Pretty website for websocketd.com

Would like a nice user friendly website.

Question: should the primary source of documentation be on the website, or here on the GitHub wiki. Leave a comment if you have an opinion.

Allow sending non line-terminated data to STDIN

Some applications are controlled by key commands without requiring the enter key (e.g. vim, less, ...). It would be great if there was a switch (maybe for the websocketd command line) to not automatically append the Enter key to input from the websocket client (Browser).

Single CGI and single static HTML situation

it's currently possible to provide following:
--cgidir=examples/cgi-bin/dump-env.sh
--staticdir=index.html

Both return 404 for any but "/" request uri. CGI just runs in this case but HTML returns awkward 301 with Location: ..// which then redirects to / back.

Both need to be fixed. Possible ways to go about it:

  1. if "--*dir" specified it has to point a directory, generate an error otherwise.
  2. both should cause websocket to respond with same script/content on any URI that's not upgraded to a websocket.

doesn't work with Firefox 26

The devconsole doesn't work with Firefox 26.0 (Ubuntu 13.10, 64 bit). With chromium-browser it works well.

Edit: the same holds for the example count.html. FF: fail, Chromium: OK.

Create a manpage

Include a manpage when installed through an installer/package-manager. Should probably be the same content as the build in help.

Environment variables from host are sent to scripts

The host operating system environment variables are being passed along to the scripts as they run. This should be restricted, with perhaps an option to add to a list (defaults to empty) of environment variables to "inherit" from the host operating system.

FIRST.

Approach to testing websocketd performance for your workload (wiki page tbw)

"thor" (https://github.com/observing/thor) is an interesting tool that could be useful to anyone who wants to check performance of websocket server endpoint. It requires node to be installed and bunch of packages to work.

Also, because it uses non-masked messages by default and to generate messages that look like STDIN (line-separated) you have to create your own generator.

Well.. All this leads me to put this placeholder and check if folks are interested to talk about this topic. Getting more confidence about thor and it's features may also help someone to suggest (or to create) new websocket benchmarking tool in go.

WebSocket Origin restrictions

Background

WebSockets are cross domain by default. e.g. A page on http://foo can open a WebSocket on ws://bar.

User defined WebSocket endpoints can see the originating origin through the Origin HTTP header (exposed as HTTP_ORIGIN environment variable to scripts). If they wanted to restrict to certain origins, they could check at the beginning of the script and exit appropriately.

Problem

The above works, but:

  • It's tedious. If there are many scripts it would have to be applied to each of them.
  • Security should be "on" by default.
  • Deployment configuration (eg hostnames) gets buried in runtime code.

Proposed solution

websocketd should perform origin checking and reject mismatches before invoking user scripts. Configuration via command line flags.

Examples:

  • Default: restrict to same origin only.
  • "--origin=foo,bar:1234": only allow if origin is from http://foo (any port) or http://bar:1234
  • "--anyorigin": Allow for any origin

Websocketd likely needs some resource limits in future

To prevent forkbombing the server it would be interesting to discuss ability to limit possible amount of connections that server should sustain. After that new connections should be dropped.

Also, for whom it might matter, might prefer to set timeouts on longest living websockets.

I don't know websocket golib well to know if some protection is already there.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.