observing / thor

The WebSocket god of thunder

Thor

Thor is WebSocket benchmarking/load generator. There are a lot of benchmarking tools for HTTP servers. You've got ab, siege, wrk and more. But all these tools only work with plain ol HTTP and have no support for WebSockets - even if they did they wouldn't be suitable, as they would be testing short running HTTP requests instead of long running HTTP requests with a lot of messaging traffic. Thor fixes all of this.

Dependencies

Thor requires Node.js to be installed on your system. If you don't have Node.js installed you can download it from http://nodejs.org or build it from the github source repository: http://github.com/joyent/node.

Once you have Node.js installed, you can use the bundled package manager npm to install this module:

npm install -g thor

The -g command flag tells npm to install the module globally on your system.

Usage

thor [options] <urls>

Thor can hit multiple URL's at once; this is useful if you are testing your reverse proxies, load balancers or just simply multiple applications. The url that you supply to thor should be written in a WebSocket compatible format using the ws or wss protocols:

thor --amount 5000 ws://localhost:8080 wss://localhost:8081

The snippet above will open up 5000 connections against the regular ws://localhost:8080 and also 5000 connections against the secured wss://localhost:8081 server, so a total of 10000 connections will be made.

One thing to keep in mind is you probably need to bump the amount of file descriptors on your local machine if you start testing WebSockets. Set the ulimit -n on machine as high as possible. If you do not know how to do this, Google it.

Options

  Usage: thor [options] ws://localhost

  Options:

    -h, --help                      output usage information
    -A, --amount <connections>      the amount of persistent connections to generate
    -C, --concurrent <connections>  how many concurrent-connections per second
    -M, --messages <messages>       messages to be send per connection
    -P, --protocol <protocol>       WebSocket protocol version
    -B, --buffer <size>             size of the messages that are send
    -W, --workers <cpus>            workers to be spawned
    -G, --generator <file>          custom message generators
    -M, --masked                    send the messaged with a mask
    -b, --binary                    send binary messages instead of utf-8
    -V, --version                   output the version number

Some small notes about the options:

• --protocol is the protocol version number. If you want to use the HyBi drafts 07-12 use 8 as argument or if you want to use the HyBi drafts 13-17 drafts which are the default version use 13.
• --buffer should be size of the message in bytes.
• --workers as Node.js is single threaded this sets the amount of sub processes to handle all the heavy lifting.

Custom messages

Some WebSocket servers have their own custom messaging protocol. In order to work with those servers we introduced a concept called generators a generator is a small JavaScript file that can output utf8 and binary messages. It uses a really simple generator by default.

Checkout https://github.com/observing/thor/blob/master/generator.js for an example of a generator.

thor --amount 1000 --generator <file.js> ws://localhost:8080

Example

thor --amount 1000 --messages 100 ws://localhost:8080

This will hit the WebSocket server that runs on localhost:8080 with 1000 connections and sends 100 messages over each established connection. Once thor is done with smashing your connections it will generate a detailed report:

Thor:                                                  version: 1.0.0

God of Thunder, son of Odin and smasher of WebSockets!

Thou shall:
- Spawn 4 workers.
- Create all the concurrent/parallel connections.
- Smash 1000 connections with the mighty Mjölnir.

The answers you seek shall be yours, once I claim what is mine.

Connecting to ws://localhost:8080

  Opened 100 connections
  Opened 200 connections
  Opened 300 connections
  Opened 400 connections
  Opened 500 connections
  Opened 600 connections
  Opened 700 connections
  Opened 800 connections
  Opened 900 connections
  Opened 1000 connections


Online               15000 milliseconds
Time taken           31775 milliseconds
Connected            1000
Disconnected         0
Failed               0
Total transferred    120.46MB
Total received       120.43MB

Durations (ms):

                     min     mean     stddev  median max
Handshaking          217     5036       4094    3902 14451
Latency              0       215         104     205 701

Percentile (ms):

                      50%     66%     75%     80%     90%     95%     98%     98%    100%
Handshaking          3902    6425    8273    9141    11409   12904   13382   13945   14451
Latency              205     246     266     288     371     413     437     443     701

License

MIT

thor's Issues

Thor with uWebSocket

See the issue on the uWebSockets project for details: https://github.com/uWebSockets/uWebSockets/issues/371

Sadly the communication is not going very smoothly and the author of uWebSockets marked the issue as invalid for now. I'm sorry folks if reading about thor compared to a grandma is a little saddening for some :)

Web Socket in Linux

I am testing websocket in linux, but it is not working. Thor is taking long time to give output.Please check on linux.

Ignore self signed cert

Anyway to ignore self signed certs for local testing?

Install issue (solved)

Hello. Thanks for project.
Hope u can fix small issue with installing:

npm install -g thor

npm ERR! Error while executing:
npm ERR! /usr/local/bin/git ls-remote -h -t git://github.com/3rd-Eden/ws.git
npm ERR!
npm ERR! fatal: remote error:
npm ERR! Repository not found.
npm ERR!
npm ERR! exited with error code: 128

Regards.

URL being truncated after first query parameter

I was playing around with Thor, and found the following:

thor --amount 5 ws://localhost:8080/?key1=val1&key2=val2&key3=val3...

produces the following:

Thor: version: 1.0.0

God of Thunder, son of Odin and smasher of WebSockets!

Thou shall:

• Spawn 2 workers.
• Create all the concurrent/parallel connections.
• Smash 5 connections with the mighty Mjölnir.

The answers you seek shall be yours, once I claim what is mine.

Connecting to thor --amount 5 ws://localhost:8080/?key1=val1 //<- Missing the rest of the query parameters after the first '&'

Which on my case means an invalid request. Any chance you can look at it?

No responding!

My WebSocket server (node.js + npm package: ws) just console.log out each message it received. When testing, the command line was always hung out with the last line is "Opened 100 connections", see the capture I attached.

use process.hrtime

For more accurate stats about the latency and handshaking.

RangeError Invalid WebSocket frame: MASK must be set

--masked shouldn't be an option

https://tools.ietf.org/html/rfc6455#section-5.1 says:

a client MUST mask all frames that it sends to the server

So I don't understand why masking is an option in the Thor client and even less why it's disabled by default.

I came across this issue while investigating python-websockets/websockets#36.

ws.git is not found

the command npm install -g thor fails because of a missing dependency.

npm ERR! Linux 4.8.0-32-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "-g" "thor"
npm ERR! node v7.0.0
npm ERR! npm  v4.0.5
npm ERR! code 128

npm ERR! Command failed: git clone --template=/home/user/.npm/_git-remotes/_templates --mirror git://github.com/3rd-Eden/ws.git /home/user/.npm/_git-remotes/git-github-com-3rd-Eden-ws-git-2e9f11e5

the remote in question is not found on github.

insall error/warnings

C:\Users\amir>npm install -g thor
|


> [email protected] install C:\Users\amir\AppData\Roaming\npm\node_modules\thor\node_mod
ules\ws
> (node-gyp rebuild 2> builderror.log) || (exit 0)

/
C:\Users\amir\AppData\Roaming\npm\node_modules\thor\node_modules\ws>node "C:\Pro
gram Files\nodejs\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp
\bin\node-gyp.js" rebuild
Building the projects in this solution one at a time. To enable parallel build,
please add the "/m" switch.
  bufferutil.cc
..\src\bufferutil.cc(55): warning C4267: 'initializing' : conversion from 'size
_t' to 'unsigned int', possible loss of data [C:\Users\amir\AppData\Roaming\npm
\node_modules\thor\node_modules\ws\build\bufferutil.vcxproj]
..\src\bufferutil.cc(66): warning C4267: 'initializing' : conversion from 'size
_t' to 'unsigned int', possible loss of data [C:\Users\amir\AppData\Roaming\npm
\node_modules\thor\node_modules\ws\build\bufferutil.vcxproj]
C:\Users\amir\.node-gyp\0.10.29\deps\v8\include\v8.h(184): warning C4506: no de
finition for inline function 'v8::Persistent<T> v8::Persistent<T>::New(v8::Hand
le<T>)' [C:\Users\amir\AppData\Roaming\npm\node_modules\thor\node_modules\ws\bu
ild\bufferutil.vcxproj]
          with
          [
              T=v8::Object
          ]
     Creating library C:\Users\amir\AppData\Roaming\npm\node_modules\thor\node_
  modules\ws\build\Release\bufferutil.lib and object C:\Users\amir\AppData\Roam
  ing\npm\node_modules\thor\node_modules\ws\build\Release\bufferutil.exp
  Generating code
  Finished generating code
  bufferutil.vcxproj -> C:\Users\amir\AppData\Roaming\npm\node_modules\thor\nod
  e_modules\ws\build\Release\\bufferutil.node
  validation.cc
C:\Users\amir\.node-gyp\0.10.29\deps\v8\include\v8.h(184): warning C4506: no de
finition for inline function 'v8::Persistent<T> v8::Persistent<T>::New(v8::Hand
le<T>)' [C:\Users\amir\AppData\Roaming\npm\node_modules\thor\node_modules\ws\bu
ild\validation.vcxproj]
          with
          [
              T=v8::Object
          ]
     Creating library C:\Users\amir\AppData\Roaming\npm\node_modules\thor\node_
  modules\ws\build\Release\validation.lib and object C:\Users\amir\AppData\Roam
  ing\npm\node_modules\thor\node_modules\ws\build\Release\validation.exp
  Generating code
  Finished generating code
  validation.vcxproj -> C:\Users\amir\AppData\Roaming\npm\node_modules\thor\nod
  e_modules\ws\build\Release\\validation.node
C:\Users\amir\AppData\Roaming\npm\thor -> C:\Users\amir\AppData\Roaming\npm\node
_modules\thor\bin\thor
[email protected] C:\Users\amir\AppData\Roaming\npm\node_modules\thor
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected] ([email protected])
├── [email protected] ([email protected], [email protected], [email protected], [email protected].
0)
├── [email protected]
└── [email protected] ([email protected], [email protected], [email protected])

after this running thor return:

'thor' is not recognized as an internal or external command,
operable program or batch file.

I get 'thor help was called with args...' when trying to run

I was able to install and when attempting to run thor -A 50 <my url> I get an error message:
ERROR: thor help was called with arguments ["-A", "50", "<my url>"] Usage: "thor help [COMMAND].

I am not attempting to call the help command. Any help with this issue will be most helpful.

After created 28194 connection, it stucked

After created 28194 connection succesfully. it stucked without any response.

thor doesn't seem to make more than 105 parallel connections, even with `-C 10000`

I'm trying to test my server with the command thor --masked --amount 10000 -C 10000 ws://localhost:8080/hello/. The report is the following:

Online               30299 milliseconds
Time taken           30299 milliseconds
Connected            9864
Disconnected         0
Failed               136
Total transferred    13.87MB
Total received       13.86MB

Durations (ms):

                     min     mean     stddev  median max
Handshaking          5       6387       4898    4768 16523
Latency              0       3             6       2 112

Percentile (ms):

                      50%     66%     75%     80%     90%     95%     98%     98%    100%
Handshaking          4768    8278    10867   11795   14049   15224   16041   16287   16523
Latency              2       3       4       5       7       9       12      15      112

Received errors:

129x                 connect ECONNRESET 127.0.0.1:8080
7x                   read ECONNRESET

It says it did (almost) 10k connections. But the server indicates that they were not made all in parallel; instead, at most 105 connections were opened at the same time.

Is there a way to force thor to make a specific number of parallel connections? Because the main reason I need it is to test how many connections my server can keep at the same time.

(Also, FYI — I'm using Jetty for the Websocket server implementation).

Using Thor to test STOMP using Websocket

Anyone managed to generate the message for STOMP using thor?
Thanks

Generator <file.js>

I can't create a "file.js" without errors. Could you please include an example of this file...

Thank you.

Got "NaN" for latency

I am trying to test a echo websocket written in java.
this is the command that i gave thor --amount 1000 --messages 100 ws://localhost:8085/web-soc/websocketendpoint
(the echo websocket server is working fine)

Result

Thor:                                                  version: 1.0.0

God of Thunder, son of Odin and smasher of WebSockets!

Thou shall:
- Spawn 8 workers.
- Create all the concurrent/parallel connections.
- Smash 1000 connections with the mighty Mjölnir.

The answers you seek shall be yours, once I claim what is mine.

Connecting to ws://localhost:8085/web-soc/websocketendpoint



Online               5988 milliseconds
Time taken           5989 milliseconds
Connected            1000
Disconnected         0
Failed               0
Total transferred    333.98kB
Total received       329.1kB

Durations (ms):

                     min     mean     stddev  median max
Handshaking          122     379         257     294 5694
Latency              NaN     NaN         NaN     NaN NaN

Percentile (ms):

                      50%     66%     75%     80%     90%     95%     98%     98%    100%
Handshaking          294     398     491     558     718     775     816     833     5694
Latency              NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN

Here latency is NaN, what might be the reason for this issue ?

Name conflict with other binaries

I have a ruby gem on my machine which is also called thor, so when installing this package, there's a conflict. I tried to search for a way to install this using a different name but it doesn't seem like npm support this. Any other solutions?

keep connections alive

Hello,

I need to benchmark a small application where most of the traffic is going from the server to the client. Basically the client subscribes then wait for occasional messages from the server. It seems like a pretty common use of web sockets by the way.

From what I see in the doc it seems that thor closes its connections right after sending messages. It would be great to specify a delay during which the connections are waiting for descending messages.

Thanks !

Could not establish any connection

Hi, I tried to use thor for stress testing my socket.io 1.3.5 server but it doesn't seem to be able to establish the connection even with the least amount possible. This is the log:

➜  ~ thor --amount 1 ws://localhost:6001/

Thor:                                                  version: 1.0.0

God of Thunder, son of Odin and smasher of WebSockets!

Thou shall:
- Spawn 4 workers.
- Create all the concurrent/parallel connections.
- Smash 1 connections with the mighty Mjölnir.

The answers you seek shall be yours, once I claim what is mine.

Connecting to ws://localhost:6001/



Online               188 milliseconds
Time taken           188 milliseconds
Connected            0
Disconnected         0
Failed               1
Total transferred    0B
Total received       0B

Durations (ms):

                     min     mean     stddev  median max
Handshaking          NaN     NaN         NaN     NaN NaN
Latency              NaN     NaN         NaN     NaN NaN

Percentile (ms):

                      50%     66%     75%     80%     90%     95%     98%     98%    100%
Handshaking          NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN
Latency              NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN

Received errors:

1x                   read ECONNRESET

I also tried with a more detailed websocket url, result is the same:

➜  ~ thor --amount 1 ws://localhost:6001/socket.io/?EIO=3&transport=websocket
[1] 99052
➜  ~
Thor:                                                  version: 1.0.0

God of Thunder, son of Odin and smasher of WebSockets!

Thou shall:
- Spawn 4 workers.
- Create all the concurrent/parallel connections.
- Smash 1 connections with the mighty Mjölnir.

The answers you seek shall be yours, once I claim what is mine.

Connecting to ws://localhost:6001/socket.io/?EIO=3



Online               184 milliseconds
Time taken           184 milliseconds
Connected            0
Disconnected         0
Failed               1
Total transferred    0B
Total received       0B

Durations (ms):

                     min     mean     stddev  median max
Handshaking          NaN     NaN         NaN     NaN NaN
Latency              NaN     NaN         NaN     NaN NaN

Percentile (ms):

                      50%     66%     75%     80%     90%     95%     98%     98%    100%
Handshaking          NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN
Latency              NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN

Received errors:

1x                   read ECONNRESET

[1]  + 99052 done       thor --amount 1 ws://localhost:6001/socket.io/?EIO=3

Any idea what could have gone wrong and what should I look into ? Thanks

Better documentation for Generators

I'm looking to use this as part of a project and I'm required to send specific messages, with which I react in different ways. This is the case for the majority of Web Applications and so would make sense to include more information here.

It would be nice if you gave more obvious examples or documentation as to how we can write our own generators to do what we wish.

WampServer

Any ideas how to bench WampServer?
http://socketo.me/docs/wamp

Generator <file.js>

I wan to covet this in generater file
var sock = new WebSocket('ws://localhost:5000/test', ['test']);
Thank you.

Add option to supply SSL options

I couldn't find if there is a place to supply SSL certificate to open connection, or SSL options in general.

Would like to see this being added.

Does Thor support HTTP cookie?

The question as in title. Sometimes we use cookie for socket authorization. Can we do that with Thor?

Losing a considerable portion of connections due to EADDRNOTAVAIL

I'm trying to stress test a WebSocket server using your software and when I turn the number of connections high, a considerable portion of connections are lost to EADDRNOTAVAIL (10% to 50%).

But this behavior is not consistent. And it seems like once your software hits some condition, it just quits without trying for the rest. For instance, one time I started the test for 20k connections and it took few seconds (10 or 15) to get around 10k connections before it hits the problem (EADDRNOTAVAIL) and once it did, it gave up the rest of test in matter of seconds (one or two) prompting that ~9k connections had the EADDRNOTAVAIL error!

The next time I ran the exact same test (without even restarting the server), it completed without any errors, not even a single one! Even though the mentioned problem is not consistent but it happens often enough.

I'm using thor v1.0.0 and node v0.10.25 in Ubuntu 14.04 with ulimit -n set to 200k. And here's how I run your software:

./thor --masked --amount 50000 --message 8096 --generator ../../../gen.js ws://localhost:8585

And here's its output:

  Opened 42100 connections


Online               116927 milliseconds
Time taken           116927 milliseconds
Connected            42177
Disconnected         0
Failed               7823
Total transferred    9.65MB
Total received       8.02MB

Durations (ms):

                     min     mean     stddev  median max    
Handshaking          278     41268     18709   44746 64915  
Latency              0       71          282      11 2237   

Percentile (ms):

                      50%     66%     75%     80%     90%     95%     98%     98%    100%   
Handshaking          44746   55088   58286   59500   61681   62050   62977   63639   64915  
Latency              11      22      30      35      67      121     1463    1888    2237   

Received errors:

7823x                connect EADDRNOTAVAIL

BTW, since I'm not sure if it's important or not, my WebSocket server gives back a response to the given request from your software.

unable to install

I am unable to install this package using npm i -g thor

it's throwing ssh publicKey err

Installation fail because of moved dependencies

$ sudo npm install -g thor
npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t git://github.com/3rd-Eden/ws.git
npm ERR! 
npm ERR! fatal: remote error: 
npm ERR!   Repository not found.
npm ERR! 
npm ERR! exited with error code: 128

I believe that ws has been moved here: https://github.com/websockets/ws

P.S. Sorry, it is a duplicate..

How to pass query params in thor.

How to send Authorization header?

Is there a way to pass an Authorization header in this?

Websocket subprotocol support

Thor version command returns with bad version number

npm install thor@1.0 -g
thor version

will return the following value: 0.19.1

100% CPU usage

Its taking a lot CPU even with one worker targeting total 10K connections.
Even though I specified -W=1 (one worker), I see lots of node process running, and within 30 mins consumed around 6GB memory.

Cannot tested ws server on GO

Hi all!

Run ws server on GO:

package main

import (
	"net/http"

	"github.com/gobwas/ws"
	"github.com/gobwas/ws/wsutil"
)

func main() {
	http.ListenAndServe(":4375", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		conn, _, _, err := ws.UpgradeHTTP(r, w)
		if err != nil {
			// handle error
		}
		go func() {
			defer conn.Close()

			for {
				msg, op, err := wsutil.ReadClientData(conn)
				if err != nil {
					// handle error
				}
				err = wsutil.WriteServerMessage(conn, op, msg)
				if err != nil {
					// handle error
				}
			}
		}()
	}))
}

And then run test:

thor --amount 100 ws://curex.ll:4375/

Thor:                                                  version: 1.0.0

God of Thunder, son of Odin and smasher of WebSockets!

Thou shall:
- Spawn 4 workers.
- Create all the concurrent/parallel connections.
- Smash 100 connections with the mighty Mjölnir.

The answers you seek shall be yours, once I claim what is mine.

Connecting to ws://curex.ll:4375/
                     
Online               152 milliseconds
Time taken           174 milliseconds
Connected            100
Disconnected         0
Failed               100
Total transferred    0B
Total received       0B

Durations (ms):

                     min     mean     stddev  median max    
Handshaking          15      28            7      25 48     
Latency              NaN     NaN         NaN     NaN NaN    

Percentile (ms):

                      50%     66%     75%     80%     90%     95%     98%     98%    100%   
Handshaking          25      27      32      33      40      44      47      48      48     
Latency              NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN     NaN    

Received errors:

100x                 continuation frame cannot follow current opcode

But receive an error:

Failed 100

WS server works it's for shure. I tested him with client:

<script>
    var ws = new WebSocket('ws://curex.ll:4375/');
    ws.onmessage = function(evt) { alert(evt.data); };
    ws.onopen = function (event) { ws.send('test!'); }
</script>

Support benchmarking socket.io sockets

Connections are not persistent

When Thor sends messages to the server, after messages has been responded, the connection will close. But this happens even before all connections have been established, which means some connections are still trying to send messages to the server and at the same time, some connections have been closed.

minor bug:here are to -M options.

-M, --messages messages to be send per connection
-M, --masked send the messaged with a mask

Is this Project Maintained ?

The last commit dates back to 2 years and the last merged PR dates back to 1 year. I am just curious is this project maintained ?

Thanks

couldn't see any result

I install thor following the instruction. But when I input

thor --amount 1000 --messages 100 ws://192.168.37.1:8888

I could'n see result, as the following picture show.

"Thou shall" is bad grammar

Currently, thor's output contains "Thou shall", which is like writing "he hate" instead of "he hates".

The proper conjugation is "Thou shalt". (Wikipedia's Thou page does a good job of explaining this in simple terms. Shall is one of the five irregular verbs where you don't just add whichever of -st or -est sounds better.)

Option to pass custom headers

Option to pass a custom header to be used for websocket handshake request. Will be useful for web socket which does authentication in Oauth2 standard.

Parse Error

Hi,
I'm having an issue with thor.
It seems not to be able to open connections and the result is that I obtain NaN on Handshake and Latency and "Parse Error" as received error. Obviusly I have a 100% rate of failure..

I report here the stacktrace of the error:

console.trace()
Trace
at repl:1:10
at REPLServer.self.eval (repl.js:110:21)
at Interface. (repl.js:239:12)
at Interface.emit (events.js:95:17)
at Interface._onLine (readline.js:202:10)
at Interface._line (readline.js:531:8)
at Interface._ttyWrite (readline.js:760:14)
at ReadStream.onkeypress (readline.js:99:10)
at ReadStream.emit (events.js:98:17)
at emitKey (readline.js:1095:12)
undefined

Thanks

Variable URL

First of all, thanks a lot for the great tool for benchmarking websocket applications.

I have a single limitation that's keeping me from testing my app.

My URL is: ws://myserver/E5B21AF1-C86E-4A08-8DD0-92EE01E02081

What this mean is that for a meaningful benchmark I need to vary the URL for each connection that Thor does. Is that feasible today? Do you think it's a capability that's easy to add?

Latency shows as NaN

Thor is returning NaN for latency values.

Unable to "npm install -g thor"

I get the following error on a mac:

Command failed: git clone --template=/Users/JD/.npm/_git-remotes/_templates --mirror git://github.com/3rd-Eden/ws.git /Users/JD/.npm/_git-remotes/git-github-com-3rd-Eden-ws-git-2e9f11e5

Bad Data Benchmark options

the default option for ws is 'per-message-deflate' this makes the packets that are 1024 bytes of lots of zeros only 16 bytes

b\030\005\243\024\214T\000\000\000\000\377\377\000

Which sort of defeats the data size throughput measurement; you should disable this option when creating a new ws client conneciton.
Or generate packets that don't compress to nothing... maybe build them with randomness... hard to gauge real world data and how much it would compress, but it would be somewhere between maximal (all the same byte) and minimal (math.random() for instance)