ahyatt / emacs-websocket Goto Github PK

Here is the full tracebacks:
https://gist.github.com/3046935

I checked in Emacs 24.1.50.1 and 23.1.1. Some machine information:

% uname -m
i686
% lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 10.10
Release:        10.10
Codename:       maverick

In 64 bit machine, the test was fine. Note that websocket.el actually works in the 32 bit machine. I haven't notice the problem at all until now.

Furthermore, @epatters and @Boothead report that loading websocket.el fails with load-with-code-conversion: Arithmetic overflow error: "4294967296" while reading the number 4294967296. Note that I have no problem loading websocket.el even though most-positive-fixnum (=536870911) is smaller than 4294967296.

The original report is here: tkf/emacs-ipython-notebook#8

Not a big problem, but emacs-websocket currently signals all errors using the (error) form. This makes it somewhat difficult for client code to handle errors because error types can't be distinguished and additional error data (e.g. HTTP status code) can't be extracted short of parsing the error message.

It would be nice if errors were instead signalled using a custom symbol and with extra data added as-is, i.e. using the (signal) form.

Since you already have a human-readable description for each error condition, it probably couldn't hurt to include it in the extra data so that clients still have an easy way of presenting the error to the user.

On a side note, it would be great if websocket-verify-response-code would pass on the whole HTTP response in case of error. The body sometimes carries useful extra information on the error condition, and the headers might be needed to interpret the body.

I'm developing an interactive websocket client to use for testing different APIs, but one of them requires the HTTP request to have a custom header. It would be great if there were a dynamic variable websocket-extra-headers that websocket-create-headers used to insert extra headers on the initial request.

What I want while using websocket.el is debugging support. Currently errors from callback does not reach to the Emacs debugger because it is suppressed by the filter function. I guess it should be possible not to catch errors in filter function while not breaking parser state of websocket.el.

As package.el uses header comments to get some information, I suggest to add the following comments after the "Maintainer" part.

;; Keywords: Communication
;; Version: 1.0

See also: melpa/melpa#185

I guess I mentioned it somewhere, but it also would be nice to have a constant variable to store version number. So that websocket.el user can detect future API change (if any).

(defvar websocket-version "1.0"
  "Websocket version number")

I found a blog post by @syohex claiming that you need to encode string explicitly to send a multibyte string, like this:

(websocket-send-text websocket (encode-coding-string "あいうえお" 'raw-text))

Probably we can call encode-coding-string in websocket-send-text?

Original blog post (Japanese):
http://d.hatena.ne.jp/syohex/20120827/1346076494

29d8f2f fixed sending non-ascii characters. But receiving them still appears to have a problem. I think the following test code should work, but it results

Server received text!
ws frame: "\303\244\302\275\302\240\303\245\302\245\302\275"
cl--assertion-failed: Assertion failed: (equal (car wstest-msgs) "你好")

Is it caused by the implementation of websocket.el, or am I misunderstanding the usage?

(setq wstest-closed nil)
(let ((server-conn (websocket-server
                    9998
                    :host 'local
                    :on-message (lambda (ws frame)
                                  (message "Server received text!")
                                  (websocket-send-text ws
                                                       (websocket-frame-payload frame)))
                    :on-open (lambda (_websocket) "Client connection opened!")
                    :on-close (lambda (_websocket)
                                (setq wstest-closed t)))))

  (setq wstest-msgs nil
        wstest-ws
        (websocket-open
         "ws://localhost:9998"
         :on-message (lambda (_websocket frame)
                       (push (websocket-frame-payload frame) wstest-msgs)
                       (message "ws frame: %S" (websocket-frame-payload frame)))))

  (assert (websocket-openp wstest-ws))
  (websocket-send-text wstest-ws "你好")
  (sleep-for 0.3)
  (assert (equal (car wstest-msgs) "\344\275\240\345\245\275"))
  (websocket-server-close server-conn))
(assert wstest-closed)
(websocket-close wstest-ws)

As Emacs 25 is released, people upgrading to packages that depend on web-socket will be expecting this to work (hopefully). Currently the websocket tests fail on emacs 25.

Debugger entered--Lisp error: (file-error "make client process failed" "Connection refused" :name "websocket to ws://127.0.0.1:9999" :buffer nil :host "127.0.0.1" :service 9999 :nowait nil)
  make-network-process(:name "websocket to ws://127.0.0.1:9999" :buffer nil :host "127.0.0.1" :service 9999 :nowait nil)
  websocket-open("ws://127.0.0.1:9999" :on-message (lambda (_websocket frame) (setq wstest-msgs (cons (websocket-frame-text frame) wstest-msgs)) (message "ws frame: %S" (websocket-frame-text frame)) (error "Test error (expected)")) :on-close (lambda (_websocket) (setq wstest-closed t)))
  (defvar wstest-ws (websocket-open "ws://127.0.0.1:9999" :on-message (function (lambda (_websocket frame) (setq wstest-msgs (cons (websocket-frame-text frame) wstest-msgs)) (message "ws frame: %S" (websocket-frame-text frame)) (error "Test error (expected)"))) :on-close (function (lambda (_websocket) (setq wstest-closed t)))))
  eval-buffer()  ; Reading at buffer position 2028
  funcall-interactively(eval-buffer)

Let me know if you cannot reproduce this, I tested with an empty config.

first, sorry for my poor English.

I write a websocket server in emacs with your awesome project.
the code is

(require 'websocket)
(eval-when-compile (require 'cl))

;;; the status of websocket-client closed
(setq ws-closed nil)

(let ((server-conn (websocket-server
		    9988
		    :Host "0.0.0.0"
		    :on-message (lambda (ws frame)
				  (message "Server received a messge")
				  (websocket-send-text
				   ws (websocket-frame-text frame)))
		    :on-open (lambda (_ws) (message "client connected open"))
		    :on-close (lambda (_ws) (setq ws-closed t))))))

and i have a client use javascript

var ws = new WebSocket('ws://localhost:9988')

ws.onopen = function() {
  ws.send("{type: 'identified', symbol: 'web'}")
};

ws.onmessage = function(event){
  console.log(event.data);
}

while(true) {
  ws.send()
}

but i received this message:

Error (websocket): in callback `on-message': Symbol's function definition is void: websocket-frame-text

please help me. Thank you very much!

In the RFC (see http://tools.ietf.org/html/rfc6455#page-58), it is possible to send and accept extensions, but this isn't implemented yet. We really need to implement this to have full compatibility with the RFC.

This issue may be the continuation of the last one (issue #31).

Coding system setting is still broken on Windows (mine is Windows 8). The symptom is that Emacs shows the websocket in ready-state open, but browser shows connecting (Chrome). Therefore the outgoing message to the browser must have been corrupted. After setting the encoding to be also binary the problem appears to be fixed.

diff --git a/websocket.el b/websocket.el
index 7ba3a53..2dfd73a 100644
--- a/websocket.el
+++ b/websocket.el
@@ -843,7 +843,7 @@ connection, which should be kept in order to pass to
     (set-process-coding-system client 'unix 'unix)
     (process-put client :websocket ws)
     (set-process-filter client 'websocket-server-filter)
-    (set-process-coding-system client 'binary)
+    (set-process-coding-system client 'binary 'binary)
     (set-process-sentinel client
      (lambda (process change)
        (let ((websocket (process-get process :websocket)))

We should support wss, secure websockets.

Hi Andrew,

I hope this message finds you well.

I was in the process of updating some of my Emacs packages this morning, and noticed I couldn't update because of a problem cloning this repository. I suspected a branch rename might be the issue, and found the following issue from a fellow Doom user where a workaround was shared:

doomemacs/doomemacs#3435

Given this is a breaking change I wonder if you'd be open to adding a note to the README mentioning this? Perhaps with a guide on how to update your local repository?

When the connection is closed by the remote side the sentinel functions call delete-process on the connection, but does not call kill-buffer ala websocket-close. Excess buffers can build up over long emacs sessions.

If I'm understanding correctly, ELPA could perhaps catch up to 11-2020 per MELPA. Also, I was thinking of using this to create something using obs-websocket. I'll make a PR for the brag-list if my creation comes to life :)

You can check it by running websocket-functional-test.el.
I checked in Emacs 23.1.1

It seems Emacs 23's open-network-stream does not have the optional PARAMETERS argument:
https://github.com/emacsmirror/emacs/blob/emacs-23/lisp/subr.el#L1757
https://github.com/emacsmirror/emacs/blob/master/etc/NEWS#L1887 (News in Emacs 24)

Hi,

I'm in the process of evaluating Atomic Chrome for Emacs for inclusion into the Debian archive. Emacs Websocket is a dependency of it. While reviewing the latter I noticed that all lisp files declare GPL-3+, but the provided full-text of the GPL is still 2+. For your convenience, here is a link to GPL-3+:

https://www.gnu.org/licenses/gpl-3.0.txt

according to @dickmao in tkf/emacs-request#193 (comment),
the bug described in tkf/emacs-request#193 and millejoh/emacs-ipython-notebook#754 is coming from websocket.
Please let me know if I could be of any assistance in debugging.
thank you.

Hi, it's good to see the rfc6455 branch is moving forward!

In the current master, websocket.el fails to filter out the handshake reply when the last line (the 16 bytes MD5 sum) contains "\0". So websocket-open probabilistically fails to start "healthy" connection.

I guess we won't fix it as discussed in #7. I thought I'd post it just for sharing the current problem because it took me a while to see what is wrong.

This trips up EIN when trying to connect to password-protected notebooks in IPython. emacs-websocket also needs to specify the port along with the URL in the header for the connection to be succesful in IPython.

I've coded up a crude solution (87ab400) over at my fork.

Hi,

I noticed that websocket.el declares version 1.12, but 1.10 is the latest available tag. Would you please consider tagging 1.12?

Thanks!
Nicholas

Not 100% on all the details, but I don't think emacs-websocket handles ping messages correctly. Per the RFC (and apparently Tornado's implementation of the protocol) a ping can have a payload which must be echoed in the pong response. I don't think Tornado cares if the ping payload is echoed, but it certainly expects the pong response to have a payload, even if said payload is empty.

I have hacked away at this a bit and seem to have this working on my fork - want me to send a pull request?

If this library is ever to be added to emacs, it should not have the cl package required.

…
passed  28/33  websocket-server-close
Invalid client headers found in: 

   passed  29/33  websocket-server-filter
Test websocket-to-bytes backtrace:
  signal(ert-test-failed (((should-error (websocket-to-bytes 536870912
  ert-fail(((should-error (websocket-to-bytes 536870912.0 8) :type (qu
  ert--should-error-handle-error((lambda nil form-description-331) (wr
  (condition-case -condition- (unwind-protect (setq value-329 (apply f
  (let ((errorp332 nil) (form-description-fn-333 (function (lambda nil
  (let (form-description-331) (let ((errorp332 nil) (form-description-
  (let ((value-329 (quote ert-form-evaluation-aborted-330))) (let (for
  (let* ((fn-327 (function websocket-to-bytes)) (args-328 (condition-c
  (lambda nil (let* ((fn-312 (function equal)) (args-313 (condition-ca
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name websocket-to-bytes :documentation nil
  ert-run-or-rerun-test(#s(ert--stats :selector t :tests [#s(ert-test 
  ert-run-tests(t #f(compiled-function (event-type &rest event-args) #
  ert-run-tests-batch(nil)
  ert-run-tests-batch-and-exit()
  eval((ert-run-tests-batch-and-exit))
  command-line-1(("-l" "package" "--eval" "(add-to-list 'package-direc
  command-line()
  normal-top-level()
Test websocket-to-bytes condition:
    (ert-test-failed
     ((should-error
       (websocket-to-bytes 536870912.0 8)
       :type 'websocket-frame-too-large)
      :form
      (websocket-to-bytes 536870912.0 8)
      :condition
      (wrong-type-argument integerp 536870912.0)
      :fail-reason "the error signaled did not have the expected type"))
   FAILED  30/33  websocket-to-bytes
Websocket client connection: Host header not found
Websocket client connection: Upgrade: websocket not found
Websocket client connect: No key sent
Websocket client connect: No key sent
Websocket client connection: HTTP/1.1 not found
   passed  31/33  websocket-verify-client-headers

Full build log: https://tests.reproducible-builds.org/debian/rb-pkg/unstable/i386/emacs-websocket.html

Hi,

Thanks again for maintaining this software. As mentioned in another issue I'm investigating packaging this library for Debian. Would you please consider adapting websocket-functional-test.el to use ert-deftest? I maintain the Debian package for Elpy, so I know it's possible ;-) As for the "why" our Emacsen Team exclusively uses ERTs for QA and CI.

Sincerely,
Nicholas

emacs-websocket doesn't support connecting through HTTP proxies. It probably should read url-proxy-services and use that to automatically go through an HTTP proxy for the websocket connnection if one is configured.

I don't know the websocket protocol but I'm fairly motivated to get this working so I might try doing this myself - do you think it will be much work?

As Takafumi has suggested, this API should hew as closely as possible to the w3c API.

Right now we don't parse these at all.

See: http://dev.w3.org/html5/websockets/#the-websocket-interface

Problem is that you will need to parse handshake reply from server. I think implementing handshake parser for current supported version (draft 76) is useless as it is old. Do you think we should wait until RFC 6455 implementation (#1) is finished? For now, I am working around this problem simply sleep after connecting server and call "onopen" function.

Can we package this for marmalade? It looks like a good implementation.

I was considering writing an implementation on top of Elnode... but then I realized you had a server as well so it seems a bit silly to add another.

I have an interesting use case for this so I'd really like it packaged.

What do you think?

I noticed the following compiler warning.

In websocket-verify-headers:
websocket.el:472:53:Warning: function `mapcan' from cl package called at runtime

I am not sure, but I think it can be a problem when user do not
require cl at runtime, because websocket.el requires cl only at
compile time.

I am unable to connect to a simple server from javascript:

The socket gets immediately closed from javascript under Chrome. Connecting from netcat keeps the socket open. I suspect it's something with the handshake?

Running emacs 24.1. I am executing the elisp by eval-buffer on a scratch buffer.

(require 'websocket)
(setq websocket-debug t)

(defun on-message (socket frame)
(message "message"))

(defun on-error (socket frame)
(message "error"))

(defun on-open (websocket)
(message "websocket open %S" websocket))

(defun on-close (websocket)
(message "websocket close"))

(setq ws-server
(websocket-server
9999
:on-message 'on-message
:on-open 'on-open
:on-error 'on-error
:on-close 'on-close))

var ws = new WebSocket("ws://127.0.0.1:9999/");
ws.onopen = function(e) { console.log("opened"); }
ws.onerror = function(e) { console.log("error"); }
ws.onmessage = function(e) { console.log("message"); }
function (e) { console.log("message"); }

Thank you!
Joe

With this server:

(defvar server
  (websocket-server
   9009
   :host 'local
   :on-message (lambda (ws frame)
                 (let* ((b64-image (websocket-frame-text frame))
                        )
                   (with-current-buffer (get-buffer-create "*image-text-output*")
                     (insert "\nCHUNK:\n")
                     (insert b64-image))
                   )
                 )
   :on-open (lambda (ws) (message "Client connected."))
   :on-close (lambda (ws) (message "Client disconnected."))))

If I send

ws = new WebSocket("ws://localhost:9009/");
      ws.onopen    = function()    {
        ws.send(data);
      };

where data comes from here, :on-message is called twice, one for each chunk.

That's not what I expected. I'd expect the whole .send() payload to come in as one message. I can implement my own buffering, but I thought that the websocket API was supposed to take care of this?

Every once in a while, a *Warnings* buffer pops up, displaying error messages such as:

Error (websocket): in callback `on-message': JSON readtable error: 89
Error (websocket): in callback `on-message': JSON readtable error: 65
Error (websocket): in callback `on-message': JSON readtable error: 66

This is with websocket 20190621.54 from MELPA with:
GNU Emacs 26.1 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.4)
of 2019-02-03, modified by Debian

The functional tests do not pass on Windows. I've found three issue so far. (Let me know if you want me to open these as different issues.)

• The call to stop-process fails on Windows with the error "No SIGTSTP support". Potential fix is to skip stop-process if Windows is detected was merged with #43..
• (sleep-for n) doesn't pause reliably for me (I'm using GNU Emacs 24.5.1). This may be due to a known bug.
• The secure websocket test fails to connect with the message "gnutls.c: 1 GnuTLS library not found". Probably want to test whether there's TLS support and only run the test if there is.

When websocket connection is invalid. websocket-outer-filter will force close the connection and signal an error. What should I do if I want to handle that error? (For example, I want to try to reconnect when an error occured)

My solution is create a new filter for the process but I think it's dirty hack

Documented in issue #600 over at ein. Apparently the accept header check in websocket-verify-headers needs to be case-insensitive along with the upgrade header check.

I cannot figure why a case-sensitive search would break this check, so I am a little concerned this is a Tornado/Jupyter specific issue in which case it would be better for me to work around the issue inside of ein (somehow).

It would be nice for this repository to have its releases tagged (at least version 1.7), so that packages can be generated for melpa-stable (see melpa/melpa#2824). Right now only the latest git master is packaged.

Hi,

Prioritise this however you'd like. I'm only reporting it because someday the obsolete macro may be removed.

websocket-test.el: ‘flet’ is an obsolete macro (as of 24.3); use either ‘cl-flet’ or ‘cl-letf’; use either ‘cl-flet’ or ‘cl-letf’. It seems like it work be orthogonal to converting websocket-functional-test.el and websocket-test.el to also use lexical-binding.

P.S. I'm also curious why you're using explicit dynamic binding by using defvar instead of setq. To be fair, my bias is "use lexical, whenever possible" ;-)

Thanks,
Nicholas

The fourth test for websocket-to-bytes tests is failing on my 32-bit machine. If I had to guess, commit d29b171 looks like it introduced the failure. lsh doesn't like it when you give it a float.

Here's the backtrace I get when evaluating (websocket-to-bytes 536870912 8). Notice that I did not turn 536870912 into a float, but the reader did that for me, like what is happening in the test.

Debugger entered--Lisp error: (wrong-type-argument integerp 536870912.0)
  lsh(536870912.0 -32)
  (let ((hi-32bits (lsh val -32)) (low-32bits (if (= 0 (expt 2 32)) val (logand 4294967295.0 val)))) (if (or (> hi-32bits 0) (> (lsh low-32bits -29) 0)) (progn (signal (quote websocket-frame-too-large) val))) (bindat-pack (quote ((:val vec 2 u32))) (list (cons (quote :val) (vector hi-32bits low-32bits)))))
  (progn (let ((hi-32bits (lsh val -32)) (low-32bits (if (= 0 (expt 2 32)) val (logand 4294967295.0 val)))) (if (or (> hi-32bits 0) (> (lsh low-32bits -29) 0)) (progn (signal (quote websocket-frame-too-large) val))) (bindat-pack (quote ((:val vec 2 u32))) (list (cons (quote :val) (vector hi-32bits low-32bits))))))
  (if (= nbytes 8) (progn (let ((hi-32bits (lsh val -32)) (low-32bits (if (= 0 (expt 2 32)) val (logand 4294967295.0 val)))) (if (or (> hi-32bits 0) (> (lsh low-32bits -29) 0)) (progn (signal (quote websocket-frame-too-large) val))) (bindat-pack (quote ((:val vec 2 u32))) (list (cons (quote :val) (vector hi-32bits low-32bits)))))) (bindat-pack (list (list (quote :val) (cond ((= nbytes 1) (quote u8)) ((= nbytes 2) (quote u16)) ((= nbytes 4) (quote u32)) (t (error "websocket-to-bytes: Unknown NBYTES: %s" nbytes))))) (list (cons (quote :val) val))))
  websocket-to-bytes(536870912.0 8)
  eval((websocket-to-bytes 536870912.0 8) nil)
  eval-last-sexp-1(nil)
  eval-last-sexp(nil)
  call-interactively(eval-last-sexp nil nil)
  command-execute(eval-last-sexp)

Relevant emacs info

In GNU Emacs 24.5.1 (i686-pc-mingw32)
 of 2015-04-11 on LEG570
Windowing system distributor `Microsoft Corp.', version 6.3.9600
Configured using:
 `configure --prefix=/c/usr --host=i686-pc-mingw32'

I have haphazardly documented in millejoh/emacs-ipython-notebook#559. The problems seem to start sometime after commit 0d96ba2.

Unfortunately I have very little information to contribute as all the debug buffer states is [WS] State change to connection broken by remote peer. This does appear to be an issue with the current release (1.11) of websocket, though, since if I revert to an earlier version I stop seeing the disconnects.

Let me know if there is anything else I can do to help track down the cause of this.

websocket-ensure-connected triggers an infinite reconnect loop.

Please see millejoh/emacs-ipython-notebook#750 for details.

I would appreciate guidance in debugging this.

Thank you!

See: http://tools.ietf.org/html/rfc6455

It appears that websocket cannot communicate with a server using TLSv1.{2,3}. I had to enable TLSv1.

debug message from nginx:

`client sent invalid method while reading client request line, client ... request: "\210^@"``

A personal observation: when sending a payload of a PNG image from a web browser on localhost to the emacs-websocket server, the websocket-read-frame function takes 452ms to consume it.

websocket-read-frame: 0ms [29 times]
websocket-read-frame: 452ms

@ahyatt Are you aware of some obvious low-hanging fruit in the performance? It seems like O(n) time, perhaps it could be mitigated?

For my use-case I'm probably going to switch to making a POST request to Emacs and use a raw socket to consume the binary chunks. Theoretically it should take maybe a millisecond to receive 100KB. It seems like the websocket protocol is a bit of a pain to implement.