Comments (12)
I can use ert within the test, but do you also want it to not be run from the command line to execute?
from emacs-websocket.
I've attached a build log for reference.
emacs-websocket_1.12+1.g74e4b82-1_amd64-2019-11-24T01:05:53Z.build.zip
The relevant section is L733 to L796, and L734 is where we run ert-run-tests-batch-and-exit
to test the package as-installed, with the variables isolated to only the package that is being tested. For the purposes of the Debian package it's enough to call testserver.py
in the appropriate ERT test unit, and/or fork a process that multiple units can test against. Of course you're welcome to use any other additional test runners :-) (eg: may projects have a "test" target in a Makefile or use Cask, but we ignore those)
It looks like websocket-server-close
, websocket-to-bytes
, and maybe others also need to be more strict, because it seems like these units should fail. Additionally I'm surprised that the error output isn't attached to an error-handling test unit. If it's an async process, maybe a test unit later in the series could catch the output?
Finally, it would be nice to move away from the obsolete flet
macro, to either ‘cl-flet’ or ‘cl-letf’. That's "wishlist" priority though, I don't want to be a PITA!
P.S. There's already been an uncharacteristically early positive response to my ITP (intent to package email), so I think you'll appreciate knowing that people are excited about seeing your work included in Debian :-)
from emacs-websocket.
P.S. what I mean by "error-handling" test unit is a test unit designed to test the libraries error-handling capabilities. eg: test for expected failure error string, and pass if it matches or fail if the error string isn't triggered when the failure case is triggered.
from emacs-websocket.
Can you comment further on why websocket-server-close
and websocket-to-bytes
should fail? I'm not sure what you are referring to - the ert tests you are referring to work, because the code under test works. I took another look at the code but didn't see anything obviously wrong.
About error-handling tests, I don't personally believe it is all that useful to test actual error messages. We do test that the library does error out when it is supposed to.
from emacs-websocket.
from emacs-websocket.
I think there may be some confusion here. There's two main testing files in the websocket package: the websocket-functional-test.el
file and the websocket-test.el
. The unit tests you mention are in websocket-test.el
, and, like all the ert tests in the package, are true unit test, not dependent on anything being set up outside of the tests themselves. This is why they mock out functionality with flet (which I agree should be cleaned up eventually). This is why they pass without a Tornado server being run, because they were never written to require one.
Please let me know if I'm misunderstanding your point.
from emacs-websocket.
PTAL at commit 69ee80a and let me know if it seems acceptable to you. It should fulfill the requirements as described.
from emacs-websocket.
I've merged this into the master branch now. Thank you for the suggestion!
from emacs-websocket.
@ahyatt, sorry for the delay replying; I got lost in the backlog of work and messages. Thank you very much for working on this! It looks good, with one issue:
The new method does not work on standalone systems because it depends on connectivity to echo.websocket.org, eg:
https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/emacs-websocket.html and search for "Test websocket-client-with-remote-server backtrace"
I'll understand if this isn't something you're interested in supporting ;-) It would be nice to have though, since the alternative is disabling this test on Debian infrastructure.
Thank you!
Nicholas
from emacs-websocket.
I think it's valuable to have functional tests that use an outside source of truth such as echo.websocket.org. Disabling the functional tests in Debian seems like a reasonable idea. We do the opposite in the emacs elpa repository - we don't include the unit tests, since there have been contributions there I have authors that haven't signed the FSF papers.
from emacs-websocket.
Sorry for the delay again; it took a while to find the time to look into this!
I think it's valuable to have functional tests that use an outside source of truth
Agreed :-)
such as echo.websocket.org.
Unless one is testing DNS or routing, I don't see how this is more valuable than a server (that is independent of emacs-websocket) running on localhost. For example, after reading up on what other software uses to validate its websocket support I found that the Autobahn|Testsuite seems to the standard by which all others are measured. For the purposes of CI, it's available as the python3-autobahn
package as early as Ubuntu bionic (18.04LTS) and Debian stretch (Debian 9). There are other server implementations that declare compliance with this testsuite that may be more convenient to use, but I haven't looked into them in any kind of depth.
Disabling the functional tests in Debian seems like a reasonable idea.
Pulling out teeth also seems like a reasonable idea ;-) and I think we'll agree that context and consideration of the available alternatives is important.
We do the opposite in the emacs elpa repository - we don't include the unit tests, since there have been contributions there I have authors that haven't signed the FSF papers.
If an author has disappeared, has refused to sign the papers, objects to the FSF's invasive identity verification, or the concept of copyright assignment to a 3rd party, then no other options exist (other than a clean-room rewrite by someone who hasn't seen the existing unit tests...but that's pedantic, no?).
If you don't want to work on this, would you accept a PR that enables the use of an alternative "outside source of truth" such as Autobahn running on localhost? I imagine it will require these: a variable for the URL; a variable for a "server-start-command" and another for "server-kill-command"; possibly a third variable to hold the server's PID; functions to make use of these, plus a branch to take the "server-start-command" path rather than the echo.websocket.org one if the "websocket-test-url" variable is set. Everything apart from the URL allows Emacs to manage the server--to minimise dependencies on external frameworks.
Thanks,
Nicholas
from emacs-websocket.
I previously used Python's Tornado webserver to test out websocket in our functional test. Yes, it was nice to have something purely on-machine, but I found the disadvantages outweighed the advantages, the main disadvantage being the user needed to have something installed to test with. Installing things is not always so easy. For most manual runs of the functional tests, the dependency was a source of failures and didn't buy a lot.
However, if you wanted to set up some sort of continuous functional test using Autobahn because this is useful for Debian, that's fine with me. Send over a PR and I will take a look.
One idea I never implemented is to use the emacs websocket client to talk to the server. Benefit: no outside dependencies at all. Drawback: you can't really be sure you are testing anything.
from emacs-websocket.
Related Issues (20)
- websocket-frame-text error HOT 1
- Add release tags HOT 3
- Custom headers in handshake HOT 4
- How to handle an error occured in `websocket-outer-filter`? HOT 6
- A single message ws.send(data); comes in as two messages HOT 5
- Speed feedback HOT 10
- Randomly losing connections under emacs-ipython-notebook with version 1.11 HOT 5
- Error (websocket): in callback `on-message': JSON readtable error HOT 4
- Setting case-fold-search to nil results "Incorrect handhake from websocket" error HOT 7
- declares GPL-3+, but provided full-text license is GPL-2+ HOT 3
- Please tag a 1.12 release HOT 1
- TLSv1.3 HOT 1
- Branch rename breaks package installation HOT 6
- websocket-test.el: ‘flet’ is an obsolete macro (as of [emacs] 24.3) HOT 3
- websocket-to-bytes signal type error HOT 8
- Support connecting through proxies HOT 7
- infinite failure loop - emacs locked up HOT 18
- error 403 with ein over proxy HOT 14
- Push to ELPA? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from emacs-websocket.