Lightweight, scriptable browser as a service with an HTTP API
License: BSD 3-Clause "New" or "Revised" License
Splash is a javascript rendering service with an HTTP API. It's a lightweight browser with an HTTP API, implemented in Python 3 using Twisted and QT5.
It's fast, lightweight and state-less which makes it easy to distribute.
Documentation is available here: https://splash.readthedocs.io/
To use Splash with Scrapy, please refer to the scrapy-splash library.
Open source support is provided here in GitHub. Please create a question issue.
Commercial support is also available by Scrapinghub.
First request to scrapinghub.com finishes successfully, but the second leads to segfault on my machine. Python 2.7.5, Qt 4.8.5 and PyQT 4.10.2 installed using homebrew.
It seems that has something to do with custom fonts: https://gist.github.com/kmike/311343897a5051890c12
Looks more like PyQT/QT bug, but anyways.
Would authentication be a welcomed feature or do you suggest it's handled elsewhere? I did some quick poking around and it doesn't seem too difficult to add. I'm still evaluating splash but happy to do it if I move forward with it!
Some sites are quite bloated, and it takes a long time to fully load them including all ads and tracking codes; sometimes pages don't even fit into timeout.
What do you think about adding "network access profiles" similar to "proxy profiles" with a whitelist and blacklist for network access?
Implementation should be quite straightforward; it'll probably involve extending SplashQNetworkAccessManager and making it pluggable.
only HTML is supported so far
What do you think about setting defaults.WAIT_TIME to something like 0.5?
This has 2 advantages:
I run python -m splash.tests.stress and 0.5 wait didn't result in a slowdown, as expected.
This change could bring up the memory required by splash because more requests are stayed in memory at the same time; also, it will require more processing power to execute javascript for 0.5s in real-word webpages.
This feature was removed here: #5
I was able to reproduce this issue. But setting non-zero "wait" parameter (implemented here: #13) fixed this problem for me.
What do you think about the following plan?
Currently tests can fail or behave incorrectly when splash instance is running. It could be better to use temporary ports for splash server, mock server and proxy server.
>>> import socket
>>> s = socket.socket()
>>> s.bind(("", 0))
>>> s.getsockname()
('0.0.0.0', 54485)
For proxy server it may require more code because its port is in config that is stored in VCS; it may be OK not to fix this issue for proxy server.
"No thumbnail" thumbnail always come in 320px x 240px form.
I installed splash on a bare instance of Ubuntu (EC2). After getting everything up and running, I noticed I was receiving this error on every request:
Error 5: the operation was canceled via calls to abort() or close() before it was finished.
The error is documented here as QNetworkReply::OperationCanceledError
I'm invoking splash with this command:
curl http://localhost:8050/render.html?url=http://www.getsidewalk.com
Full log below:
2014-01-10 21:39:49+0000 [network] Error 203: the remote content was not found at the server (similar to HTTP error 404) (http://www.getsidewalk.com/assets/layouts/default/wing-left-eaa1fe4bec1a41ce80b911bff557e710.png)
2014-01-10 21:39:49+0000 [network] Error 203: the remote content was not found at the server (similar to HTTP error 404) (http://www.getsidewalk.com/assets/layouts/default/wing-right-baa18b10d7709bc86454d66ecaded0d6.png)
2014-01-10 21:39:49+0000 [stats] {"maxrss": 63352, "load": [0.0, 0.01, 0.05], "fds": 27, "qsize": 0, "rendertime": 0.6076819896697998, "active": 0, "path": "/render.html", "args": {"url": ["http://www.getsidewalk.com"]}, "_id": 29090232}
2014-01-10 21:39:49+0000 [-] 127.0.0.1 - - [10/Jan/2014:21:39:49 +0000] "GET /render.html?url=http://www.getsidewalk.com HTTP/1.1" 200 3910 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"
2014-01-10 21:39:49+0000 [network] Error 5: the operation was canceled via calls to abort() or close() before it was finished. (http://www.google-analytics.com/collect?v=1&_v=j15&a=18518730&t=pageview&_s=1&dl=http://www.getsidewalk.com/&ul=en-us&de=UTF-8&dt=Sidewalk&sd=8-bit&sr=640x480&vp=1024x768&je=0&_u=ME~&cid=732692695.1389389989&tid=UA-41559438-1&z=1586397198)
Note that I do not get this error when running it on OS X
Any idea how to fix this?
We need to support retrieving the content of iframes. Currently, render.html only returns the main frame content.
We need to limit the number of renders that run concurrently, to avoid getting the servers overloaded when visiting a page with many thumbnails on the panel.
We need to use a single instance of QNetworkDiskCache to avoid reloading cache on each request.
My simple fix on 11a9b8d didn't work because the network manager takes ownership of the QNetworkDiskCache instance and deletes it once it dies.
I suspect the solution would be around having a single network manager instance, which may require to rethink or rewrite the allowed_domains functionality.
Ideas welcome!
I'm trying to get the debian package installed from the github repo. After quite a bit of googling around, it seems I have gotten dpckg install to work and yet splash is still not recognized. Here are the steps I took:
I was under the impression that this would install splash so I can start it as an upstart job but "start splash" does nothing
I see the upstart in /etc/init/splash.conf.dpkg-new
but not recognized in initctl list even after reloading conf sudo initctl reload-configuration
So then I do init-checkconf /etc/init/splash.conf.dpkg-new to get
ERROR: file must end in .conf
Then I change the file to end in conf, reload the config again sudo initctl reload-configuration
and now I see it in the list initctl list as splash stop/waiting
I start splash
$ sudo start splash
splash start/running, process 32079
Looks successful but not according to the logs!
chown: cannot access `/var/log/splash': No such file or directory
chown: cannot access `/etc/splash/proxy-profiles': No such file or directory
chown: cannot access `/var/cache/splash': No such file or directory
chown: cannot access `/etc/splash/js-profiles': No such file or directory
chown: cannot access `/var/log/splash': No such file or directory
chown: cannot access `/etc/splash/proxy-profiles': No such file or directory
chown: cannot access `/var/cache/splash': No such file or directory
chown: cannot access `/etc/splash/js-profiles': No such file or directory
...
So I manually made the directories and then started splash and it worked. I'd be happy to update the README with install instructions but I'm thinking that maybe I'm just not knowledgable about idiosyncrasies regarding installing deb packages from source. Can you please comment and let me know:
Again, if the current installation process involves all this work, I can boil it down to a few steps and update the README
When rendering some pages, it works only once, and then it is not possible to retrieve anything else from the same domain, unless i restart the server.
curl "http://33.33.33.10:8050/render.png?url=http://panel.scrapinghub.com/" > render.png
first time ok. Second time, response is 0 length, and the splash2 process output shows the following:
[33.33.33.10] out: QPainter::begin: Paint device returned engine == 0, type: 3
[33.33.33.10] out: QPainter::setRenderHint: Painter must be active to set rendering hints
[33.33.33.10] out: QPainter::setBrush: Painter not active
[33.33.33.10] out: QPainter::pen: Painter not active
[33.33.33.10] out: QPainter::setPen: Painter not active
[33.33.33.10] out: QPainter::end: Painter not active, aborted
If that helps to give a clue, I first found the problem using splash2 for rendering html code retrieved from my vm HS server and i solved using the parameter baseurl with the base url of the html code i retrieved. So i uploaded an example for you to test with storage.scrapinghub.com:8002. However, in this case i could not reproduce the bug in that server. Only happens with my vm version. Anyway this is the test page for rendering:
http://storage.scrapinghub.com:8002/collections/645/cs/Pages/075b2960be370059076be43cfd65a11a6ea62cc3/body?apikey=&format=html"
which, rendered with splash2 using curl:
curl --get --data-urlencode "url=http://storage.scrapinghub.com:8002/collections/645/cs/Pages/075b2960be370059076be43cfd65a11a6ea62cc3/body?apikey=&format=html" --data-urlencode "baseurl=http://www.icone.com/" http://33.33.33.10:8050/render.png > render.png
and adding baseurl:
curl --get --data-urlencode "url=http://33.33.33.11:8002/collections/9/cs/Pages/075b2960be370059076be43cfd65a11a6ea62cc3/body?apikey=ffffffffffffffffffffffffffffffff&format=html" --data-urlencode "baseurl=http://www.icone.com/" http://33.33.33.10:8050/render.png
This returns a "not found" when hitting Splash:
http://www.yelp.com/biz/abra%C3%A7o-new-york-2
but gets the response with curl
$ curl http://www.yelp.com/biz/abra%C3%A7o-new-york-2
I noticed this is a lot slower and causes crashing. I'm mainly using it for the javascript rendering
I don't know if it can be done in PyQT, but it could enable many cool things (e.g. taking screenshots of individual HTML elements).
I am trying to render a page with Splash using the Crawlera service as a proxy, however I am getting into timeouts. I think we should support a greater default timeout.
I started implementing proxy support here: https://github.com/kmike/splash/tree/proxy-support. The idea was to have
but it strikes me that it is too opinionated and project-specific (e.g. no multiple proxies to choose from, the blacklist/whitelist thing).
Also, this requires some project-specific configuration, namely there should be a way to specify proxy params, blacklist and whitelist. It is hard to do it via command line; in order to be consistent and easy to use this would probably require adding "settings.py" support to splash. Simple INI files are not a good fit because of blacklist/whitelit being lists.
What do you think about not adding settings.py support to splash and refactoring code to make it possible to use custom QNetworkProxyFactory subclasses instead? This changes philosophy a bit - instead of just installing and using splash user is supposed to customize it to project needs by writing some code.
Request headers are passed to remote pages only for POST requests when splash works in proxy mode.
"nosetests" fail for some reason on dev.scrapinghub.com, even through the service works fine there.
Need to investigate why, I suspect processes spawned from tests not being managed properly.
The idea is by @nramirezuy. It will allow to use HTTP methods other than GET, access response headers (and e.g. preserve cookies between requests), etc.
This happens with amazon.com, sometimes. I haven't yet figured out exactly what is causing loadFinished to get triggered twice but it's related with the sign in box that appears when you go into amazon.com.
When using splash as a proxy, and making a POST request with gzip encoding header, it seems the content does not get decoded before taking the screenshot. This does not happen when the method is GET or the encoding is deflate. The render method html is affected too.
For example:
POST request with deflate encoding
curl -x localhost:8051 -X POST -H 'Accept-Encoding: deflate' -H 'X-Splash-render: png' -H 'X-Splash-wait: 1' http://www.facebook.com
POST request with gzip encoding
curl -x localhost:8051 -X POST -H 'Accept-Encoding: gzip' -H 'X-Splash-render: png' -H 'X-Splash-wait: 1' http://www.facebook.com
When using Splash as a proxy the response HTTP headers are not returned. They can be returned either as headers in the Splash response or as a fields in the JSON response.
when I render bestbuy.ca and some other sites, the html rendered is not what I expected. elements sprawl and overlap, it doesn't resemble anything like the screenshot.
give it a try
render.html?url=http://www.bestbuy.ca
is it possible to fix this somehow? what is causing such malformed html?
I have a javascript that inserts <base> tag inside the <head>
I want to make this run on every page that I request.
For example, if there's an image and it's relative url, it won't get rendered. running this javascript helps me.
Right now it requires POST request. however I need it to run on each get request when I request render.html?js=/etc/js-profile/new&url=http://a.com
I can't reproduce it myself, but this failure happened:
======================================================================
FAIL: test_blacklist (splash.tests.test_proxy.HtmlProxyRenderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/lib/buildbot/slave/builders/splash/build/splash/tests/test_proxy.py", line 72, in test_blacklist
self.assertProxied(frame['html'])
File "/var/lib/buildbot/slave/builders/splash/build/splash/tests/test_proxy.py", line 94, in assertProxied
assert 'PROXY_USED' in html
AssertionError:
-------------------- >> begin captured logging << --------------------
requests.packages.urllib3.connectionpool: INFO: Starting new HTTP connection (1): localhost
requests.packages.urllib3.connectionpool: DEBUG: "GET /render.json?url=http%3A%2F%2Flocalhost%3A8998%2Fiframes&html=1&iframes=1&proxy=test HTTP/1.1" 200 None
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
I'm using latest revision and rendering this pages shows a difference in the footer layout being overly large:
Is there a workaround to avoid this behavior?
The following Splash calls return "Error rendering page":
The target page have in common that they redirect to another URL, not sure if that is the cause of the issue.
Big XML sitemap files can take a lot of CPU time to render; timeout doesn't work for them. It actually happened that such 4MB sitemap (https://gust.com/sitemap.xml) took 100% CPU for 30 minutes, and splash became unresponsive. We must fix this somehow.
There are several parameters that are not documented in the Readme, those can be seen by running the help option from the command line.
Usage: server.py [options]
Options:
-h, --help show this help message and exit
-f LOGFILE, --logfile=LOGFILE
log file
-m MAXRSS, --maxrss=MAXRSS
exit if max RSS reaches this value (in KB) (default:
0)
-p PORT, --port=PORT port to listen to (default: 8050)
-s SLOTS, --slots=SLOTS
number of render slots (default: 50)
--cache enable local cache (active by default)
--no-cache disable local cache
-c CACHE_PATH, --cache-path=CACHE_PATH
local cache folder
--cache-size=CACHE_SIZE
maximum cache size in Kb (default: 51200)
--proxy-profiles-path=PROXY_PROFILES_PATH
path to a folder with proxy profiles
Hi,
I'm facing some problems executing a JS on a specific website:
➜ ~ curl -X POST -H 'content-type: application/javascript' -d 'showModalDimmer(); dojo.publish("showResultsForPageNumber",[{pageNumber:"2",pageSize:"12", linkId:"WC_SearchBasedNavigationResults_pagination_link_right_categoryResults"}]);' 'http://localhost:8050/render.html?url=http://www.hhgregg.com/appliances-home/washers&timeout=60&wait=0.5'
<html><head>
<title>Invalid URL</title>
</head><body>
<h1>Invalid URL</h1>
The requested URL "/appliances-home/washers", is invalid.<p>
Reference #9.9b043bbb.1394120505.1585c2c6
</p></body></html>%
This JS code is executed when I click the "Next" button.
However, it works with a simpler JS code:
➜ ~ curl -X POST -H 'content-type: application/javascript' -d 'document.write("hello");' 'http://localhost:8050/render.html?url=http://www.hhgregg.com/appliances-home/washers&timeout=60&wait=0.5'
<html><head></head><body>hello</body></html>%
Also, splash doesn't seem to properly render the page when I visualize it using the browser.
Any thoughts?
It seems unlikely we'll use code from qtrender.py in future.
I have a splash call to render.json that gets the full screenshot of the page (viewport=full) and at the same time uses a Javascript function to get the position of certain elements in the page.
However when I compare the coordinates obtained with the generated screenshot they don't match.
I think the problem is that the javascript code is executed before the viewport is applied. Therefore the javascript is getting the elements position with the default viewport and the screenshot is generated with the full viewport.
Twisted==11.1.0, qt4reactor==1.0
test_whitelist (splash.tests.test_proxy.BlackWhiteProxyFactoryTest) ... ok
test_blacklist (splash.tests.test_proxy.HtmlProxyRenderTest) ... Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 69, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 586, in _doReadOrWrite
why = selectable.doRead()
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 199, in doRead
rval = self.protocol.dataReceived(data)
File "/usr/lib/python2.7/dist-packages/twisted/protocols/basic.py", line 572, in dataReceived
return self.rawDataReceived(data)
File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 503, in rawDataReceived
self.handleResponseEnd()
File "/usr/lib/python2.7/dist-packages/twisted/web/proxy.py", line 88, in handleResponseEnd
self.father.finish()
File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 866, in finish
"Request.finish called on a request after its connection was lost; "
exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.
Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 69, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 586, in _doReadOrWrite
why = selectable.doRead()
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 199, in doRead
rval = self.protocol.dataReceived(data)
File "/usr/lib/python2.7/dist-packages/twisted/protocols/basic.py", line 572, in dataReceived
return self.rawDataReceived(data)
File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 503, in rawDataReceived
self.handleResponseEnd()
File "/usr/lib/python2.7/dist-packages/twisted/web/proxy.py", line 88, in handleResponseEnd
self.father.finish()
File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 866, in finish
"Request.finish called on a request after its connection was lost; "
exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.
Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 69, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 586, in _doReadOrWrite
why = selectable.doRead()
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 199, in doRead
rval = self.protocol.dataReceived(data)
File "/usr/lib/python2.7/dist-packages/twisted/protocols/basic.py", line 572, in dataReceived
return self.rawDataReceived(data)
File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 503, in rawDataReceived
self.handleResponseEnd()
File "/usr/lib/python2.7/dist-packages/twisted/web/proxy.py", line 88, in handleResponseEnd
self.father.finish()
File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 866, in finish
"Request.finish called on a request after its connection was lost; "
exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.
ok
test_insecure (splash.tests.test_proxy.HtmlProxyRenderTest) ... ok
test_no_proxy_settings (splash.tests.test_proxy.HtmlProxyRenderTest) ... ok
test_nonexisting (splash.tests.test_proxy.HtmlProxyRenderTest) ... ok
test_proxy_works (splash.tests.test_proxy.HtmlProxyRenderTest) ... Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 69, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 586, in _doReadOrWrite
why = selectable.doRead()
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 199, in doRead
rval = self.protocol.dataReceived(data)
File "/usr/lib/python2.7/dist-packages/twisted/protocols/basic.py", line 572, in dataReceived
return self.rawDataReceived(data)
File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 503, in rawDataReceived
self.handleResponseEnd()
File "/usr/lib/python2.7/dist-packages/twisted/web/proxy.py", line 88, in handleResponseEnd
self.father.finish()
File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 866, in finish
"Request.finish called on a request after its connection was lost; "
exceptions.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.
ok
test_basic (splash.tests.test_render.IframesRenderTest) ... ok
Try the following URL:
The flash objects are not displayed, if you go to the URL directly you can see them: http://copyboost.com/ad-examples/good-flash-ad-examples/
We may want to add a setting to turn on/off flash support on Splash requests.
Although splash and scrapy are independent project, they are often used together.
But the setup is apparently not that straightforward (see for example https://stackoverflow.com/questions/21564234/how-to-integrate-javascript-rendering-module-to-scrapy)
It would be nice to have a step-by-step guide on how to make scrapy and splash work together nicely.
the setup is centos 5 python2.7
$ rpm -qa | grep qt
qt4-devel-4.7.1-0
qt4-4.7.1-0
to execute the server with xvfb-run
$ /usr/local/bin/xvfb-run -a -s "-screen 0 640x480x8" python -m splash.server
2014-03-18 22:00:05+0800 [-] Log opened.
2014-03-18 22:00:05+0800 [-] Open files limit: 1024000
2014-03-18 22:00:05+0800 [-] Can't bump open files limit
2014-03-18 22:00:05+0800 [-] Traceback (most recent call last):
2014-03-18 22:00:05+0800 [-] File "/usr/local/lib/python2.7/runpy.py", line 162, in _run_module_as_main
2014-03-18 22:00:05+0800 [-] "__main__", fname, loader, pkg_name)
2014-03-18 22:00:05+0800 [-] File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code
2014-03-18 22:00:05+0800 [-] exec code in run_globals
2014-03-18 22:00:05+0800 [-] File "/usr/local/lib/python2.7/site-packages/splash-1.0-py2.7.egg/splash/server.py", line 233, in <module>
2014-03-18 22:00:05+0800 [-] main()
2014-03-18 22:00:05+0800 [-] File "/usr/local/lib/python2.7/site-packages/splash-1.0-py2.7.egg/splash/server.py", line 224, in main
2014-03-18 22:00:05+0800 [-] proxy_portnum=opts.proxy_portnum)
2014-03-18 22:00:05+0800 [-] File "/usr/local/lib/python2.7/site-packages/splash-1.0-py2.7.egg/splash/server.py", line 144, in default_splash_server
2014-03-18 22:00:05+0800 [-] manager = network_manager.FilteringQNetworkAccessManager()
2014-03-18 22:00:05+0800 [-] File "/usr/local/lib/python2.7/site-packages/splash-1.0-py2.7.egg/splash/network_manager.py", line 105, in __init__
2014-03-18 22:00:05+0800 [-] super(FilteringQNetworkAccessManager, self).__init__()
2014-03-18 22:00:05+0800 [-] File "/usr/local/lib/python2.7/site-packages/splash-1.0-py2.7.egg/splash/network_manager.py", line 48, in __init__
2014-03-18 22:00:05+0800 [-] self.sslErrors.connect(self._sslErrors)
2014-03-18 22:00:05+0800 [-] TypeError: pyqtSignal must be bound to a QObject, not 'FilteringQNetworkAccessManager'
however to examine from with ipython
In [8]: from PyQt4.QtNetwork import QNetworkAccessManager, QNetworkProxyQuery, QNetworkReply
In [9]: import inspect
In [10]: inspect.getmro(QNetworkAccessManager)
Out[10]:
(PyQt4.QtNetwork.QNetworkAccessManager,
PyQt4.QtCore.QObject,
sip.wrapper,
sip.simplewrapper,
object)
Have no clue what's wrong, it's the latest splash master
e.g.
when run curl -x http://localhost:8051 -H "X-Splash-render: json" -H "X-Splash-html: 0" http://steinmetz-maxwald.at/materialien/, ideally splash should send the 404 status code back. but currently the splash always return 200 for the url above,
make splash2 log errors to sentry
Cache responses on S3 for faster retrieval.
Add setting in *.ini for user agents.
[useragents]
chrome:'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.17 Safari/537.36'
firefox:'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0'
Example on changes user agent on QNetworkAccessManager:
QNetworkAccessManager* mgr = new QNetworkAccessManager();
....
QNetworkRequest req;
req.setUrl(QUrl("enter url") );
req.setRawHeader( "User-Agent" , "Mozilla Firefox" );
mgr->get(req);
The addition will have to be two part addition by adding way to parse the config .INI file and the setting the raw header for QT.
Our staff would like to know if there is a way to turn logging off or on for this software.
Adding this information to the document would be very helpful.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.