kiorky / spynner

Programmatic web browsing module with AJAX support for Python

Intro

Contents

• Intro
• Credits
  • Companies
  • Authors
  • Contributors
• Dependencies
• Feedback
• Install
• Running Spynner without X11

Spynner is a stateful programmatic web browser module for Python. It is based upon PyQT and WebKit. It supports Javascript, AJAX, and every other technology that !WebKit is able to handle (Flash, SVG, ...). Spynner takes advantage of JQuery. a powerful Javascript library that makes the interaction with pages and event simulation really easy.

Using Spynner you would able to simulate a web browser with no GUI (though a browsing window can be opened for debugging purposes), so it may be used to implement crawlers or acceptance testing tools.

See usage on: https://github.com/kiorky/spynner/tree/master/src/spynner/tests/spynner.rst Or below if the section is preset

Credits

Companies

• Planet Makina Corpus
• Contact us

Authors

• Mathieu Le Marec - Pasquet <[email protected]>
• Arnau Sanchez <[email protected]>

Contributors

• Leo Lou <https://github.com/l4u>

Dependencies

• Python >=26
• PyQt > 443
• Libxml2 / Libxslt libraries and includes files for lxml
• autopy which in turns need xtst lib & headers on linux (aka Xtest)

Feedback

Open an Issue to report a bug or request a new feature. Other comments and suggestions can be directly emailed to the authors.

Install

• Throught regular easy_install / buildout:

  easy_install spynner
  

  (In Windows, you may have to install autopy through its installer at https://pypi.python.org/pypi/autopy/)

• The bleeding edge version is hosted on github:

  git clone https://github.com/kiorky/spynner.git
  cd spynner
  python setup.py install
  

Running Spynner without X11

• Spynner needs a X11 server to run. If you are running it in a server without X11. You must install the virtual Xvfb server. Debian users can use the small wrapper (xvfb-run). If you are not using Debian, you can download it here: http://www.mail-archive.com/[email protected]/msg69632/x-run

  xvfb-run python myscript_using_spynner.py
  

• You can also use tightvnc, which is the solution of the actual maintainer [kiorky].

spynner's Issues

Click on href

Hi everybody,

I need to click on the following link:

Empty

Is it possible somehow with spynner?

How to use multi instance of browser simultaneously crawler pages?

Login to cisco.com stopped working in spynner 2.7

With spynner 2.6 I can successfully login in to cisco.com using their single-sign-on page https://sso.cisco.com/

With spynner 2.7 the login no longer works. After submitting the login form at https://sso.cisco.com/ I keep getting redirected back to the form. I have tried Python 2.6 and Python 2.7, with the same result.

The code I am using to perform the login is as follows:

        self.cscc_url = (
            'https://apps.cisco.com/CustAdv/ServiceSales/smcam/'
            'requestStatusDispatch.do?methodName=onDashboardAction')

        self.browser = spynner.Browser(ignore_ssl_errors=False,
                                       user_agent=self.user_agent,
                                       debug_level=spynner.DEBUG,
                                       debug_stream=sys.stderr)

        try:
            self.browser.load(self.cscc_url, load_timeout=timeout)
        except spynner.SpynnerTimeout:
            # FIXME
            raise

        #                                                                       
        # if we got redirected to sso.cisco.com, log in first                   
        #                                                                       

        attempts = 0

        while self.browser.url.find('sso.cisco.com') != -1:
            if attempts >= self.auth_attempts:
                raise CSCCAuthenticationError

            attempts += 1

            if self.cco_username:
                self.browser.fill('input[name=userid]', self.cco_username)

            if self.cco_password:
                self.browser.fill('input[name=password]', self.cco_password)

            self.browser.submit('input[id=login-button][name=login-button]')

        assert self.browser.url == self.cscc_url

With spynner 2.7 I get my CSCCAuthenticationError exception after three attempts. With spynner 2.6 the login is successful on the first attempt.

Please check out debug traces from successful and unsuccessful logins.

.text() also prints html comments content

This is what currently happens:

        >>> doc = PyQuery('<div>hello <!-- foo --> world</div>')
        >>> print(doc.text())
        hello foo world

But it should output "hello world".

A fix could be changing this in pyquery.py line 884:

            if tag.text:
                text.append(tag.text)

For this:

            if tag.text and tag.tag != lxml.etree.Comment:
                text.append(tag.text)

And import lxml.etree.Comment at the head

Javascript data files aren't installed with package

setup.py doesn't include the package data js files.

diff -u a/setup.py b/setup.py
--- a/setup.py 2012-08-20 16:26:36.605462862 -0700
+++ b/setup.py 2012-08-20 16:30:40.309080935 -0700
@@ -71,6 +71,7 @@
+ '\n'
),
include_package_data=True,

• package_data={'spynner':['javascript/*.js']},
  extras_require = {
  'test': ['ipython', 'plone.testing']
  },

How can I get the response's header of spynner.browser ?

cookie broken?

the error like:
File "/usr/local/lib/python2.7/dist-packages/spynner-2.15-py2.7.egg/spynner/browser.py", line 1651, in createRequest
self.cookies,
AttributeError: 'Browser' object has no attribute 'cookies'

The program run for some time , then error happens . how can I fix it ?

AttributeError: 'XPathExpr' object has no attribute 'add_post_condition'

Running wordreference example on ubuntu 12.04:

Traceback (most recent call last):
  File "gh.py", line 20, in <module>
    imagedata = browser.download(browser.soup("img:first").attr('src'))
  File "/usr/local/lib/python2.7/dist-packages/pyquery/pyquery.py", line 237, in __call__
    result = self.__class__(*args, parent=self, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyquery/pyquery.py", line 213, in __init__
    xpath = self._css_to_xpath(selector)
  File "/usr/local/lib/python2.7/dist-packages/pyquery/pyquery.py", line 223, in _css_to_xpath
    return self._translator.css_to_xpath(selector, prefix)
  File "/usr/local/lib/python2.7/dist-packages/cssselect/xpath.py", line 188, in css_to_xpath
    for selector in selectors)
  File "/usr/local/lib/python2.7/dist-packages/cssselect/xpath.py", line 188, in <genexpr>
    for selector in selectors)
  File "/usr/local/lib/python2.7/dist-packages/cssselect/xpath.py", line 208, in selector_to_xpath
    xpath = self.xpath(tree)
  File "/usr/local/lib/python2.7/dist-packages/cssselect/xpath.py", line 230, in xpath
    return method(parsed_selector)
  File "/usr/local/lib/python2.7/dist-packages/cssselect/xpath.py", line 272, in xpath_pseudo
    return method(self.xpath(pseudo.selector))
  File "/usr/local/lib/python2.7/dist-packages/pyquery/cssselectpatch.py", line 19, in xpath_first_pseudo
    xpath.add_post_condition('position() = 1')
AttributeError: 'XPathExpr' object has no attribute 'add_post_condition'

Errors while using spynner

Hi,
I am trying to use spynner for one of my projects.
I have tried using both the latest stable version (1.10) and the code from trunk, (version 1.11), but its giving me the same error.

code:
browser = spynner.Browser()
browser.load(self.url)
self.html = browser.html
print self.html
browser.close()

I am getting the following error:

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/spynner-1.11dev-py2.7.egg/spynner/browser.py", line 232, in _manager_create_request
reply = QNetworkAccessManager.createRequest(self.manager,
AttributeError: 'Browser' object has no attribute 'manager'

This error arises when the browser.close() function is called (as part of the destructor or when it is called explicitly):
if self.manager:
del self.manager
if self.webpage:
del self.webpage

(if self.webpage gives the error)

I tried a couple of things such as commenting these lines (I saw a version of code - (svn/trunk version, surprisingly different from github - http://code.google.com/p/spynner/source/browse/trunk/spynner/browser.py) where these lines were not present in the close function) but other errors start surfacing.
(Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/spynner-1.11dev-py2.7.egg/spynner/browser.py", line 222, in _manager_create_request
operation_name = self._operation_names[operation].upper()
AttributeError: 'Browser' object has no attribute '_operation_names')

I noticed an issue with the trunk packaging too. The jquery files dont get copied to the package (dist-packages/spynner-1.11dev-py2.7.egg/spynner/javascript/) when I install from the source code. I manually copied them over.

Please let me know if I am doing something wrong or there is some kind of bug there.
If you need more information on the error, let me know.

can't connect to x server

when i put the source code anothergoogle.py on the cloud ubuntu12.04 32bit server, error : cannt connect to x server, the same to http://forum.linode.com/viewtopic.php?t=8140

how to accelerate the execution

Hi,pal! I need loading operations almost one million times. my code is :

import os
import re
import random,time
from lxml.html import document_fromstring
from spynner import browser
from PyQt4 import QtCore

pat = re.compile(ur'<span[ ]id="scd_num">([\d,]+)</span>')
## <span id="scd_num">2,106,696</span>
re_flags = re.M|re.U|re.S|re.X
os.environ['DISPLAY'] = os.environ.get('GMB_DISPLAY', ':0')

def get_tree(h):
    """h can be either a zc.testbrowser.Browser or a string."""
    if isinstance(h, file):
        h = h.read()
    if isinstance(h, browser.Browser):
        h = h.html
    if not isinstance(h, basestring):
        h = h.contents
    return document_fromstring(h) 


def getQtBrowser():
    #debug=4
    debug = 0
    br = browser.Browser(embed_jquery_simulate=False, want_compat=False, debug_level=debug,)
    return br

def do(br,term):
    #logger = logging.getLogger('scrapper.test')
    br.load('http://www.example.com/')
    def can_continuea(abrowser):
        t = get_tree(abrowser)
        #print len(t.xpath("//input[@name='query']"))
        return len(t.xpath("//input[@name='query']")) > 0
    br.wait_for_content(can_continuea,6, u'Timeout while loading account data')  
    br.wk_fill('input[name="query"]',term.decode("utf8"))
    input_sel = "input[id='stb']"
    br.wk_click(input_sel)
    def can_continueb(abrowser):
        t = get_tree(abrowser)
        return len( t.xpath('//*[@id="scd_num"]')) > 0
    #print "url:",br.url
    br.wait_for_content(can_continueb,6, u'Timeout while loading account data')
    result = br.html
    m = pat.search(result)
    if m:
        return m.group(1).replace(',','')
    else:
        return '0'




if __name__ == '__main__':
    br = getQtBrowser()
    br.create_webview()
    br.webview.setWindowState(QtCore.Qt.WindowMaximized)
    br.webview.show()
    a = set()
    flag=0
    fo = open('./lexicon.txt')
    for line in fo:
        line=line.strip()
        if line:
            a.add(line)
    b  = a.copy()
    for i in a:
        if i=='令人恼火':
            continue
        b.remove(i)
        for j in b:
            if flag:
                try:
                    fw = open('./two.txt','a')
                    resultCNT = do(br,'"'+i+'"'+' "'+j+'"')
                    fw.write(i+'   '+j+'   '+resultCNT.encode('utf8')+'\n')
                except Exception,e:
                    print e
                    print i,j
                finally:
                    fw.close()  
                time.sleep(random.uniform(0.05,0.09))

            if i+'   '+j=='质疑   低价位':
                flag=1

TypeError: C++ type 'QList<QSslError>' is not supported as a slot argument type

Hi
I used spynner with WindowsXP 32 bit before, without a hitch.

Now i have Windows 7 64 bit,
Python 2.7.3 (also 64bit)
and of course 64bit pyqt.

Here's the thing

.>>> import spynner
.>>> import pyquery
.>>> browser = spynner.Browser(debug_level=spynner.INFO)
Traceback (most recent call last):
File "", line 1, in
File "C:\programs\python\lib\site-packages\spynner-2.5.dev0-py2.7.egg\spynner\browser.py", line 160, in init
self._on_manager_ssl_errors)
TypeError: C++ type 'QList' is not supported as a slot argument type
.>>>

?

10x

No CustopOperation in QNetworkAccessManager

Spynner cannot run with last PyQt4 version due to the follow error:

AttributeError: type object 'QNetworkAccessManager' has no attribute 'CustomOperation'

That's because QNetworkAccessManager doesn't have this attribute.

>>> from PyQt4.QtNetwork import QNetworkAccessManager
>>> from PyQt4.QtCore import PYQT_VERSION_STR as version
>>> print version
4.7.3
>>> dir(QNetworkAccessManager)
['DeleteOperation', 'GetOperation', 'HeadOperation', 'Operation', 'PostOperation', 'PutOperation', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'authenticationRequired', 'blockSignals', 'cache', 'childEvent', 'children', 'connect', 'connectNotify', 'cookieJar', 'createRequest', 'customEvent', 'deleteLater', 'deleteResource', 'destroyed', 'disconnect', 'disconnectNotify', 'dumpObjectInfo', 'dumpObjectTree', 'dynamicPropertyNames', 'emit', 'event', 'eventFilter', 'findChild', 'findChildren', 'finished', 'get', 'head', 'inherits', 'installEventFilter', 'isWidgetType', 'killTimer', 'metaObject', 'moveToThread', 'objectName', 'parent', 'post', 'property', 'proxy', 'proxyAuthenticationRequired', 'proxyFactory', 'put', 'pyqtConfigure', 'receivers', 'removeEventFilter', 'sender', 'setCache', 'setCookieJar', 'setObjectName', 'setParent', 'setProperty', 'setProxy', 'setProxyFactory', 'signalsBlocked', 'sslErrors', 'startTimer', 'staticMetaObject', 'thread', 'timerEvent', 'tr', 'trUtf8']

Delete "Custom" from src/spynner/browser.py:221 fix the problem.

Regards

Segfaults

Running Spynner on Python 2.7 I get fairly regular segfaults.

The easiest way I can reproduce is this call to wk_click:

br.wk_click('img[id="continue"]', wait_load=True, timeout=60)

If it times-out I wasn't catching the SpynnerTimeout exception. So I'd get this in stderr:

Traceback (most recent call last):
  File "test.py", line 13, in <module>
    o.convert(recipient)
  File "/usr/local/cuna/apps/submitter/submitter.py", line 186, in convert
    self.validate()
  File "/usr/local/cuna/apps/submitter/submitter.py", line 350, in validate
    self.browser().wk_click('img[id="continue"]', wait_requests=1, wait_load=True, timeout=60)
  File "/home/sharter/.virtualenvs/cana/local/lib/python2.7/site-packages/spynner/browser.py", line 747, in wk_click
    return self.wk_click_element(element, wait_load=wait_load, wait_requests=wait_requests, timeout=timeout)
  File "/home/sharter/.virtualenvs/cana/local/lib/python2.7/site-packages/spynner/browser.py", line 716, in wk_click_element
    return self._wait_load(timeout)
  File "/home/sharter/.virtualenvs/cana/local/lib/python2.7/site-packages/spynner/browser.py", line 366, in _wait_load
    raise SpynnerTimeout("Timeout reached: %d seconds" % timeout)
spynner.browser.SpynnerTimeout: Timeout reached: 60 seconds
Segmentation fault (core dumped)

So fine, I catch that exception. But during runtime, I see supervisord having to start the process several times a day when it segfaults and exits. Those cases aren't preceeded by uncaught exceptions so I'm not getting a stack trace, I just get the Segmentation Fault error.

Do you have any hints? Anything that is known to cause this issue in Spynner. I know a segfault is common with PyQt if Python is trying to access a Qt object that's been destroyed or likewise if Qt is trying to access a Python object that's been destroyed. But trying to find that bug is a needle in a giant stack of needles.

Thanks!
Shane

How to select the first input[type=submit]

Hello,
My question may be slightly off the base.I am using the spynner on my project,in which I want to click each input[type=submit] one by one. the function wk_click() requires webkit xpath selector. But I can't find the xpath selector expressing k-th input[type=submit] .. I has tried //input[type=submit][1] or (input[type=submit])[1] or input[type=submit and position()=1].All of them don't work.
Could you please help me.Thank you in advance.Thank you.

SpynnerTimeout

I get a timeout error on every url. My internet connection is okay.

false set usergent

http://pastebin.com/RkYzeMBk

import spynner
import pyquery
browser = spynner.Browser()

useragent= "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0"

browser = spynner.Browser(
#debug_level=spynner.WARNING,
#~ debug_level=spynner.ERROR,
debug_level=spynner.DEBUG,
#~ debug_level=spynner.INFO,

    user_agent = useragent

)

browser.create_webview()
browser.show()
ret = browser.load_jquery(True)

browser.load("http://whoer.net",load_timeout=1000)

browser.browse()

---

result :
Headers Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0

JavaScript Mozilla/5.0 (Windows NT 5.1) AppleWebKit/534.34 (KHTML, like Gecko) Qt/4.8.4 Safari/534.34

====>Mismatch

i try different script:
http://pastebin.com/H6aLTNid

!/usr/bin/python

-- coding: utf-8 --

from urlparse import urlparse

from PyQt4 import QtCore, QtGui, QtWebKit

UA_STRING = """Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0"""

class YWebPage(QtWebKit.QWebPage):
def init(self):
super(QtWebKit.QWebPage, self).init()

def userAgentForUrl(self, url):
    return UA_STRING

class Yaris(QtGui.QMainWindow):
def init(self):
super(Yaris, self).init()

    widget = QtGui.QWidget()
    self.setCentralWidget(widget)

    self.buffers = []
    self.codecs = [QtCore.QLatin1String(QtCore.QTextCodec.codecForMib(mib).name())
                   for mib in QtCore.QTextCodec.availableMibs()]
    self.codecs.sort()

    topFiller = QtGui.QWidget()
    topFiller.setSizePolicy(QtGui.QSizePolicy.Expanding,
            QtGui.QSizePolicy.Expanding)

    self.yPage = YWebPage()

    self.webview = QtWebKit.QWebView(self)
    self.webview.urlChanged.connect(self.updStatusUrl)
    self.webview.loadProgress.connect(self.updProgress)
    self.webview.setPage(self.yPage)

    self.minibuffer = QtGui.QLineEdit()
    self.minibuffer.hide()
    self.minibuffer.returnPressed.connect(self.mbExec)

    bottomFiller = QtGui.QWidget()
    bottomFiller.setSizePolicy(QtGui.QSizePolicy.Expanding,
            QtGui.QSizePolicy.Expanding)

    vbox = QtGui.QVBoxLayout()
    vbox.setMargin(5)
    vbox.addWidget(topFiller)
    vbox.addWidget(self.webview)
    vbox.addWidget(bottomFiller)
    vbox.addWidget(self.minibuffer)
    widget.setLayout(vbox)

    self.setWindowTitle(self.tr("Yaris"))
    self.setMinimumSize(160,160)
    self.resize(1024,600)

def loadUrl(self, url):
    self.webview.load(QtCore.QUrl(url))
    self.webview.show()
    self.updStatusUrl()

def miniBuffer(self, prompt):
    pass

def mbExec(self):
    text = str(self.minibuffer.text())
    if urlparse(text).scheme == '':
        text = 'http://' + text
    self.loadUrl(text)
    self.minibuffer.hide()
    self.minibuffer.clear()

def updStatusUrl(self, url=None):
    url = self.webview.url()
    message = url.toString()
    self.statusBar().showMessage(message)
    self.updTitle()


def updTitle(self):
    self.setWindowTitle(self.webview.title())

def updProgress(self, p):
    pass

def keyPressEvent(self, event):
    key = event.key()
    if event.modifiers() == QtCore.Qt.ControlModifier:
        if key == QtCore.Qt.Key_G:
            self.webview.stop()
            self.minibuffer.hide()
    else:
        if key == QtCore.Qt.Key_G:
            self.minibuffer.show()
            self.minibuffer.setFocus()
        elif key == QtCore.Qt.Key_B:
            self.webview.back()
        elif key == QtCore.Qt.Key_F:
            self.webview.forward()
        elif key == QtCore.Qt.Key_R:
            self.webview.reload()
        elif key == QtCore.Qt.Key_N:
            print "N pressed"
        else:
            super(Yaris, self).keyPressEvent(event)

if name == "main":

import sys

app = QtGui.QApplication(sys.argv)
#QtCore.QTextCodec.setCodecForTr(QtCore.QTextCodec.codecForName("UTF-8"))
yaris = Yaris()
yaris.show()

#~ yaris.loadUrl("file:///usr/doc/qt-4.5.1/html/index.html")
yaris.loadUrl("http://whoer.net")
#yaris.loadUrl("file:///home/zert/Work/Prog/Python/yaris/test.html")

sys.exit(app.exec_())

---

result ::

Headers Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0

JavaScript Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0

===> Match

i think spynner false when change useragent .

ImportError: No module named QtWebKit

How to solve this problem when I run "run_tests.sh"?

OutPut:

System: Linux 3.2.6 x86_64 x86_64 GNU/Linux
Python: 2.6.6
PyQT: 4.6.2

Traceback (most recent call last):
File "test/test_browser.py", line 25, in
import spynner
File "/usr/lib/python2.6/site-packages/spynner-2.8.dev0-py2.6.egg/spynner/init.py", line 2, in
from browser import *
File "/usr/lib/python2.6/site-packages/spynner-2.8.dev0-py2.6.egg/spynner/browser.py", line 57, in
from PyQt4.QtWebKit import QWebPage, QWebView
ImportError: No module named QtWebKit

The arg of wait_callback

When I try to add a arg to load method,it return empty html.

#!/usr/bin/env python
import spynner
import pyquery

def test(browser):
    print '.........'
    return '++++++++++++'

browser = spynner.Browser(debug_level=spynner.DEBUG)
browser.create_webview()
browser.show()
browser.set_html_parser(pyquery.PyQuery)
browser.load("http://link.gongju.com", wait_callback = test)
print browser.html
browser.fill("input[name=address]", "www.3533.com")
browser.click("input[name=submit]")
browser.wait_load()
browser.wait(2)
print "url:", browser.url

# Soup is a PyQuery object
browser.soup.make_links_absolute(base_url=browser.url)
print "html:", browser.soup("#Otbl").html()

# Demonstrate how to download a resource using PyQuery soup
imagedata = browser.download(browser.soup("img:first").attr('src'))
print "image length:", len(imagedata)
browser.close()

The page returns <html><head></head><body></body></html>,the result made me so depressed

Error when load google.es

I am try navigate in google.es:
s.load("http://google.es")
(s is var of browser) and say:
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/spynner/browser.py", line 530, in load
return self._wait_load(timeout = load_timeout)
File "/usr/local/lib/python2.7/dist-packages/spynner/browser.py", line 422, in _wait_load
raise SpynnerTimeout("Timeout reached: %d seconds" % timeout)
spynner.browser.SpynnerTimeout: Timeout reached: 10 seconds

I try:
s.load("http://google.es", headers=[('User-agent', 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)')])
And say:
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/spynner/browser.py", line 530, in load
return self._wait_load(timeout = load_timeout)
File "/usr/local/lib/python2.7/dist-packages/spynner/browser.py", line 422, in _wait_load
raise SpynnerTimeout("Timeout reached: %d seconds" % timeout)
spynner.browser.SpynnerTimeout: Timeout reached: 10 seconds

And with paginasamarillas.es is same, but with paginasamarillas.es if I open windows (.show() method) load without error.

Fullscreen Spynner browser , minimizes after browse()

Hello,

so , i have used this code to run google with a maximized webview

import spynner

if __name__ == '__main__':
     b = spynner.Browser()
     b.debug_level = spynner.DEBUG
     url ="http://www.google.com/"
     b.load(url)
     b.create_webview()
     b.webview.showFullScreen()
     b.browse()

after the call to browse is made , the spyner window becomes minmized , what should i do to keep it in fullscreen mode ?

proxy not working with https: pages.

Hi!
I am trying to use proxy for my SSL requests.
Default way
browser.set_proxy()
seems to work for http: requests only, leaving https to go directly.

As workaround I set proxy globally:
proxy = QtNetwork.QNetworkProxy(QtNetwork.QNetworkProxy.HttpProxy, PROXY_HOST,
without calling browser.set_proxy() and than it seems to work.

Example code:

! /usr/bin/python

PROXY_HOST = "put.proxy.here"
PROXY_PORT = 3128
PROXY = "http://%s:%d" % (PROXY_HOST, PROXY_PORT, )

if 0:
from PyQt4 import QtNetwork
proxy = QtNetwork.QNetworkProxy(QtNetwork.QNetworkProxy.HttpProxy, PROXY_HOST, PROXY_PORT)
QtNetwork.QNetworkProxy.setApplicationProxy(proxy)

import spynner
import pyquery

browser = spynner.Browser(debug_level=spynner.INFO)
if 1:
browser.set_proxy("PROXY")

browser.create_webview()
browser.show()

browser.load("http://www.whatismyip.com/", load_timeout=30)
browser.browse()

error: Setup script exited with error: command 'gcc' failed with exit status 1

trying to install spynner on centos 6.4, with python 👍
Python 2.6.6 (r266:84292, Feb 22 2013, 00:00:18)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2

Running autopy-0.51/setup.py -q bdist_egg --dist-dir /tmp/easy_install-8PTfEA/autopy-0.51/egg-dist-tmp-rFQP5a
In file included from src/autopy-color-module.c:1:
src/autopy-color-module.h:5:20: error: Python.h: No such file or directory
In file included from src/autopy-color-module.c:1:
src/autopy-color-module.h:11: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘initcolor’
src/autopy-color-module.c:9: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘’ token
src/autopy-color-module.c:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘’ token
src/autopy-color-module.c:19: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘ColorMethods’
src/autopy-color-module.c:32: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘initcolor’
src/autopy-color-module.c:38: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘’ token
src/autopy-color-module.c:48: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘attribute’ before ‘’ token
error: Setup script exited with error: command 'gcc' failed with exit status 1

spynner with PyQT4

Hello!
I can not understand! Is it possible to insert an object into a form spynner created in PyQT4?
I need a box of my program emullirovat browser and perform some action in it.

release 2.10 broken

error: Setup script exited with error: can't copy 'examples/anothergoogle.py': doesn't exist or not a regular file

Multithread problem

Test code:

#!/usr/bin/env python
#coding=utf8

import threading
import spynner

def load():
    url = 'http://www.baidu.com/'
    br = spynner.Browser()
    status = br.load(url = url)
    br.close()
    print 'Finished'

def process_test():
    print 'in process'
    load()
    print 'out process'

def thread_test():
    print 'in thread'
    thrd = threading.Thread(target=load)
    thrd.start()
    thrd.join()
    print 'out thread'

Test cases:

thread_test()
thread_test()

in thread
Finished
out thread
in thread
Core Dump

thread_test()
process_test()

in thread
Finished
out thread
in thread
Core Dump

process_test()
thread_test()

in process
Finished
out process
in thread
Exception in thread Thread-1:
 ...
SpynnerTimeout: Timeout reached: 10 seconds

process_test()
process_test()

in process
Finished
out process
in process
Finished
out process

Cookies broken?

Python 2.7, recent Kubuntu:

import spynner
browser = spynner.browser.Browser()
print browser.get_cookies()
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/spynner-2.13-py2.7.egg/spynner/browser.py", line 1343, in get_cookies
return self.cookiesjar.mozillaCookies()
AttributeError: 'Browser' object has no attribute 'cookiesjar'

native_click not working

i used window xp , i tried native_click.py in examples folder but it not working , mouse not move to logo and not click to button ....

Error in example

When I run file spynner/examples/google.py I got error: AttributeError: 'Browser' object has no attribute 'choose'.
I assume there should be "select".

Regards,
jezozwierzak

Segfault when exiting program using spynner.Browser

Hi. I'm getting segfault on exit when using spynner browser. I'm using atexit.register(browser.close()). Version of spynner is current master from git.

[Thread 0x7fffde674700 (LWP 7872) exited]

Program received signal SIGSEGV, Segmentation fault.
0x00007fffeb70ad45 in QNetworkAccessManager::cookieJar() const () from /usr/lib64/libQtNetwork.so.4

(gdb) new-backtrace
#0 0x7fffeb70ad45 in QNetworkAccessManager::cookieJar() const ()Error occurred in Python command: 'module' object has no attribute 'solib_address'

This is what I get. Unfortunately I have no debugging symbols in Python so this is the only useful information I can get.

More examples link not working

The link given in https://github.com/makinacorpus/spynner/blob/master/src/spynner/tests/spynner.rst is not working :

See more examples in the repository: https://github.com/kiorky/spynner/tree/master/examples

Add timeout parameter to the download function

Here's a simple patch (against v2.4) for allowing a timeout value to be set when calling download:

--- /usr/local/lib/python2.7/dist-packages/spynner-2.4-py2.7.egg/spynner/browser.py 2012-10-10 08:14:58.296039078 -0700
+++ browser.py  2012-10-14 13:35:39.877374096 -0700
@@ -1254,7 +1254,7 @@

     #{ Download files

-    def download(self, url, outfd=None):
+    def download(self, url, outfd=None, timeout=None):
         """
         Download a given URL using current cookies.

@@ -1277,6 +1277,7 @@
         cj = _ExtendedNetworkCookieJar()
         cj.setAllCookies(self.cookiesjar.allCookies())
         manager.setCookieJar(cj)
+        itime = time.time()
         reply = manager.get(request)
         if reply.error():
             raise SpynnerError("Download error: %s" % reply.errorString())
@@ -1287,6 +1288,8 @@
             outfd = StringIO()
         self._start_download(reply, outfd)
         while self._download_reply_status is None:
+            if timeout and time.time() - itime > timeout:
+                raise SpynnerTimeout("Timeout reached: %d seconds" % timeout)
            self._events_loop()
         if outfd_set:
             return (reply.downloaded_nbytes if not reply.error() else None)

Any performance tests?

We're looking for the fastest webkit-based controllable browser. Since spynner and selenium+phantomjs and uzbl are quite similar to the end-user, the performance is the only thing that matters for us (and probably for someone else).

Did you perform any benchmarks or maybe you have ideas why one should be fast and another - not?

Thank you!

browser.click("button[name=btnK]",wait_load=True)

when I browser.load("https://www.google.com/webhp?hl=en&tab=ww") and
browser.click("button[name=btnK]",wait_load=True) ,but the click didn't work with message:
SpynnerJavascriptError: error on click: jq('button[name=btnK]').simulate('click');
Could you give me a hand , thanks in advance!

AttributeError: 'Browser' object has no attribute 'wait_page_load'

import spynner

browser = spynner.Browser()
browser.load("http://www.wordreference.com")
browser.runjs("console.log('I can run Javascript!')")
browser.runjs("_jQuery('div').css('border', 'solid red')") # and jQuery!
browser.select("#esen")
browser.fill("input[name=enit]", "hola")
browser.click("input[name=b]")

browser.wait_page_load()

print browser.url, len(browser.html)
browser.close()

error:

File "test.py", line 10, in
browser.wait_page_load()
AttributeError: 'Browser' object has no attribute 'wait_page_load'

download() doesn't use proxy

Hi,

The download feature works fine, except it doesn't use the proxy specified with set_proxy(). When the target website is not reachable without proxy, or when it relies on the visitor's IP address, it can cause problems.

I don't know the best way to fix this, but I temporarily solved it by patching the new() method of NManager, since it is called by download(). Adding this code just before return manager seems to work fine.

try:
    manager.setProxy(spynner.manager.proxy())
except:
    pass

browser.wait_load() browser don't exit when close gui

import spynner
from BeautifulSoup import BeautifulSoup

browser = spynner.Browser()
browser.set_html_parser(BeautifulSoup)

browser.debug_level = spynner.WARNING
browser.create_webview()
browser.show()
browser.load("https://accounts.google.com/ServiceLogin?service=mail&passive=true&rm=false&continue=https://mail.google.com/mail/&ss=1&scc=1&ltmpl=default&ltmplcache=2")

browser.runjs("console.log('I can run Javascript')")

browser.runjs("console.log('I can run jQuery: ' + jQuery('a:first').attr('href'))")

~ browser.fill("input[name="Email"]", "username")

~ browser.fill("input[name="Passwd"]", "password")

browser.click("input[name=""]")

~ browser.click("input[name=signIn]", True)

browser.wait_load()
browser.close()

i try this example but when i close by click "X" button but browser not exit (python.exe still running) .

Ajax Link Pops up with Form and won't fill out

I have a browser that clicks an Ajax Link but won't fill out the form. The values are correct from what I can see. Any ideas, how to solve this issue

        browser.click_ajax('a[rel="async-post"]')
        browser.wait_load()
        browser.wk_select('#country', "United States (+1)")
        browser.wk_fill('input[name="contact_point"]', "123-456-7890")

Thanks

Error installing spynner due to error in autopy

pip install spynner
fails due to an error in installing autopy==0.51

Here's the work around : http://stackoverflow.com/a/13150697/2259106

Using File data as Input to wk_fill

Hello I have an issue using spynner with file data

I loop through the file and put each line in a variable. The variable is of type string. But the data on the form if I run the code is never filled out.

But if I put a value as "Michael" say for first name it works Not sure what I am doing wrong. Or if this is a semi issue with spynner but thought I would see

with open("test.txt", 'r') as infile:
    lines_gen = infile.readlines()
    for i in range(0, len(lines_gen), 8):
        (firstname,
         lastname,
         email,
         email_confirm,
         password,
         month,
         day,
         year) = lines_gen[i:i+8]

        browser = spynner.Browser()
        browser.load("someurl")
        browser.create_webview()
        #browser.webview.setWindowTitle('Browser Title')
        browser.show()
        `
        print type(firstname)
        browser.wk_fill('input[name="firstname"]', firstname)
        print type(lastname)
        browser.wk_fill('input[name="lastname"]', lastname)
        print type(email)
        browser.wk_fill('input[name="reg_email__"]', email)

Thank you

Import error

Hi,

When I run a code which imports spynner, I get the following error:

> python .\spynner-test.py
Traceback (most recent call last):
  File ".\spynner-test.py", line 10, in <module>
    import spynner
  File "C:\Anaconda\lib\site-packages\spynner\__init__.py", line 2, in <module>
    from browser import *
  File "C:\Anaconda\lib\site-packages\spynner\browser.py", line 56, in <module>
    from PyQt4.QtNetwork import QNetworkCookie, QNetworkAccessManager, QSslConfiguration, QSslCipher
ImportError: DLL load failed: The specified procedure could not be found.

Funny thing is that I can import those things in normal python shell without any problem:

> python
Python 2.7.5 |Anaconda 1.9.2 (64-bit)| (default, Jul  1 2013, 12:37:52) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from PyQt4.QtNetwork import QNetworkCookie, QNetworkAccessManager, QSslConfiguration, QSslCipher
>>>

But after import spynner fails,

> python
Python 2.7.5 |Anaconda 1.9.2 (64-bit)| (default, Jul  1 2013, 12:37:52) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import spynner
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda\lib\site-packages\spynner\__init__.py", line 2, in <module>
    from browser import *
  File "C:\Anaconda\lib\site-packages\spynner\browser.py", line 56, in <module>
    from PyQt4.QtNetwork import QNetworkCookie, QNetworkAccessManager, QSslConfiguration, QSslCipher
ImportError: DLL load failed: The specified procedure could not be found.
>>> from PyQt4.QtNetwork import QNetworkCookie, QNetworkAccessManager, QSslConfiguration, QSslCipher
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed: The specified procedure could not be found.
>>>

Does spynner change some PATH related variable?

Python 3 - spynner

Are there any plans to make spynner work with Python 3?

So far i've tried installing via easy_install3 and it almost works but the following code: lenfield = QString(u'length') in browser.py does not validate, syntax for handling exceptions is wrong and unicode strings are wrong. Also the dependency for beautifulsoup needs to be updated to beautifulsoup4 (this might be a distro specific thing!). Cookielib is also now http.cookiejar in Python 3. That's were I gave up.

Make use of shouldInterruptJavaScript()

In https://github.com/tuky/spynner I made a quick and dirty implementation that overrides QWebPage to automatically terminate stuck javascript. I would really like to see a similar functionality upstream. When developing I hit the bug as described in http://stackoverflow.com/questions/6868286/override-shouldinterruptjavascript-in-qwebpage-with-pyside and I could only solve it by building PyQt4 against Qt5.

Interested? Do you have the same problems with (Py)Qt4? PySide 1.1.1 has the same bug, maybe it can also be built against Qt5?

Best regards!

Crash on browser.select and browser.input in Fedora 20

Hi, all!

I switched to Fedora 20 and my old Spynner scripts stopped working. It turned out that even basic Spynner example is not operating correctly:

import spynner
browser = spynner.Browser()
browser.load("http://www.wordreference.com")
loaded the Generic plugin
True
browser.runjs("console.log('I can run Javascript!')")
browser.show()
browser.load("http://www.wordreference.com")
True
browser.select("#esen")
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/site-packages/spynner-2.17.dev0-py2.7.egg/spynner/browser.py", line 1236, in select
self._runjs_on_jquery("select", jscode)
File "/usr/lib/python2.7/site-packages/spynner-2.17.dev0-py2.7.egg/spynner/browser.py", line 468, in _runjs_on_jquery
if not isinstance(self.get_js_obj_length(res), int):
File "/usr/lib/python2.7/site-packages/spynner-2.17.dev0-py2.7.egg/spynner/browser.py", line 451, in get_js_obj_length
if res.type() != res.Map:
AttributeError: 'NoneType' object has no attribute 'type'

import spynner
browser = spynner.Browser()
browser.load("http://www.wordreference.com")
loaded the Generic plugin
True
import spynner
KeyboardInterrupt
browser.fill("input[name=enit]", "hola")
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/site-packages/spynner-2.17.dev0-py2.7.egg/spynner/browser.py", line 1148, in fill
self._runjs_on_jquery("fill", jscode)
File "/usr/lib/python2.7/site-packages/spynner-2.17.dev0-py2.7.egg/spynner/browser.py", line 468, in _runjs_on_jquery
if not isinstance(self.get_js_obj_length(res), int):
File "/usr/lib/python2.7/site-packages/spynner-2.17.dev0-py2.7.egg/spynner/browser.py", line 451, in get_js_obj_length
if res.type() != res.Map:
AttributeError: 'NoneType' object has no attribute 'type'

System info
$ python
Python 2.7.5 (default, Feb 19 2014, 13:47:28)
[GCC 4.8.2 20131212 (Red Hat 4.8.2-7)] on linux2

$ uname -a
Linux ChaoticRoman 3.14.7-200.fc20.x86_64 #1 SMP Wed Jun 11 22:38:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ yum info libxml2 libxml2-devel
....
Name : libxml2
Arch : x86_64
Version : 2.9.1
Release : 2.fc20
Size : 1.6 M
Repo : installed
...
Name : libxml2-devel
Arch : x86_64
Version : 2.9.1
Release : 2.fc20
Size : 8.8 M
Repo : installed
...

$ pip search autopy
autopylot - Put your Python on autopylot.
autopy - A simple, cross-platform GUI automation toolkit
for Python.
INSTALLED: 0.51 (latest)

$ yum info PyQt4
Loaded plugins: langpacks, refresh-packagekit
Installed Packages
Name : PyQt4
Arch : x86_64
Version : 4.10.2
Release : 5.fc20
Size : 16 M
Repo : installed
...

wait_load() not working

!/usr/bin/python

import spynner
import pyquery

browser = spynner.Browser(debug_level=spynner.INFO)
browser.create_webview()
browser.show()

~ browser.load("http://www.google.com")

browser.load("http://www.ngox.biz")
browser.wait_load()
browser.browse()

spynner not wait to load finished website

---

File "google2.py", line 9, in
browser.load("http://www.ngox.biz")
File "V:\python\python.v2.66_portable\App\lib\site-packages\spynner-2.5.dev0-py2.6.egg\spynner\browser.py", line 460, in load
return self._wait_load(timeout = load_timeout)
File "V:\python\python.v2.66_portable\App\lib\site-packages\spynner-2.5.dev0-py2.6.egg\spynner\browser.py", line 365, in _wait_load
raise SpynnerTimeout("Timeout reached: %d seconds" % timeout)

spynner.browser.SpynnerTimeout: Timeout reached: 10 seconds

keeping sessions going

Thanks for releasing this. I have run into a problem, the workflow I am looking for is:

1. load a login url
2. fill in user/pwd
3. submit
4. wait for authentication to complete and the new page to load
5. navigate to a different url within the same site
6. analyze the page, etc.

I can get up to 4 without any problem. However when I call load() a second time (for setp 5) or if I do browser.webframe.setUrl(..) I can see in the browser window that I have been redirected to the correct page but somehow my session information is lost. The new page does not recognize that I have already authenticated. The cookie information is not being passed on for some reason it seems...

Are all calls to load() treated as new sessions? If so, how do I overcome this?

Why "AttributeError: 'Browser' object has no attribute 'manager'"?

The script worked well for several minutes, suddenly crashed:

Traceback (most recent call last):
File "C:\Users\rwang\Desktop\test\spynner\browser.py", line 232, in _
manager_create_request
reply = QNetworkAccessManager.createRequest(self.manager,
AttributeError: 'Browser' object has no attribute 'manager'

It's so strange.

Small error

On line 1529 of the file browser.py get_cookie_line it's not a method of the class and it's defined outside.

the wordreference example gives a "Javascript console: ReferenceError: Can't find variable: $"

I just commented

# browser.create_webview()
# browser.show()

then it produces:

Page load started
Request: GET http://www.wordreference.com/
User-Agent: Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/534.34 (KHTML, like Gecko) python Safari/534.34
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Request: GET http://www.google-analytics.com/ga.js
User-Agent: Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/534.34 (KHTML, like Gecko) python Safari/534.34
Accept: /
Referer: http://www.wordreference.com/
Request: GET http://b.scorecardresearch.com/beacon.js
User-Agent: Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/534.34 (KHTML, like Gecko) python Safari/534.34
Accept: /
Referer: http://www.wordreference.com/
Javascript console (http://www.wordreference.com/:727):
Request: GET http://www.wordreference.com/2012/homepage.ip2cc.aspx
User-Agent: Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/534.34 (KHTML, like Gecko) python Safari/534.34
Accept: /
Referer: http://www.wordreference.com/
Request: GET http://yui.yahooapis.com/combo?2.9.0/build/yuiloader-dom-event/yuiloader-dom-event.js&2.9.0/build/datasource/datasource-min.js&2.9.0/build/autocomplete/autocomplete-min.js&2.9.0/build/connection/connection_core-min.js&2.9.0/build/json/json-min.js
User-Agent: Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/534.34 (KHTML, like Gecko) python Safari/534.34
Accept: /
Referer: http://www.wordreference.com/
Request: GET http://www.wordreference.com/2012/scripts/2010kmin-conjAC.js
User-Agent: Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/534.34 (KHTML, like Gecko) python Safari/534.34
Accept: /
Referer: http://www.wordreference.com/
Reply: 200/OK - http://www.wordreference.com/
Cache-Control: max-age=86400
Content-Type: text/html; charset=UTF-8
Content-Encoding: gzip
Last-Modified: Tue, 21 Jan 2014 17:11:24 GMT
Accept-Ranges: bytes
ETag: "0c6f2cfcb16cf1:0"
Vary: Accept-Encoding
Server: Microsoft-IIS/8.0
Date: Fri, 02 May 2014 15:12:57 GMT
Request: GET http://www.wordreference.com/images/magnifying_glass_white_26x26.png
User-Agent: Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/534.34 (KHTML, like Gecko) python Safari/534.34
Referer: http://www.wordreference.com/
Accept: /
Reply: 200/OK - http://www.wordreference.com/2012/homepage.ip2cc.aspx
Cache-Control: private
Content-Type: text/javascript; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/8.0
X-AspNet-Version: 4.0.30319
Date: Fri, 02 May 2014 15:12:57 GMT
Reply: 200/OK - http://www.wordreference.com/2012/scripts/2010kmin-conjAC.js
Cache-Control: max-age=86400
Content-Type: application/javascript
Content-Encoding: gzip
Last-Modified: Wed, 18 Sep 2013 19:38:31 GMT
Accept-Ranges: bytes
ETag: "80559da7a6b4ce1:0"
Vary: Accept-Encoding
Server: Microsoft-IIS/8.0
Date: Fri, 02 May 2014 15:12:57 GMT
Reply: 200/OK - http://b.scorecardresearch.com/beacon.js
Content-Type: application/x-javascript
Vary: Accept-Encoding
Content-Encoding: gzip
Expires: Fri, 16 May 2014 15:12:58 GMT
Date: Fri, 02 May 2014 15:12:58 GMT
Connection: keep-alive
Cache-Control: private, no-transform, max-age=1209600
Request: GET http://b.scorecardresearch.com/b?c1=2&c2=14185589&ns__t=1399043578271&ns_c=UTF-8&c8=English to French, Italian, German & Spanish Dictionary - WordReference.com&c7=http://www.wordreference.com/&c9=
User-Agent: Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/534.34 (KHTML, like Gecko) python Safari/534.34
Referer: http://www.wordreference.com/
Accept: /
Reply: 200/OK - http://www.wordreference.com/images/magnifying_glass_white_26x26.png
Cache-Control: max-age=86400
Content-Type: image/png
Last-Modified: Mon, 08 Jul 2013 20:07:18 GMT
Accept-Ranges: bytes
ETag: "09f3ebf167cce1:0"
Server: Microsoft-IIS/8.0
Date: Fri, 02 May 2014 15:12:57 GMT
Content-Length: 534
Reply: 200/OK - http://www.google-analytics.com/ga.js
Date: Fri, 02 May 2014 08:50:12 GMT
Expires: Fri, 02 May 2014 20:50:12 GMT
Last-Modified: Thu, 10 Apr 2014 18:45:31 GMT
X-Content-Type-Options: nosniff
Content-Type: text/javascript
Vary: Accept-Encoding
Content-Encoding: gzip
Server: Golfe2
Cache-Control: public, max-age=43200
Age: 22966
Alternate-Protocol: 80:quic
Reply: 302/Moved Temporarily - http://b.scorecardresearch.com/b?c1=2&c2=14185589&ns__t=1399043578271&ns_c=UTF-8&c8=English to French, Italian, German & Spanish Dictionary - WordReference.com&c7=http://www.wordreference.com/&c9=
Content-Length: 0
Location: http://b.scorecardresearch.com/b2?c1=2&c2=14185589&ns__t=1399043578271&ns_c=UTF-8&c8=English%20to%20French%2C%20Italian%2C%20German%20%26%20Spanish%20Dictionary%20-%20WordReference.com&c7=http%3A%2F%2Fwww.wordreference.com%2F&c9=
Date: Fri, 02 May 2014 15:12:58 GMT
Connection: keep-alive
Set-Cookie: UID=a530f6e-88.221.14.64-1399043578; expires=Thu, 21-Apr-2016 15:12:58 GMT; path=/; domain=.scorecardresearch.com
UIDR=1399043578; expires=Thu, 21-Apr-2016 15:12:58 GMT; path=/; domain=.scorecardresearch.com
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Cache-Control: private, no-cache, no-cache=Set-Cookie, no-store, proxy-revalidate
Request: GET http://b.scorecardresearch.com/b2?c1=2&c2=14185589&ns__t=1399043578271&ns_c=UTF-8&c8=English to French, Italian, German & Spanish Dictionary - WordReference.com&c7=http://www.wordreference.com/&c9=
User-Agent: Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/534.34 (KHTML, like Gecko) python Safari/534.34
Referer: http://www.wordreference.com/
Accept: /
Request: GET http://www.google-analytics.com/__utm.gif?utmwv=5.5.0&utms=1&utmn=1595556708&utmhn=www.wordreference.com&utmcs=UTF-8&utmsr=1366x768&utmsc=32-bit&utmul=fr-fr&utmje=0&utmfl=-&utmdt=English to French, Italian, German & Spanish Dictionary - WordReference.com&utmhid=1576260527&utmr=-&utmp=/&utmht=1399043578501&utmac=UA-187118-1&utmcc=__utma=253024271.1234341873.1399043578.1399043578.1399043578.1;+__utmz=253024271.1399043578.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);&utmu=q~
User-Agent: Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/534.34 (KHTML, like Gecko) python Safari/534.34
Referer: http://www.wordreference.com/
Accept: /
Reply: 204/No Content - http://b.scorecardresearch.com/b2?c1=2&c2=14185589&ns__t=1399043578271&ns_c=UTF-8&c8=English to French, Italian, German & Spanish Dictionary - WordReference.com&c7=http://www.wordreference.com/&c9=
Content-Length: 0
Date: Fri, 02 May 2014 15:12:58 GMT
Connection: keep-alive
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Cache-Control: private, no-cache, no-cache=Set-Cookie, no-store, proxy-revalidate
Reply: 200/OK - http://www.google-analytics.com/__utm.gif?utmwv=5.5.0&utms=1&utmn=1595556708&utmhn=www.wordreference.com&utmcs=UTF-8&utmsr=1366x768&utmsc=32-bit&utmul=fr-fr&utmje=0&utmfl=-&utmdt=English to French, Italian, German & Spanish Dictionary - WordReference.com&utmhid=1576260527&utmr=-&utmp=/&utmht=1399043578501&utmac=UA-187118-1&utmcc=__utma=253024271.1234341873.1399043578.1399043578.1399043578.1;+__utmz=253024271.1399043578.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);&utmu=q~
Pragma: no-cache
Expires: Wed, 19 Apr 2000 11:43:00 GMT
Last-Modified: Wed, 21 Jan 2004 19:51:30 GMT
X-Content-Type-Options: nosniff
Content-Type: image/gif
Date: Thu, 01 May 2014 20:50:15 GMT
Server: Golfe2
Content-Length: 35
Cache-Control: private, no-cache, no-cache=Set-Cookie, proxy-revalidate
Age: 66163
Alternate-Protocol: 80:quic
Reply: 200/OK - http://yui.yahooapis.com/combo?2.9.0/build/yuiloader-dom-event/yuiloader-dom-event.js&2.9.0/build/datasource/datasource-min.js&2.9.0/build/autocomplete/autocomplete-min.js&2.9.0/build/connection/connection_core-min.js&2.9.0/build/json/json-min.js
Vary: Accept-Encoding
Cache-Control: max-age=315360000
Last-Modified: Wed, 30 Apr 2014 07:21:55 GMT
Content-Type: application/javascript; charset=utf-8
Expires: Sat, 05 Sep 2026 00:00:00 GMT
Content-Encoding: gzip
Date: Wed, 30 Apr 2014 07:21:55 GMT
Age: 201063
Via: http/1.0 c1.ycs.bf1.yahoo.com (ApacheTrafficServer/4.0.2), http/1.1 l9.ycs.dcb.yahoo.com (ApacheTrafficServer/4.0.2)
Server: ATS
Connection: keep-alive
Page load finished (24016 bytes): http://www.wordreference.com/ (successful)
Run Javascript code: $('option:selected', $('#esen').parents('select')[0]).removeAttr('selected');
$('#esen').attr('selected', 'selected');

Javascript console: ReferenceError: Can't find variable: $
Run Javascript code: $('input[name=w]').val('hola')
Javascript console: ReferenceError: Can't find variable: $

css are not loaded when reading html on local machine

Hi,

I crawled several webpages and saved on my computer. When I opened the html file using spynner, I found it didn't load the css file (The css file is there). Is it an issue or did I miss something?

Thanks,
KJ