pallets / werkzeug Goto Github PK

I want to deprecate the builtin template language because it uses deprecated Python functionality and is not too helpful. Jinja2 can be configured to nearly the same syntax, so there is an upgrade path.

It would be nice if werkzeug.test.Client would allow one to easily forge cookies for testing purposes. After some digging, I've found out that the following works.

def forge_cookie(name, value, domain, path):
  return cookielib.Cookie(version=0, name=name, value=value, port=None,
      port_specified=False, domain=domain, domain_specified=False,
      domain_initial_dot=False, path=path, path_specified=True, secure=False,
      expires=None, discard=True, comment=None, comment_url=None,
      rest={'HttpOnly':None}, rfc2109=False)

c = forge_cookie('session',
        SecureCookie({"openid":"special"}, self.secret_key).serialize(),
        'localhost.local',
        '/')

client.cookie_jar.set_cookie(c)

It might be nice to have a add_cookie method in the Client with some sane defaults to make this easier. If that cannot be done, I think adding this example to the documentation might be helpful for others.

Werkzeug needs a new tutorial. No SQLAlchemy, no thread locals. Maybe use redis instead of sqlite directly like we did in the Flask tutorial.

Hi everyone,

as my colleague posted on the tipfy dev list, there is a bug in Werkzeug.urls.redirect that ends up double quoting the location-headers. That function calls iri_to_uri in case the location is a unicode object. As of the today this is not only not necessary anymore as it will be done in Response.get_wsgi_headers anyway[1], it even causes harm. It is caused by the fact that werkzeug.utils.iri_to_uri might return a unicode and sets that as the location header and get_wsgi_header is quoting it again.

Removing those two lines in url.redirect is an easy and necessary fix for it. The work-a-round for now is to ensure you pass an quoted non-unicode-object as the location to the function.

[1] https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/wrappers.py#L951

Since the introduction of the new Werkzeug Web site, the "wiki in 30 minutes" screencast and source code has disappeared. While it is probably a bit out of date, everyone loves screencasts and everyone loves wikis, so it is probably a good idea to make it accessible again (especially since I plan on maybe using it as a base for a new Flask tutorial).

It returns the first component of Request.access_route -- instead it should be the last, according to my understanding of how X-Forwarded-For works.

There should be two branches after the upcoming release. 0.7 will be prepared for the upcoming removal of the magic imports and contain some deprecation warnings that however are pending.

After that we will have a 0.7.x branch that will just fix issues while the 1.0 branch is approaching a smaller distribution with less magic.

Since i think pyinotify 0.8 the event codes had moved to the module level. Adding an import and changing getattr(EventsCodes, a) to getattr(pyinotify, a) appers to fix this.

In the following scenario:

http://paste.pocoo.org/show/319042/

I would expect the last line to be '/a%2Fb'. The actual shown output should only be correct in the case of the following rule: "/path:test".

The following patch make this happen:

http://paste.pocoo.org/show/319051/

There was a bug in there and the fix was the switching of two arguments. However it should be ensured that this was the only bug and that it now works as expected. Unfortunately it's slightly underdocumented there.

The builtin templating language should be deprecated. Reasoning: uses deprecated functionality and cannot be ported to Python 3 at all without going through the new ast compiler of Python 2.6.

We should recommend proper templating languages instead. The last internal user of Werkzeug is the testapp and debugger.

Related tickets: #22, #24

Some things ended up in contrib that really shouldn't have been there.

When last_modified has a non-zero microseconds component, is_resource_modified() will almost always return true under normal circumstances. HTTP dates don't store microseconds, so the incoming If-Modified-Since header will be parsed into a datetime object with 0 as its microseconds component, and compared against a last_modified value which is slightly larger.

A simple solution would be to do last_modified = last_modified.replace(microsecond=0) in is_resource_modified(), although I guess that could occasionally break applications that really do change resources more than once per second. This should at least be documented as a gotcha, I think.

Vim uses temporary files when saving, so this message appears upon saving a file in vim while running the inotify reloader:

[2011-05-08 15:06:03,983 pyinotify ERROR] The pathname '/home/romme/src/proj
ects/vrusha/application.py' of this watch <Watch wd=146 path=/home/romme/src
/projects/vrusha/application.py mask=3078 proc_fun=<function signal_changed 
at 0x91e16f4> auto_add=False exclude_filter=<function <lambda> at 0x8f364c4>
 dir=False > has probably changed and couldn't be updated, so it cannot be t
rusted anymore. To fix this error move directories/files only between watche
d parents directories, in this case e.g. put a watch on '/home/romme/src/pro
jects/vrusha'.

However, the change is still getting detected successfully.

We need another bugfix release for the 0.6 series:

• f506104 for a multipart parsing bug
• 1f44af4 for a windows reloading bug

http://hg.moinmo.in/moin/2.0-dev/file/tip/contrib/wsgi/

maybe some stuff from there could be useful in werkzeug (or at least the ideas):

• profiling
• proxy (done differently than in werkzeug)

If it was in werkzeug, I could get rid of it there and maybe some other people would find it useful, too.

Hi,
The easter egg is completely broken ! (malformed closing tags for title and pre). This is not acceptable; please fix this ASAP !

To use subdomain routing you are currently basically required to pass the server_name argument. However, once you do this, a ValueError will be raised for any other domain (whereas before, your werkzeug would happily work with any domain you would point to it).

In some cases, a workaround is:

     try:
        urlmap = urls.patterns.bind_to_environ(environ, server_name="example.org")
    except ValueError:
        urlmap = urls.patterns.bind_to_environ(environ)

The code in bind_to_environ() seems complicated enough that duplicating it should be avoided.

I wonder whether werkzeug could expose a more flexible API here. For starters, it could accept a list of possible domains for server_name. It should also support placeholders - say, server_name=['*.com'] to allow all .com domains.

Ideally, I could route multiple domains, with individual routes only working on particular domains.

In docs/test.rst:

   .. attribute:: environ_base

      The dict used as base for the newly create environ.

   .. attribute:: environ_base

      A dict with values that are used to override the generated environ.

I’d provide a patch but I’m not sure which one should be kept.

Since ''.split(',') == [''] and bool(['']) == True

We're running behind varnish and only caching anonymous (non-cookied) users so sometimes we've got the header and sometimes not.

I'd been manually grabbing the header until I discovered this middleware (when I went to write some myself).

Here's a patch:

http://paste.pocoo.org/show/447542/

Because setuptools does not order special versions by update date it would be necessary to rewrite or remove old links from old descriptions to update to the latest development version.

Either ==dev should no longer be supported or one should write a tool that rewrites old entries. Right now Werkzeug==dev points to a nonexisting hg repo.

Python version 3.2 finally resolves the long-standing issue of Unicode support in WSGI headers (PEP 3333). Thus, it is now possible (and desirable) to port Werkzeug to Python 3 and embrace the future.

See also: http://docs.python.org/dev/whatsnew/3.2.html

The BaseWSGIServer (or more precisely the Python HTTPServer) is unreliable with multiple concurrent connections.
The socket will reset connections when to many requests come in. It's configurable with the request_queue_size attribute of the server.

Solution is easy, just add this:

request_queue_size = 128

here: https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/serving.py#L301

How to reproduce:

ab -c 64 -n 128 http://localhost:8080

Currently one can happily add newlines to headers and nothing will stop you from doing that. However as far as I am concerned, there are few use cases for that and there are security implications associated with it. Furthermore many servers don't split up headers properly if they contain newlines so if one passes untrusted user input to the header that could cause problems.

Reddit hat that problem recently.

In werkzeug/templates.py it says

def parse_python(self, expr, type='exec'):
    if isinstance(expr, unicode):
        expr = '\xef\xbb\xbf' + expr.encode('utf-8')
    try:
        node = parse(expr, type)

But adding a BOM to the front of an expression in Python 2.7 inserts a node of type 340 into the parse tree, so that Python's stdlib dies before parse() ever returns with the exception:

  File "/home/brandon/rover/goby/venv/lib/python2.7/site-packages/werkzeug/templates.py", line 112, in parse_python
    node = parse(expr, type)
  File "/usr/lib/python2.7/compiler/transformer.py", line 53, in parse
    return Transformer().parseexpr(buf)
  File "/usr/lib/python2.7/compiler/transformer.py", line 132, in parseexpr
    return self.transform(parser.expr(text))
  File "/usr/lib/python2.7/compiler/transformer.py", line 124, in transform
    return self.compile_node(tree)
  File "/usr/lib/python2.7/compiler/transformer.py", line 167, in compile_node
    raise WalkerError, ('unexpected node type', n)
WalkerError: ('unexpected node type', 340)

Is there another solution to performing this step in Werkzeug that would not involve passing Python's compiler module a node type that it cannot deal with? Or is this a bug that we should report to the standard library folks (or will they ignore it anyway since "compiler" is deprecated — and is the problem here simply that Werkzeug is using a deprecated module)?

Thanks for any help or pointers!

When the process receives SIGTERM, it should shut down and exit, with as few operations as possible and without printing anything.

A werkzeug.serving.run_simple process receiving SIGTERM generally result in a return code of 141 (symptom of an un/mishandled SIGPIPE), and when using the reloader the process goes zombie (it has to be killed manually, as the port stays bound).

Adding a signal handler for SIGTERM which simply invokes sys.exit(0) is sufficient to fix the issue (in that there is no more misbehavior of the process), but I am not sure it's the actually correct fix.

The routing system should use a proper parser instead of falling back to eval which also has pretty weird side effects.

This may be due to a messy python package configuration on Ubuntu 10.04 (mix of apt-get and easy_install), or I might be doing something really stupid. Regardless,

Linux redacted 2.6.32-32-generic #62-Ubuntu SMP Wed Apr 20 21:52:38 UTC 2011 x86_64 GNU/Linux
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
werkzeug.version == '0.7'
pyinotify.version=='0.9.2'

Debug output (when starting a Flask app with debug=True):

• Running on http://127.0.0.1:5000/
• Restarting with reloader: inotify events
  Traceback (most recent call last):
  File "bomtracker.py", line 19, in
  app.run(debug=True)
  File "/usr/local/lib/python2.6/dist-packages/Flask-0.7.2-py2.6.egg/flask/app.py", line 553, in run
  return run_simple(host, port, self, **options)
  File "/usr/local/lib/python2.6/dist-packages/Werkzeug-0.7-py2.6.egg/werkzeug/serving.py", line 609, in run_simple
  run_with_reloader(inner, extra_files, reloader_interval)
  File "/usr/local/lib/python2.6/dist-packages/Werkzeug-0.7-py2.6.egg/werkzeug/serving.py", line 528, in run_with_reloader
  reloader_loop(extra_files, interval)
  File "/usr/local/lib/python2.6/dist-packages/Werkzeug-0.7-py2.6.egg/werkzeug/serving.py", line 436, in reloader_loop
  reloader(fnames, interval=interval)
  File "/usr/local/lib/python2.6/dist-packages/Werkzeug-0.7-py2.6.egg/werkzeug/serving.py", line 464, in _reloader_inotify
  mask = reduce(lambda m, a: m | getattr(EventsCodes, a), mask, 0)
  File "/usr/local/lib/python2.6/dist-packages/Werkzeug-0.7-py2.6.egg/werkzeug/serving.py", line 464, in
  mask = reduce(lambda m, a: m | getattr(EventsCodes, a), mask, 0)
  AttributeError: type object 'EventsCodes' has no attribute 'IN_DELETE_SELF'

If the server secret leaks an attacker can forge cookies and get the server to unpickle them, which gives him remote code execution.

Therefore, something else should be used for serialization. Maybe json.

Proof of concept:

Python 2.7.1 (r271:86832, Jan  6 2011, 11:51:37) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle; pickle.loads("cos\nsystem\n(S'whoami'\ntR.")
fxkr

Current SecureCookie implementation hashes the cookie against a secret key to guarantee authenticity. On the other hand, if the secret key is compromised or the hashing function implementation is broken, an attacker suddenly has power to execute arbitrary code, where normally it would be limited to session hi-jacking.

Proof of concept: hcarvalhoalves/werkzeug@792e14e

A better option would be using JSON by default. Problem is that user code may be relying on the fact session data is pickable today and use it for more than primitive data types, so I don't know an upgrade path if this is fixed by default.

Otherwise, add a disclaimer to the documentation stating that SecureCookie pickles data from the client, so security concerned developers can derive their own subclass.

When using werkzeug.Response.make_conditional, it will sometimes turn the response into a "304 Not modified" response, but will leave all the headers intact. In particular, it will leave the "Content-Length" and "Content-Type" headers of the original response. This is against the RFC 2616, which reads in section 10.3.5 on the "304 Not modified" response:

If the conditional GET used a strong cache validator (see section 13.3.3), the response SHOULD NOT include other entity-headers. Otherwise (i.e., the conditional GET used a weak validator), the response MUST NOT include other entity-headers; this prevents inconsistencies between cached entity-bodies and updated headers.

From the bug reports I get from my users, a 304 response with a non-zero Content-Length header seems to trip up at least one web browser -- Safari -- and is possibly problematic for other browsers. The workaround I use it to manually check the status code of the response after calling its make_conditional method, and to remove any remaining Content-Length and Content-Type headers.

Those headers should get removed in get_wsgi_headers method when the status is 304, and it happens most of the time -- except that the header is getting added in the very next block of code if it wasn't there in the first place and the response is a string. My solution is a little naive, but works for me:

The constructor for pylibmc's memcached client interface isn't correctly being called by werkzeug.contrib.cache.

Can you cherry-pick sean-@d6aaee1

? I can submit it as a pull request, but the patch is so trivial that I don't think it's worth the effort.

Speaking of cache, it would be nice to be able to pass options down to werkzeug.contrib.cache via a set of config options. Right now Flask-Cache makes an attempt to do that, but I'm not sure how CACHE_OPTIONS or CACHE_ARGS actually gets passed down to pylibmc. Ideally I'd like to set:

CACHE_OPTIONS = { 'binary': False,
'behaviors': { 'tcp_nodelay': True,
'ketama': True } }

in a config file and have it used to configure the library. The attempt to auto-detect which memcache client library to use seems like a well intended idea, but is rough. As is, I have to set these options by hand after cache.init_app(app).

Hi Armin,

We use Werkzeug's debug middleware extensively when deploying our WSGI apps on development and QA environments and we've love to use the one-click "lodge it" button when things break, but we'd prefer not to publish our stack traces on paste.pocoo.org.

Do you think it would be possible to make the paste destination configurable?

Many people want to use the integrated Werkzeug webserver for local applications and there it would be nice if one could restart and shutdown the server from the application. Maybe we can find a way to make this happen.

One of the shortcomings I've found in every existing python web session library is that they don't support secret key rotation. It's one of the best ways I know of to protect against brute force and hash algorithm attacks, so I've been looking into implementing it with Werkzeug SecureCookie (and Flask sessions). Getting my web application to periodically generate a new secret key seems easy enough. The tricky part is getting the securecookie code to manage multiple keys properly. I think it would have to work something like this:

1. Keep a list (or ring buffer) of secret keys instead of just one.
2. When signing a cookie, use the newest key that's at least a few minutes old.
   (Don't use brand new keys because other web server instances might not have them yet.)
3. When validating a signed cookie, try every key in the list.
   (In case a new key was added after the cookie was last signed.)
4. Allow the web application to periodically rotate the keys, adding a new one and deleting the oldest one.

I looked into implementing this behavior in a SecureCookie subclass, but it looks like the hmac-related code is pretty well entwined with the other code in serialize() and unserialize(), making it a little difficult to do cleanly. (Especially item 3: validating against multiple keys.) Would you consider breaking the signing and validation code into separate methods, in such a way that key rotation would be easy to implement?

If it worked well, perhaps key rotation could even become a standard feature in SecureCookie. (That would be reason enough for some of us to use it instead of the various other web session libraries available for python.)

Hi,

We(binfire.com) use werkzeug in our file handling backend(a gevent pywsgi server).
The upload is made by uploadify(Flash-based upload,http://www.uploadify.com/).

We observed that a specific file was failing to upload to a specific path,i analyzed the results and found out that adding 4 bytes by changing one form parameter(called folder that uploadify sends) fix it.

I did a very long debug session in winpdb and found out that werkzeug is failing here:
for line in iterator:
if not line:
raise ValueError('unexpected end of stream')
(line 262 in formparser.py , Werkzeug 0.6.2)

but if i just add those 4 bytes it works ok!

here is a zip with all the test code:

http://www.binfire.com:8080/serve_file/3af80053e94d4b79b219d076b633f2db

contents:

bad.pickle -> a enviroment(as in wsgi env) that doesn't work,it includes the uploaded file as string
good.pickle -> same as bad but work :)
test_pickle.py -> a script that recives pickle name and tries to run parse_form_date on the env
test_bad_http.py -> a sample gevent.pywsgi server with uploadify that breaks the code depending if you add in static/js.js folder = '/collaboration/files/8/test' or omit the test
part(omitting make it fail)
binfire_users-April_2_11.xls -> the file that started all this,it is a 0 file(ascii 0) with the same size as original and causes the same effect.
static -> bunch of js/swf files for uploadify.

Thanks,

P.S.:this is a aswome library and almost the only one that does streaming(i.e. doesn't store all
the data in memory)
upload correctl

Right now Werkzeug uses the ignore error handling for incoming unicode data. This was a mistake, it should have been replace. The reason for that is that it's hard to spot if things are removed because of improper encoding.

While this is a backwards incompatible change it still should be done.

The other problem is that Werkzeug supports a fallback:charset error handling. This instead should have been implemented with the help of the codecs module as an optional error handler that can be registered.

We want to deprecate the internal templates, this is a current user.

When joining stuff, unexpected URLs happen if joined name has a colon inside:

>>> import werkzeug as w
>>> h = w.Href('/')
>>> h('foo')
'/foo'                        <--- ok
>>> h('foo:bar')
'foo:bar'                     <--- resulting in browser trying wrong protocol
>>> h('a&o')
'/a%26o'                      <--- ok
>>> h(u'übel')
'/%C3%BCbel'                  <--- ok

Armin found that this is probably a side effect of using urljoin, that it should be fixed either in werkzeug or stdlib and proposed this workaround:

>>> urlparse.urljoin('/', 'foo:bar')
'foo:bar'
>>> urlparse.urljoin('/', './foo:bar')
'/foo:bar'

werkzeug.test does not respect headers and requested method on redirections for HTTP 307 requests.

Example:

from werkzeug import Client
from werkzeug import Headers


headers = Headers(['AUTHORIZATION', 'BASIC ' + 'user:pass'.encode('base64')])

client = Client(App())
response = client.post('/Accounts/1/Images/5',
                       data={'foo': 'bar'},
                       headers=headers)
assert response.status_code 307  # works fine

# testing with following redirects
response = client.put('/Accounts/1/Images/5',
                      data={'foo': 'bar'},
                      headers=headers,
                      follow_redirects=True)
assert response.status_code 201  # failure, returns 401!

The Rule endpoint that handles this is:

from werkzeug.utils import redirect

def redirect307(request, account_sid, images_sid):
    base, _, redirect_to = request.url.partition('/Accounts/' + account_sid)
    response = redirect(base + redirect_to, code=307)

    # extend the headers, inheriting the authentication information
    response.headers.extend(request.headers)
    return response

When digging into werkzeug/test.py, the comment in the test client
redirect handler says:

# the redirect request should be a new request, and not be based on
# the old request
redirect_kwargs = {}
redirect_kwargs.update({
    'path':             script_root,
    'base_url':         base_url,
    'query_string':     qs,
    'as_tuple':         True,
    'buffered':         buffered,
    'follow_redirects': False,
})
environ, rv = self.redirect_client.open(**redirect_kwargs)
status_code = int(rv[1].split(None, 1)[0])

Notice that the redirect kwargs do not respect the METHOD nor the
header authentication values. Is this intended?

[17:01] I'm using Flask's app.errorhandler to return a custom 404 page. I have a before_request function that might sometimes want to cause a 404 response. However, if I return NotFound() from this function, I get the stock 404 page and not my custom one. If I raise NotFound(), the exception does not get caught and it becomes a 500. I could instead return my_custom_404_handler() here, but that would be less general and thus inferior.

[17:57] open an issue

According to the HTML5 specs this is a valid html element - http://dev.w3.org/html5/markup/syntax.html#syntax-attr-empty.

Currently it is supported with html.input(disabled=True), which has confusing semantic meaning. An empty string (or None?) could be a better semantic choice - html.input(disabled='') / html.input(disabled=None).

The current werkzeug debugger has no security mechanism in place to safeguard against urls crafted by an attacker in a way to cause arbitrary code execution. Refer to http://news.ycombinator.com/item?id=2802281 for a discussion of the problem (albeit within the context of Rails). It'd be very helpful if werkzeug was modified in a way to defend against these attacks.

Thanks,
John

Currently Werkzeug provides a magical import system which causes some troubles on GAE and other python implementations. This really shouldn't be necessary.

This can't be done for 0.7 but we can certainly aim for that for 1.0

Pasting my patch file with comments, should be self-explanatory:

#Werkzeug's FileMultiDict class has a bug: it doesn't act like multi-dict,
#it acts like a regular dict insttead, i.e. it only stores one value (file) 
#per key.

import mimetypes
from werkzeug.datastructures import FileStorage, FileMultiDict

# New implementation of FileMultiDict.add_file():
def new_add_file(self, name, file, filename=None, content_type=None):
    if isinstance(file, FileStorage):
        value = file
    else:
        if isinstance(file, basestring):
            if filename is None:
                filename = file
            file = open(file, 'rb')
        if filename and content_type is None:
            content_type = mimetypes.guess_type(filename)[0] or \
                           'application/octet-stream'
        value = FileStorage(file, filename, name, content_type)

    # this line is critical to this patch.
    # Werkzeug used to do self[name] = value
    self.add(name, value)

# patch:
FileMultiDict.add_file = new_add_file

I have a Linux Mint system. Domain name "localhost" is resolved to both "::1" (IPv6) and "127.0.0.1" (IPv4).

When Werkzeug's run_simple binds to "localhost", it uses the first corresponding address - in case of my system, the IPv6 one. But, for some reason, Mozilla Firefox seems to interpret "localhost" as 127.0.0.1 only. Thus, opening "localhost:5000" in Firefox results in error - while in Chrome it works fine.

I think, for the time being, Werkzeug should prefer IPv4 addresses over IPv6 ones when binding to a domain name (or, at least, when binding to localhost).

See there, would be nice if we could remove that monkeypatching:

http://hg.moinmo.in/moin/2.0/file/e6f585ac9766/MoinMoin/util/monkeypatch.py

This code still triggers an infinite redirect because the routing system does not check if the default based redirect is unnecessary.

m = Map([
    Rule('/', defaults={'page': 1, 'bar': False}, endpoint='x'),
    Rule('/<int:page>', defaults={'bar': False}, endpoint='x'),
    Rule('/bar/', defaults={'page': 1, 'bar': True}, endpoint='x'),
    Rule('/bar/<int:page>', defaults={'bar': True}, endpoint='x')
])
a = m.bind('example.com')

assert a.match('/') == ('x', {'page': 1, 'bar': False})
assert a.match('/2') == ('x', {'page': 2, 'bar': False})
assert a.match('/bar/') == ('x', {'page': 1, 'bar': True})
assert a.match('/bar/2') == ('x', {'page': 2, 'bar': True})