aio-libs / yarl

Yet another URL library

yarl

The module provides handy URL class for URL parsing and changing.

Introduction

Url is constructed from str:

>>> from yarl import URL
>>> url = URL('https://www.python.org/~guido?arg=1#frag')
>>> url
URL('https://www.python.org/~guido?arg=1#frag')

All url parts: scheme, user, password, host, port, path, query and fragment are accessible by properties:

>>> url.scheme
'https'
>>> url.host
'www.python.org'
>>> url.path
'/~guido'
>>> url.query_string
'arg=1'
>>> url.query
<MultiDictProxy('arg': '1')>
>>> url.fragment
'frag'

All url manipulations produce a new url object:

>>> url = URL('https://www.python.org')
>>> url / 'foo' / 'bar'
URL('https://www.python.org/foo/bar')
>>> url / 'foo' % {'bar': 'baz'}
URL('https://www.python.org/foo?bar=baz')

Strings passed to constructor and modification methods are automatically encoded giving canonical representation as result:

>>> url = URL('https://www.python.org/шлях')
>>> url
URL('https://www.python.org/%D1%88%D0%BB%D1%8F%D1%85')

Regular properties are percent-decoded, use raw_ versions for getting encoded strings:

>>> url.path
'/шлях'

>>> url.raw_path
'/%D1%88%D0%BB%D1%8F%D1%85'

Human readable representation of URL is available as .human_repr():

>>> url.human_repr()
'https://www.python.org/шлях'

For full documentation please read https://yarl.aio-libs.org.

Installation

$ pip install yarl

The library is Python 3 only!

PyPI contains binary wheels for Linux, Windows and MacOS. If you want to install yarl on another operating system (like Alpine Linux, which is not manylinux-compliant because of the missing glibc and therefore, cannot be used with our wheels) the the tarball will be used to compile the library from the source code. It requires a C compiler and and Python headers installed.

To skip the compilation you must explicitly opt-in by using a PEP 517 configuration setting pure-python, or setting the YARL_NO_EXTENSIONS environment variable to a non-empty value, e.g.:

$ pip install yarl --config-settings=pure-python=false

Please note that the pure-Python (uncompiled) version is much slower. However, PyPy always uses a pure-Python implementation, and, as such, it is unaffected by this variable.

Dependencies

YARL requires multidict library.

API documentation

The documentation is located at https://yarl.aio-libs.org.

Why isn't boolean supported by the URL query API?

There is no standard for boolean representation of boolean values.

Some systems prefer true/false, others like yes/no, on/off, Y/N, 1/0, etc.

yarl cannot make an unambiguous decision on how to serialize bool values because it is specific to how the end-user's application is built and would be different for different apps. The library doesn't accept booleans in the API; a user should convert bools into strings using own preferred translation protocol.

Comparison with other URL libraries

• furl (https://pypi.python.org/pypi/furl)

  The library has rich functionality but the furl object is mutable.

  I'm afraid to pass this object into foreign code: who knows if the code will modify my url in a terrible way while I just want to send URL with handy helpers for accessing URL properties.

  furl has other non-obvious tricky things but the main objection is mutability.

• URLObject (https://pypi.python.org/pypi/URLObject)

  URLObject is immutable, that's pretty good.

  Every URL change generates a new URL object.

  But the library doesn't do any decode/encode transformations leaving the end user to cope with these gory details.

Source code

The project is hosted on GitHub

Please file an issue on the bug tracker if you have found a bug or have some suggestion in order to improve the library.

Discussion list

aio-libs google group: https://groups.google.com/forum/#!forum/aio-libs

Feel free to post your questions and ideas here.

Authors and License

The yarl package is written by Andrew Svetlov.

It's Apache 2 licensed and freely available.

yarl's Issues

Passwords without users

Currently, yarl refuses to set an password if a user isn't set. That's a bit unfortunate because that’s exactly what you need if you’re building redis URLs: https://redis-py.readthedocs.io/en/latest/#redis.StrictRedis.from_url

Quote method raises on some nasty strings

I'm writing REST APIs using Aiohttp. I test requests by passing naughty strings from here: https://github.com/minimaxir/big-list-of-naughty-strings

Before including Yarl into Aiohttp tests passed properly. But after upgrading to Aiohttp 1.1.2 Yarl began to raise errors. Here are some cases:

<script src="\\%(jscript)s"></script>
http://a/%%30%30
1;DROP TABLE users  // this one is parsed as int
%x('ls -al /')
%*.*s

Dont encode : in path

See aio-libs/aiohttp#1433

[CI] Doctests not running

Should run doctests in Travis CI.

Properly handle non-UTF8 percent encoded characters in query part

See for the reference aio-libs/aiohttp#1364

Unexpected compare behaviour

from yarl import  URL
URL('http://test.com/') == URL('http://test.com')
False

It should be True

Can not use subclass' of int for query={'key': <int subclass>}

Should be able to pass subclass' of both str and int to URL query parameter.

Example:

from yarl import URL

class Example(int):
    def __new__(cls, value):
        return super().__new__(cls, value)

url = URL.build(scheme='https', host='github.com',
                path='/aio-libs/yarl/search?utf8=%E2%9C%93&q=elif+type%28v%29+%3D%3D+int%3A+&type=',
                query={'test': Example(0)})

print(url)

double unquote

Yarl unquotes query string twice. To test it:

    def test_yarl_double_unquote():
        url = 'http://base.place?' + parse.urlencode({'a': '/////'})
        query = parse.urlencode({'url': url})
        full_url = 'http://test_url.aha?' + query

        yurl = yarl.URL(full_url)
>       assert yurl.query['url'] == url
E       AssertionError: assert 'http://base.place?a=/////' == 'http://base.pl...%2F%2F%2F%2F%2F'
E         - http://base.place?a=/////
E         + http://base.place?a=%2F%2F%2F%2F%2F

tmpbuf - compiling error

Hello,

I get this compiling error:

`sudo pip3 install yarl
Downloading/unpacking yarl
Downloading yarl-0.8.1.tar.gz (121kB): 121kB downloaded
Running setup.py (path:/tmp/pip-build-nm8vbb7v/yarl/setup.py) egg_info for package yarl

Error compiling Cython file:
------------------------------------------------------------
...
        if ch in ALLOWED:
            PyUnicode_WriteChar(ret, ret_idx, ch)
            ret_idx +=1
            continue

        tmpbuf = PyUnicode_AsUTF8AndSize(ch, &tmpbuf_size)
              ^
------------------------------------------------------------

yarl/_quoting.pyx:117:15: Assignment to non-lvalue 'tmpbuf'
Traceback (most recent call last):
  File "<string>", line 17, in <module>
  File "/tmp/pip-build-nm8vbb7v/yarl/setup.py", line 27, in <module>
    extensions = cythonize(extensions)
  File "/usr/lib/python3/dist-packages/Cython/Build/Dependencies.py", line 840, in cythonize
    cythonize_one(*args[1:])
  File "/usr/lib/python3/dist-packages/Cython/Build/Dependencies.py", line 959, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: yarl/_quoting.pyx
Compiling yarl/_quoting.pyx because it changed.
Cythonizing yarl/_quoting.pyx
Complete output from command python setup.py egg_info:

Error compiling Cython file:

---

...

    if ch in ALLOWED:

        PyUnicode_WriteChar(ret, ret_idx, ch)

        ret_idx +=1

        continue



    tmpbuf = PyUnicode_AsUTF8AndSize(ch, &tmpbuf_size)

          ^

---

yarl/_quoting.pyx:117:15: Assignment to non-lvalue 'tmpbuf'

Traceback (most recent call last):

File "", line 17, in

File "/tmp/pip-build-nm8vbb7v/yarl/setup.py", line 27, in

extensions = cythonize(extensions)

File "/usr/lib/python3/dist-packages/Cython/Build/Dependencies.py", line 840, in cythonize

cythonize_one(*args[1:])

File "/usr/lib/python3/dist-packages/Cython/Build/Dependencies.py", line 959, in cythonize_one

raise CompileError(None, pyx_file)

Cython.Compiler.Errors.CompileError: yarl/_quoting.pyx

Compiling yarl/_quoting.pyx because it changed.

Cythonizing yarl/_quoting.pyx

---

`

TypeError in URL.with_query

Hello, I have a problem with generating query_params. I try to make query

session = aiohttp.ClientSession()
session.request('get', 'some_url', {'p1': 1, 'p1': 2, 'p2': 3})

and got TypeError

  File "/lib/python3.5/site-packages/aiohttp/client.py", line 553, in __await__
    resp = yield from self._coro
  File "/lib/python3.5/site-packages/aiohttp/client.py", line 198, in _request
    proxy=proxy, proxy_auth=proxy_auth, timeout=timeout)
  File "lib/python3.5/site-packages/aiohttp/client_reqrep.py", line 79, in __init__
    url2 = url.with_query(params)
  File "/lib/python3.5/site-packages/yarl/__init__.py", line 675, in with_query
    raise TypeError("Invalid variable type")
TypeError: Invalid variable type

After some research I understand that Mapping with key: Sequence works with MultiDict. But this is not obvious. Maybe it should cast dict into MultiDict or return more descriptive warning or add logic for parsing Sequences in Mapping section

Drop strict mode

We don't respect it by URL cloning anyway.
I don't see a use case where strict mode is useful.

Write sphinx doc and docstrings

path_qs

What do you think about adding path_qs property, that represent path with query string? It will be convenient with aiohttp.TestClient for cheking of relative urls with query string.

Core dump

yarl.URL('j\x1a\udcf4q\udcda/\udc97g\udcee\udccb\x0ch\udccb\x18\udce4v\x1b\udce2\udcce\udccecom/y\udccepj\x16')

with_path?

There doesn’t seem to be a way to completely replace the path with a different one? Or at least to discard the path?

Same goes for fragments when I think of it. Are those design decisions or just lack of time?

Document encoded parameter

See also aio-libs/aiohttp#2198

There should be a way to add a empty fragment with URL.with_fragment()

According to the HTML5 spec (6.7.9) the empty fragment # and the special fragment #top links to the top of the page.

I would like to be able to create an URL with an empty fragment using the URL.with_fragment() method.

Currently both URL.with_fragment(None) and URL.with_fragment("") removes the fragment from the URL.

I suggest that with an empty string parameter it should create an empty fragment #.

Thanks for aiohttp and this great library!

URL.update_query() should share implementation with URL.with_query()

The only difference is using MultiDict.update() instead of MultiDict.extend(), parameters parsing should be the same.

Support `raw_path_qs` property

It's nice to build a request with raw_path_qs, but I only find path_qs.

Usage, eg:

f'{url.method} {url.raw_path_qs} HTTP/1.1'

->

GET /path?query HTTP/1.1

Properly quote params

See aio-libs/aiohttp#1423

isinstance(url, URL) returns False for BaseURL

Specifically, this evaluates to false: isinstance(URL('http://example.com').with_fragment(None), URL)

I tested it as follows:

[adrian@chakra yarl]$ python3 -m venv venv
[adrian@chakra yarl]$ . venv/bin/activate
(venv) [adrian@chakra yarl]$ pip install yarl
Collecting yarl
  Downloading yarl-0.9.5-cp35-cp35m-manylinux1_x86_64.whl (165kB)
    100% |████████████████████████████████| 174kB 2.1MB/s 
Collecting multidict>=2.0 (from yarl)
  Downloading multidict-2.1.4-cp35-cp35m-manylinux1_x86_64.whl (357kB)
    100% |████████████████████████████████| 358kB 746kB/s 
Installing collected packages: multidict, yarl
Successfully installed multidict-2.1.4 yarl-0.9.5
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(venv) [adrian@chakra yarl]$ echo "from yarl import URL" >> test.py
(venv) [adrian@chakra yarl]$ echo "print(isinstance(URL('http://example.com').with_fragment(None), URL))" >> test.py
(venv) [adrian@chakra yarl]$ python test.py 
False
(venv) [adrian@chakra yarl]$ 

I realize that there are many chances that I am the one doing something wrong here, as I found this issue debugging an exception raised by trivial aiohttp code.

If this is as expected, I will proceed to report the issue downstream to aiohttp.

Proper way to construct URL from URL components (host and port)

Lets assume I have host and port numbers of plain HTTP web server, what is the best way to construct proper URL from them?

Looks like default-created URL is considered as relative URL:

>>> from yarl import URL
>>> URL().with_host('::1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bob/stuff/cscenter/testing/env/lib/python3.5/site-packages/yarl/__init__.py", line 604, in with_host
    raise ValueError("host replacement is not allowed "
ValueError: host replacement is not allowed for relative URLs
>>> 

Currently I'm constructing a dummy URL and replacing host and port in it:

>>> from yarl import URL
>>> URL('http://example.org').with_host('::1').with_port(8080)
URL('http://::1:8080')
>>> 

Document-relative urls like '?a=b'

URL.with_query() currently sets new instance path to '/' if it was empty.

>> URL.build(query={'a': '1'})
<< URL('/?a=1')

It prevents building relative url with only query seqment, like '?a=b'.
On the other hand, urlunsplit is working fine in this case:

urlunsplit(('', '', '', 'ab=3', None))
'?ab=3'

I think yarl should not replace empty path in with_query implementation.

Support non-UTF8 query strings

Yep, I think that's the way to go. It should also be made clear that this encoding is an encoding of a page URL was extracted from. Path and fragment should still use UTF8, not this encoding, and hostname should still use IDNA.

It is a de-facto behavior of all browsers; as all other such gotchas, this is written down in https://url.spec.whatwg.org/ (see https://url.spec.whatwg.org/#query-encoding-example).

See also #19

YARL does not support link-local ipv6 addresses

Like this one:

http://[fe80::822a:a8ff:fe49:470c%enp4s1]:1234/keys

Support kwargs for with_query

url.with_query(a='1', b='2')

Inconsistency in .with_path()

The method should cleanup query and fragment parts -- as all other path modification methods do.

Cannot install v0.9.3+ because of UnicodeDecodeError

Today I wanted to upgrade home assistant to the newest version in my virtualenv on my RPi3. Unfortunately this was not possible as the build of yarl failed. Turns out that the root of this problem is somewhere between v0.9.2 and 0.9.3, see the following error

Installing collected packages: yarl
  Found existing installation: yarl 0.9.2
    Uninstalling yarl-0.9.2:
      Successfully uninstalled yarl-0.9.2
  Running setup.py install for yarl ... error
  Rolling back uninstall of yarl
Exception:
Traceback (most recent call last):
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/compat/__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 28: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/req/req_install.py", line 878, in install
    spinner=spinner,
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/utils/__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "/srv/homeassistant/homeassistant_venv/lib/python3.4/site-packages/pip/compat/__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 28: invalid start byte

Incorrect handling of '..' in url path

Long story short

If a url's path contains a path level that is .. it is not translated by removing the previous level from the path.

Expected behaviour

Example:
A url for http://mysite.com/lvl1/lvl2/../file.tar.gz should be translated to http://mysite.com/lvl1/file.tar.gz.

Actual behaviour

The url stays as http://mysite.com/lvl1/lvl2/../file.tar.gz

Steps to reproduce

from yarl import URL

url = URL('https://pypi.python.org/simple/aiohttp-swagger/../../packages/f1/db/0d22688d79b5de9fc325c5438a0b036bca9d711f80190aa2308f7a3942ad/aiohttp-swagger-1.0.0.tar.gz')
print(url.path)

will print:
'/simple/aiohttp-swagger/../../packages/f1/db/0d22688d79b5de9fc325c5438a0b036bca9d711f80190aa2308f7a3942ad/aiohttp-swagger-1.0.0.tar.gz'

In contrary with wget :

wget https://pypi.python.org/simple/aiohttp-swagger/../../packages/f1/db/0d22688d79b5de9fc325c5438a0b036bca9d711f80190aa2308f7a3942ad/aiohttp-swagger-1.0.0.tar.gz

will result in

--2017-06-21 14:52:30--  https://pypi.python.org/packages/f1/db/0d22688d79b5de9fc325c5438a0b036bca9d711f80190aa2308f7a3942ad/aiohttp-swagger-1.0.0.tar.gz
Resolving pypi.python.org (pypi.python.org)... 151.101.112.223, 2a04:4e42:1b::223
Connecting to pypi.python.org (pypi.python.org)|151.101.112.223|:443... connected.
HTTP request sent, awaiting response... 200 OK

Note that it translates the url path before sending the request.

Your environment

Ubuntu16.04 amd64 Python3.5 yarl0.10.3

Parse URL parameter

We found a trouble on a URL with params.
home-assistant/core#4629

"ValueError: Unallowed PCT %" when there's a "%" in the url

If there's a % (without it being a valid url encoded character) in the url path, yarl raises ValueError: Unallowed PCT % which causes a very unexpected 500 error.

Using yarl==0.13.0.

Traceback:

Error handling request
Traceback (most recent call last):
  File "/home/samuel/code/aiohttp-devtools/env/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 275, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
  File "aiohttp/_http_parser.pyx", line 295, in aiohttp._http_parser.HttpParser.feed_data
  File "aiohttp/_http_parser.pyx", line 400, in aiohttp._http_parser.cb_on_header_field
  File "aiohttp/_http_parser.pyx", line 329, in aiohttp._http_parser.HttpRequestParserC._on_status_complete
  File "aiohttp/_http_parser.pyx", line 611, in aiohttp._http_parser._parse_url
  File "/home/samuel/code/aiohttp-devtools/env/lib/python3.6/site-packages/yarl/__init__.py", line 214, in build
    path = _quote(path, safe='@:', protected='/')
  File "yarl/_quoting.pyx", line 45, in yarl._quoting._quote
  File "yarl/_quoting.pyx", line 135, in yarl._quoting._do_quote
ValueError: Unallowed PCT %

Code to reproduce:

from aiohttp import web

async def handle(request):
    return web.Response(text='<h1>testing</h1>', content_type='text/html')

app = web.Application()
app.router.add_get('/', handle)


if __name__ == '__main__':
    web.run_app(app, port=8001)

then curl

~ 0  151s ➤  curl -v "http://localhost:8001/%"
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8001 (#0)
> GET /% HTTP/1.1
> Host: localhost:8001
> User-Agent: curl/7.52.1
> Accept: */*
> 
* HTTP 1.0, assume close after body
< HTTP/1.0 500 Internal Server Error
< Content-Type: text/html; charset=utf-8
< Content-Length: 141
< Date: Fri, 20 Oct 2017 16:11:26 GMT
< Server: Python/3.6 aiohttp/2.3.0
< 
* Curl_http_done: called premature == 0
* Closing connection 0
<html><head><title>500 Internal Server Error</title></head><body><h1>500 Internal Server Error</h1>Server got itself in trouble</body></html>~ 0  32ms ➤ 

Implement join()

https://tools.ietf.org/html/rfc3986.html

Percent-encoded pluses in path variables become spaces

Expected result

(aiohttp query param behavior)

> curl localhost:8080/test/x+y%2Bz
x y+z

-- OR --

(Rails, Node/Express behavior)

> curl localhost:8080/test/x+y%2Bz
x+y+z

Actual behaviour

> curl localhost:8080/test/x+y%2Bz
x y z

aio-libs/aiohttp#1647

Implement origin()

The method should return an url with only scheme, host and port parts (no user/password and no path/query/fragment)

Allow integers in qs

I think it's very common case, e.g. url.update_query(page=1) and it looks harmless.

Implement resolve()

https://tools.ietf.org/html/rfc3986.html#section-5.2

Using `with_query` with an empty mapping causes `query_string` to raise a `TypeError`

Python Version: 3.6.1
OS: Windows 10

Using with_query with an empty mapping causes the returned URL instance to contain an invalid query_string property that raises a TypeError when it is being accessed.

Example code:

import yarl
a = yarl.URL("http://www.example.com")
print(a.with_query(a.query).query_string)  
# expected output: "" 
# actual output: "TypeError: Argument should be str"

The culprit seems to be that

Looking in _get_str_query it appears the easiest fix would be to replace:

# ... snip ...
if query is None:
  query = ''
# ... snip ...

with the following code:

# ... snip ...
if not query:
  query = ''
# ... snip ...

The relevant test case - test_update_query.py:test_with_query_empty_dict - should be updated like this:

def test_with_query_empty_dict():
    url = URL('http://example.com/?a=b')
    new_url = url.with_query({})
    assert new_url.query_string == ''
    assert str(new_url) == 'http://example.com/'

A pull request with the fix is on the way.

Incorrect parsing of query parameters with %3B (;) inside

Some test below:

u = yarl.URL('/?user_agent=Mozilla%2F5.0+%28iPhone%3B+CPU+iPhone+OS+10_1_1+like+Mac+OS+X%29+AppleWebKit%2F602.2.14+%28KHTML%2C+like+Gecko%29+Mobile%2F14B100')
assert len(u.query.items()) == 1

But I'm getting 2 params.

/-joining of urls can produce multiple slashes

>>> from yarl import URL
>>> URL('http://example.com/api/') / 'sub/path'
URL('http://example.com/api//sub/path')

Should that double slash be there?

Other corner cases:

>>> URL('http://example.com/api') / '/sub/path'
URL('http://example.com/api//sub/path')
>>> URL('http://example.com/api/') / '/sub/path'
URL('http://example.com/api///sub/path')

These examples work the way I expect:

>>> URL('http://example.com/') / 'sub/path'
URL('http://example.com/sub/path')
>>> URL('http://example.com') / 'sub/path'
URL('http://example.com/sub/path')

These do not:

>>> URL('http://example.com/') / '/sub/path'
URL('http://example.com//sub/path')
>>> URL('http://example.com') / '/sub/path'
URL('http://example.com//sub/path')

Bug or explicit design decision?

Give write access to Марк Коренберг

Hereby @socketpair is nominated to yarl committers

Encode semicolon in query string

aio-libs/aiohttp#1919 aio-libs/aiohttp#1901

_quoting.c

Please avoid shipping _quoting.c. This is an issue for packaging yarl.

Support python 3.3 by including pathlib in the requirements

Hi,

Installing Yarl in Python 3.3 fails because pathlib is only included in the stdlib since 3.4: https://docs.python.org/3.4/library/pathlib.html

I saw that Python 3.3 is not included in the supported version but since pathlib is available on pypi it seems that it would be possible to support Python 3.3, at least until it is EOL (September 2017).

params do not encode right

Test Code:

from yarl import URL
url = URL('https://github.com')
print(url.with_query({'a':'b','c':'%2'}))

Output:

https://github.com/?a=b&c=

%2 does not encode right

Running setup.py fails with LC_ALL=C

When running setup.py in an environment with LC_ALL=C (or anything besides utf-8) it stops with a UnicodeDecodeError when reading the README.

Changing read() to explicitly set the encoding to utf-8 seems to fix this.

pip install fails on python 3.6

I'm currently trying to build a python script which uses yarl. When using a python 3.5 virtualenv there is no problem, and pip install yarl installs version 0.7.0 without a problem. In a python 3.6 virtualenv using pip 9.0.1, an exception is thrown in yarl's setup.py:

Running setup.py (path:/tmp/pip-build-ms_rhff9/yarl/setup.py) egg_info for package yarl
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-ms_rhff9/yarl/setup.py", line 97, in <module>
        setup(**args)
      File "/usr/lib/python3.6/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/home/eh/crawler-3.6/lib/python3.6/site-packages/setuptools/dist.py", line 268, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "/home/eh/crawler-3.6/lib/python3.6/site-packages/setuptools/dist.py", line 313, in fetch_build_eggs
        replace_conflicting=True,
      File "/home/eh/crawler-3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 836, in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
      File "/home/eh/crawler-3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1081, in best_match
        return self.obtain(req, installer)
      File "/home/eh/crawler-3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1093, in obtain
        return installer(requirement)
      File "/home/eh/crawler-3.6/lib/python3.6/site-packages/setuptools/dist.py", line 380, in fetch_build_egg
        return cmd.easy_install(req)
      File "/home/eh/crawler-3.6/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 617, in easy_install
        not self.always_copy, self.local_index
      File "/home/eh/crawler-3.6/lib/python3.6/site-packages/setuptools/package_index.py", line 601, in fetch_distribution
        return dist.clone(location=self.download(dist.location, tmpdir))
      File "/home/eh/crawler-3.6/lib/python3.6/site-packages/setuptools/package_index.py", line 517, in download
        found = self._download_url(scheme.group(1), spec, tmpdir)
      File "/home/eh/crawler-3.6/lib/python3.6/site-packages/setuptools/package_index.py", line 758, in _download_url
        return self._attempt_download(url, filename)
      File "/home/eh/crawler-3.6/lib/python3.6/site-packages/setuptools/package_index.py", line 764, in _attempt_download
        headers = self._download_to(url, filename)
      File "/home/eh/crawler-3.6/lib/python3.6/site-packages/setuptools/package_index.py", line 680, in _download_to
        block = fp.read(bs)
      File "/usr/lib/python3.6/http/client.py", line 449, in read
        n = self.readinto(b)
      File "/usr/lib/python3.6/http/client.py", line 493, in readinto
        n = self.fp.readinto(b)
      File "/usr/lib/python3.6/socket.py", line 586, in readinto
        return self._sock.recv_into(b)
      File "/usr/lib/python3.6/ssl.py", line 1002, in recv_into
        return self.read(nbytes, buffer)
      File "/usr/lib/python3.6/ssl.py", line 865, in read
        return self._sslobj.read(len, buffer)
      File "/usr/lib/python3.6/ssl.py", line 625, in read
        v = self._sslobj.read(len, buffer)
    ConnectionResetError: [Errno 104] Connection reset by peer

Retrying it several times didn't change the behavior. After switching back to the 3.5 virtualenv I could uninstall and reinstall yarl there, and then in the 3.6 virtualenv the same error occurred. I'm not sure if it's a pip bug, but other packages seem to install fine.

Python 3.6.0b2 from Ubuntu PPA
Xubuntu 16.10
pip 9.0.1
yarl 0.6.0 and 0.7.0, haven't tested earlier versions

No way to replace path without replacing query

The following is unexpected behavior:

url1 = URL("http://www.example.com/foo?bar=10")
url2 = url1.with_path("/bar")

assert url1.query = url2.query  # Assertion fails

While I understand the design decision to treat the query and fragment components as part of the path, this has left us without an (efficient) way to preserve the query and fragment components when replacing just the path. The only way to do it is:

url2 = url1.with_path(new_path).with_query(url1.query_string).with_fragment(url1.fragment)

This is a total of 4 extra method calls, at least of which does a lot of unnecessary processing. I'm assuming that there's code out there that depends on this weird behavior, so I'd like to propose and, with permission, implement a new method, with_just_path, which replaces the path but leaves the fragment and query the same.

Question: how could I join to URL list path parts?

Assume I have an URL URL('http://localhost:5984/base') and some list of path segments like ['foo', '/bar', 'baz%']. What's the proper way to join them to url to haveURL(http://localhost:5984/base/foo/%2fbar/baz%25) as result?

So far I came to:

url = URL('http://localhost:5984/base')
segments = ['foo', '/bar', 'baz%']
for seg in sements:
  url /= seg.replace('/', '%2F')

But that doesn't looks like good solution. Or it does?

Ship Python 3.6 wheels.

Without python 3.6 wheels I cannot do the things as explained here: aio-libs/multidict#59

Please fix so I can pip install on python 3.6 and so I can move my code out of just 3.5 now because of the weels that are missing for 3.6.

Strict type checking in `with_query`

The Cythonised with_query will raise TypeError on subclasses of str like lxml's lxml.etree._ElementUnicodeResult, which used to work in parametrised aiohttp requests before the introduction of this library.

In [1]: import lxml.etree

In [2]: import yarl

In [3]: issubclass(lxml.etree._ElementUnicodeResult, str)
Out[3]: True

In [4]: url = yarl.URL('http://www.example.com/')

In [5]: url.with_query([('foo', 'bar')])
Out[5]: URL('http://www.example.com/?foo=bar')

In [6]: url.with_query([(lxml.etree._ElementUnicodeResult('foo'), 'bar')])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-dea4055e616e> in <module>()
----> 1 url.with_query([(lxml.etree._ElementUnicodeResult('foo'), 'bar')])

/yarl/__init__.py in with_query(self, *args, **kwargs)
    681             quoter = partial(quote, safe='', plus=True)
    682             query = '&'.join(quoter(k)+'='+quoter(v)
--> 683                              for k, v in query)
    684         else:
    685             raise TypeError("Invalid query type")

/yarl/__init__.py in <genexpr>(.0)
    681             quoter = partial(quote, safe='', plus=True)
    682             query = '&'.join(quoter(k)+'='+quoter(v)
--> 683                              for k, v in query)
    684         else:
    685             raise TypeError("Invalid query type")

/yarl/_quoting.pyx in yarl._quoting._quote (yarl/_quoting.c:1444)()

TypeError: Expected unicode, got lxml.etree._ElementUnicodeResult