aio-libs / multidict

The multidict implementation

multidict

Multidict is dict-like collection of key-value pairs where key might occur more than once in the container.

Introduction

HTTP Headers and URL query string require specific data structure: multidict. It behaves mostly like a regular dict but it may have several values for the same key and preserves insertion ordering.

The key is str (or istr for case-insensitive dictionaries).

multidict has four multidict classes: MultiDict, MultiDictProxy, CIMultiDict and CIMultiDictProxy.

Immutable proxies (MultiDictProxy and CIMultiDictProxy) provide a dynamic view for the proxied multidict, the view reflects underlying collection changes. They implement the collections.abc.Mapping interface.

Regular mutable (MultiDict and CIMultiDict) classes implement collections.abc.MutableMapping and allows them to change their own content.

Case insensitive (CIMultiDict and CIMultiDictProxy) assume the keys are case insensitive, e.g.:

>>> dct = CIMultiDict(key='val')
>>> 'Key' in dct
True
>>> dct['Key']
'val'

Keys should be str or istr instances.

The library has optional C Extensions for speed.

License

Apache 2

Library Installation

$ pip install multidict

The library is Python 3 only!

PyPI contains binary wheels for Linux, Windows and MacOS. If you want to install multidict on another operating system (or Alpine Linux inside a Docker) the tarball will be used to compile the library from source. It requires a C compiler and Python headers to be installed.

To skip the compilation, please use the MULTIDICT_NO_EXTENSIONS environment variable, e.g.:

$ MULTIDICT_NO_EXTENSIONS=1 pip install multidict

Please note, the pure Python (uncompiled) version is about 20-50 times slower depending on the usage scenario!!!

Changelog

See RTD page.

multidict's Issues

no tar-ball or zip on pypi

Hi

would be great, if a tar-ball could be made available on pypi (useful for, e.g. linux distributions for creating a package). https://pypi.python.org/pypi/multidict currently only has whl packages.

Thanks

Arun

multidict._multidict.so is not distributable

with gdb

Program received signal SIGILL, Illegal instruction.
0x00007ffff00b34aa in PyInit__multidict () from /opt/spot4/bin/lib/python3.6/multidict._multidict.so

I moved cx_freezed code to fresh installed debian. ldd has no missed libs. But there is this trouble.

After deleting it All other libs loaded successfully. And previous version of aiohttp works well after moving.

Cython-Python pickling

Pickled pure Python MultiDicts should be unpickled as Cythonized if unpicker is executed on machine with Cython support.
And vise versa.

Consider moving package to `src` subfolder

As per https://hynek.me/articles/testing-packaging/ when testing --editable (develop) install you are not testing the package installed same way as it will be installed in by users.

@asvetlov what do you think about such approach? Sounds reasonable to me.

Implementation thoughts

Hi there! Great project! I was reviewing the implementation (love the cython bindings), but it occurred to me: have you considered using a linked-list + dict approach for the internals of the MultiDict? I may be wrong, but as it stands now, it seems like the access time may be O(n). It's good because it seems to preserve order, which I appreciate (some servers are header-order sensitive and query param order-sensitive).

I understand O(n) is probably ok most of the time for some protocols (HTTP probably has <10 headers on most requests), but I figured I'd ask anyway. Keep up the great work!

Multidicts does not preserve order on update

When updating existing key in multidict, it is moved to the end (original order is not preserved):

>>> d = CIMultiDict()
>>> d['1'] = '2'
>>> d['3'] = '4'
>>> d
<CIMultiDict('1': '2', '3': '4')>
>>> d['1'] = '10'
>>> d
<CIMultiDict('3': '4', '1': '10')>

In above example, guarantee of preserving insertion order should leave existing key '1' on its original first position.

3.3.0 release misses *.c files

warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files found matching 'multidict/_multidict.html'
warning: no previously-included files found matching 'multidict/_multidict.*.so'
warning: no previously-included files found matching 'multidict/_multidict.pyd'
warning: no previously-included files found matching 'multidict/_multidict.*.pyd'
no previously-included directories found matching 'docs/_build'
clang: error: no such file or directory: 'multidict/_multidict.c'
clang: error: no input files
************************************************************
Cannot compile C accelerator module, use pure python version
************************************************************

Steps to reproduce:

• virtualenv without Cython
• install multidict from sdist package

RTD Changelog has broken links

The documentation's changelog reads currently:

• Accept multiple keys in MultiDict.update() and CIMultiDict.update() (PR #199)

The link leads to https://github.com/aio-libs/multidict/pulls/199, which is broken; it should point to #199 (which GitHub is unhelpfully contracting here); that is s/pulls/pull. (If I knew / find where this is done, I would send a PR, but the source seems to simply say :pr:199:)

`MultiDict.copy()` doesn't work with `istr` keys

Here's a small test case:

>>> import multidict
>>> m=multidict.CIMultiDict({multidict.istr('Foo'): 'bar'})
>>> m.copy()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "multidict/_multidict.pyx", line 325, in multidict._multidict.MultiDict.copy (multidict/_multidict.c:6830)
  File "multidict/_multidict.pyx", line 251, in multidict._multidict.MultiDict.__init__ (multidict/_multidict.c:5495)
  File "multidict/_multidict.pyx", line 289, in multidict._multidict.MultiDict._extend (multidict/_multidict.c:6115)
TypeError: Expected str, got istr

MultiDict._extend expects every key to be a str. aiohttp.hdrscreates istr objects, so request.headers.copy() doesn't work.

ItemsView behavior

Good day.

I'm currently trying to rewrite multidict on top of OrderedDict/dict (4fun) and faced a behavior that is not clear for me.

So _ItemsView returns a items in order they appear and that is correct. Nonetheless we are not able to retrieve values by keys in this example:

d2 = MultiDict([('1', 'foo'), ('2', 'bar'), ('1', 'baz')])
for k in d2:
    print('%s => %s' % (k, d2[k]))
    
1 => foo
2 => bar
1 => foo

I know that it is a right behavior for MultiDict but is it a most expected one?

public class _Base

To effective detect a group of classes

if isinstance(a, BaseMultidict):
    # dosomething1
elif isinstances (a, Mapping):
    # dosomething2

The simplest implementation is less effective

BaseMultidict = (CIMultiDict, MultiDict, MultiDictProxy)

TODO: Try enabling extensions under pypy3 v6.0 once it's available in Travis CI and/or pyenv

It looks like they've improved compatibility with CPython in v6.0

Ref: https://twitter.com/pypyproject/status/989392997300260865
Ref: travis-ci/travis-ci#9542

Pre-compiled shared libraries

I'm attempting to build a distribution package from the source on PyPi. However, the source contains the compiled shared library _istr.cpython-34m.so.

This causes problems if the arch and Python version don't match up with the library. Manually removing the file from the source solves the problem but it doesn't seem like this should exist in the source package?

I also noticed the source on PyPi does not match up with the source on Github. Github does not contain the _multidict.c file. Is this also a build artifact that was compiled from the pyx file?

The url I'm pulling source from is:
https://files.pythonhosted.org/packages/source/m/multidict/multidict-%{version}.tar.gz

Update from multidict==1.2.2 to latest cause Flask app behind a nginx proxy to 404

I am have a Flask app hosted in a gunicorn server behind a nginx reverse proxy. The app is not on the / URL but under a

Updating from 1.2.2 to latest cause the Flask app to return 404 for all routes. Downgrading back to 1.2.2 fixed the issues.

Not sure if this is a Flask issue or multidict.

[notification] Sviatoslav Sydorenko has given admin rights

@webknjaz has given admin rights for the project.
He worked hard on travis and deployment setup, new access privileges will help him to do his duty faster and better.
He is a committer already BTW

latest multidict does not have an 32 and 64 bit wheel for Python 3.6 for windows and linux.

Because the latest release does not have an python 3.6 wheel multidict, now breaks installing all of my discord bot's dependencies. The bad part about that is the bot is not supposed to be offline very ling but yet multidict and the latest yarl breaks it for python 3.6 Please fix these issues.

For more information see aio-libs/aiohttp#1727

The entire output of the console (including the failure) is this:

Collecting discord.py (from -r requirements.txt (line 1))
  Using cached discord.py-0.16.7.tar.gz
Collecting aiohttp (from -r requirements.txt (line 2))
  Using cached aiohttp-1.3.5.tar.gz
Collecting PyNacl (from -r requirements.txt (line 3))
  Using cached PyNaCl-1.1.1-cp36-cp36m-win32.whl
Collecting pycares (from -r requirements.txt (line 4))
  Using cached pycares-2.1.1.tar.gz
Collecting cchardet (from -r requirements.txt (line 5))
  Using cached cchardet-1.1.3-cp36-cp36m-win32.whl
Collecting cffi (from -r requirements.txt (line 6))
  Using cached cffi-1.9.1-cp36-cp36m-win32.whl
Collecting multidict (from -r requirements.txt (line 7))
  Using cached multidict-2.1.4.tar.gz
Collecting yarl (from -r requirements.txt (line 8))
  Using cached yarl-0.10.0.tar.gz
Collecting discord.webhooks (from -r requirements.txt (line 9))
  Using cached discord.webhooks-0.0.2.tar.gz
Collecting websockets<4.0,>=3.1 (from discord.py->-r requirements.txt (line 1))
  Using cached websockets-3.2-py33.py34.py35-none-any.whl
Collecting chardet (from aiohttp->-r requirements.txt (line 2))
  Using cached chardet-2.3.0-py2.py3-none-any.whl
Collecting async_timeout>=1.1.0 (from aiohttp->-r requirements.txt (line 2))
  Using cached async_timeout-1.2.0-py3-none-any.whl
Collecting six (from PyNacl->-r requirements.txt (line 3))
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting pycparser (from cffi->-r requirements.txt (line 6))
  Using cached pycparser-2.17.tar.gz
Installing collected packages: chardet, multidict, async-timeout, yarl, aiohttp,
 websockets, discord.py, pycparser, cffi, six, PyNacl, pycares, cchardet, discor
d.webhooks
  Running setup.py install for multidict ... error
    Complete output from command E:\Python360\python.exe -u -c "import setuptool
s, tokenize;__file__='E:\\Users\\Elsword\\AppData\\Local\\Temp\\pip-build-j53pka
8u\\multidict\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read
().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" instal
l --record E:\Users\Elsword\AppData\Local\Temp\pip-61352pwp-record\install-recor
d.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build\lib.win32-3.6
    creating build\lib.win32-3.6\multidict
    copying multidict\_multidict_py.py -> build\lib.win32-3.6\multidict
    copying multidict\__init__.py -> build\lib.win32-3.6\multidict
    running egg_info
    writing multidict.egg-info\PKG-INFO
    writing dependency_links to multidict.egg-info\dependency_links.txt
    writing top-level names to multidict.egg-info\top_level.txt
    reading manifest file 'multidict.egg-info\SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no previously-included files matching '*.pyc' found anywhere in dis
tribution
    warning: no previously-included files found matching 'multidict\_multidict.h
tml'
    warning: no previously-included files found matching 'multidict\_multidict.*
.so'
    warning: no previously-included files found matching 'multidict\_multidict.p
yd'
    warning: no previously-included files found matching 'multidict\_multidict.*
.pyd'
    no previously-included directories found matching 'docs\_build'
    writing manifest file 'multidict.egg-info\SOURCES.txt'
    copying multidict\__init__.pyi -> build\lib.win32-3.6\multidict
    copying multidict\_multidict.c -> build\lib.win32-3.6\multidict
    copying multidict\_multidict.pyx -> build\lib.win32-3.6\multidict
    running build_ext
    building 'multidict._multidict' extension
    creating build\temp.win32-3.6
    creating build\temp.win32-3.6\Release
    creating build\temp.win32-3.6\Release\multidict
    E:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\cl.exe /c /nologo
 /Ox /W3 /GL /DNDEBUG /MD -IE:\Python360\include -IE:\Python360\include "-IE:\Pr
ogram Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IE:\Program Files (
x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IE:\Program Files (x86)\W
indows Kits\10\include\10.0.10586.0\ucrt" "-IE:\Program Files (x86)\Windows Kits
\NETFXSDK\4.6.1\include\um" "-IE:\Program Files (x86)\Windows Kits\10\include\10
.0.10586.0\shared" "-IE:\Program Files (x86)\Windows Kits\10\include\10.0.10586.
0\um" "-IE:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" "-IE
:\Program Files (x86)\Microsoft Visual Studio\VC98\atl\include" "-IE:\Program Fi
les (x86)\Microsoft Visual Studio\VC98\mfc\include" "-IE:\Program Files (x86)\Mi
crosoft Visual Studio\VC98\include" /Tcmultidict/_multidict.c /Fobuild\temp.win3
2-3.6\Release\multidict/_multidict.obj
    _multidict.c
    multidict/_multidict.c(11): fatal error C1083: Cannot open include file: 'Py
thon.h': No such file or directory
    ************************************************************
    Cannot compile C accelerator module, use pure python version
    ************************************************************
    running install
    running build
    running build_py
    creating build\lib
    creating build\lib\multidict
    copying multidict\_multidict_py.py -> build\lib\multidict
    copying multidict\__init__.py -> build\lib\multidict
    running egg_info
    writing multidict.egg-info\PKG-INFO
    writing dependency_links to multidict.egg-info\dependency_links.txt
    writing top-level names to multidict.egg-info\top_level.txt
    reading manifest file 'multidict.egg-info\SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no previously-included files matching '*.pyc' found anywhere in dis
tribution
    warning: no previously-included files found matching 'multidict\_multidict.h
tml'
    warning: no previously-included files found matching 'multidict\_multidict.*
.so'
    warning: no previously-included files found matching 'multidict\_multidict.p
yd'
    warning: no previously-included files found matching 'multidict\_multidict.*
.pyd'
    no previously-included directories found matching 'docs\_build'
    writing manifest file 'multidict.egg-info\SOURCES.txt'
    copying multidict\__init__.pyi -> build\lib\multidict
    copying multidict\_multidict.c -> build\lib\multidict
    copying multidict\_multidict.pyx -> build\lib\multidict
    running install_lib
    running install_egg_info
    removing 'E:\Python360\Lib\site-packages\multidict-2.1.4-py3.6.egg-info' (an
d everything under it)
    Copying multidict.egg-info to E:\Python360\Lib\site-packages\multidict-2.1.4
-py3.6.egg-info
    running install_scripts
    Traceback (most recent call last):
      File "E:\Python360\lib\distutils\_msvccompiler.py", line 382, in compile
        self.spawn(args)
      File "E:\Python360\lib\distutils\_msvccompiler.py", line 501, in spawn
        return super().spawn(cmd)
      File "E:\Python360\lib\distutils\ccompiler.py", line 909, in spawn
        spawn(cmd, dry_run=self.dry_run)
      File "E:\Python360\lib\distutils\spawn.py", line 38, in spawn
        _spawn_nt(cmd, search_path, dry_run=dry_run)
      File "E:\Python360\lib\distutils\spawn.py", line 81, in _spawn_nt
        "command %r failed with exit status %d" % (cmd, rc))
    distutils.errors.DistutilsExecError: command 'E:\\Program Files (x86)\\Micro
soft Visual Studio 14.0\\VC\\BIN\\cl.exe' failed with exit status 2

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "E:\Users\Elsword\AppData\Local\Temp\pip-build-j53pka8u\multidict\set
up.py", line 46, in build_extension
        build_ext.build_extension(self, ext)
      File "E:\Python360\lib\distutils\command\build_ext.py", line 533, in build
_extension
        depends=ext.depends)
      File "E:\Python360\lib\distutils\_msvccompiler.py", line 384, in compile
        raise CompileError(msg)
    distutils.errors.CompileError: command 'E:\\Program Files (x86)\\Microsoft V
isual Studio 14.0\\VC\\BIN\\cl.exe' failed with exit status 2

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "E:\Users\Elsword\AppData\Local\Temp\pip-build-j53pka8u\multidict\set
up.py", line 103, in <module>
        setup(**args)
      File "E:\Python360\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "E:\Python360\lib\distutils\dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "E:\Python360\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "E:\Python360\lib\site-packages\setuptools\command\install.py", line
61, in run
        return orig.install.run(self)
      File "E:\Python360\lib\distutils\command\install.py", line 545, in run
        self.run_command('build')
      File "E:\Python360\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "E:\Python360\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "E:\Python360\lib\distutils\command\build.py", line 135, in run
        self.run_command(cmd_name)
      File "E:\Python360\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "E:\Python360\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "E:\Users\Elsword\AppData\Local\Temp\pip-build-j53pka8u\multidict\set
up.py", line 40, in run
        build_ext.run(self)
      File "E:\Python360\lib\distutils\command\build_ext.py", line 339, in run
        self.build_extensions()
      File "E:\Python360\lib\distutils\command\build_ext.py", line 448, in build
_extensions
        self._build_extensions_serial()
      File "E:\Python360\lib\distutils\command\build_ext.py", line 473, in _buil
d_extensions_serial
        self.build_extension(ext)
      File "E:\Users\Elsword\AppData\Local\Temp\pip-build-j53pka8u\multidict\set
up.py", line 49, in build_extension
        raise BuildFailed()
    __main__.BuildFailed

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "E:\Users\Elsword\AppData\Local\Temp\pip-build-j53pka8u\multidict\set
up.py", line 110, in <module>
        setup(**args)
      File "E:\Python360\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "E:\Python360\lib\distutils\dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "E:\Python360\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "E:\Python360\lib\site-packages\setuptools\command\install.py", line
61, in run
        return orig.install.run(self)
      File "E:\Python360\lib\distutils\command\install.py", line 557, in run
        self.run_command(cmd_name)
      File "E:\Python360\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "E:\Python360\lib\distutils\dist.py", line 974, in run_command
        cmd_obj.run()
      File "E:\Python360\lib\site-packages\setuptools\command\install_scripts.py
", line 35, in run
        bw_cmd = self.get_finalized_command("bdist_wininst")
      File "E:\Python360\lib\distutils\cmd.py", line 298, in get_finalized_comma
nd
        cmd_obj = self.distribution.get_command_obj(command, create)
      File "E:\Python360\lib\distutils\dist.py", line 846, in get_command_obj
        klass = self.get_command_class(command)
      File "E:\Python360\lib\site-packages\setuptools\dist.py", line 489, in get
_command_class
        self.cmdclass[command] = cmdclass = ep.load()
      File "E:\Python360\lib\site-packages\pkg_resources\__init__.py", line 2291
, in load
        return self.resolve()
      File "E:\Python360\lib\site-packages\pkg_resources\__init__.py", line 2297
, in resolve
        module = __import__(self.module_name, fromlist=['__name__'], level=0)
      File "E:\Python360\lib\site-packages\setuptools\command\bdist_wininst.py",
 line 1, in <module>
        import distutils.command.bdist_wininst as orig
    ModuleNotFoundError: No module named 'distutils.command.bdist_wininst'

    ----------------------------------------
Command "E:\Python360\python.exe -u -c "import setuptools, tokenize;__file__='E:
\\Users\\Elsword\\AppData\\Local\\Temp\\pip-build-j53pka8u\\multidict\\setup.py'
;f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n')
;f.close();exec(compile(code, __file__, 'exec'))" install --record E:\Users\Elsw
ord\AppData\Local\Temp\pip-61352pwp-record\install-record.txt --single-version-e
xternally-managed --compile" failed with error code 1 in E:\Users\Elsword\AppDat
a\Local\Temp\pip-build-j53pka8u\multidict\
Press any key to continue . . .

btw how did you guys get the rst to show batdges properly and stuff? I could never get the rst to show properly in pypi:

https://pypi.python.org/pypi/discord.webhooks<-- one of my packages I could never get the rst to act right on.

clarification on versioning

Hi, I'm a distro packager and curious about the versioning with 'a'.
Are v3.1.4a0 … v3.3.0a0 meant to be alpha releases that should not be distributed?
Or are those now "regular" version schemes and distros should pick up those versions?

thanks for clarifying and sorry for the noise

Decorate multidict clases by @cython.final

Classes are not intended for inheritance.
For pure python implementation we could make a metaclass for the check easy.

surprised by: isinstance(foo, dict) == False

I was a little surprised by:

>>> import multidict
>>> foo = multidict.CIMultiDict()
>>> print(isinstance(foo, dict))
False

TODO: Add multiple issue templates to evaluate how it works

GitHub has publicly released feature of having several different templates for different issue types a few days ago. It looks pretty neat and I'm looking forward to integrating them into other projects.

Ref: maintainers/early-access-feedback#139

different tag name for 2.1.4

Usually you use v prefix, but not in 2.1.4. Mistake or plan for further releases?

CIMultiDict copy not working as expected

There is an issue with CIMultiDict copy method

import multidict
multidict.__version__  # '3.1.1'
a = multidict.CIMultiDict()
a['foo'] = 6
b = a.copy()
b['foo'] = 7
b # <CIMultiDict('foo': 7)>
a # <CIMultiDict('foo': 7)>

And using the copy function

import copy
c = copy.copy(a)

I get

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../venv/lib/python3.6/copy.py", line 106, in copy
    return _reconstruct(x, None, *rv)
  File ".../venv/lib/python3.6/copy.py", line 274, in _reconstruct
    y = func(*args)
  File "multidict/_multidict.pyx", line 531, in multidict._multidict.CIMultiDict.__init__ (multidict/_multidict.c:9519)
  File "multidict/_multidict.pyx", line 327, in multidict._multidict.MultiDict._extend (multidict/_multidict.c:6702)
TypeError: CIMultiDict takes either dict or list of (key, value) tuples

Address boundary error

Python 3.5.2, multidict-2.0.0 from PyPI, following code fails with SIGSEGV:

from multidict import MultiDict
set() - MultiDict().keys()

If i understood correctly, error happens in PyAnySet_Check call, since set substraction is basically difference method call with type check:

if (!PyAnySet_Check(so) || !PyAnySet_Check(other)) {
    Py_INCREF(Py_NotImplemented);
    return Py_NotImplemented;
}
return set_difference(so, other);

It also can be reproduced with dict and list objects, so i think problem may be related to type checking.

Python MultiDict constructor discards generator content

When using the pure Python implementation, if a MultiDict is constructed with a generator argument:

headers = CIMultiDict(
    (
        k.decode('utf-8', 'surrogateescape'),
        v.decode('utf-8', 'surrogateescape'),
    )
    for k, v in event.headers
)

then the resulting MultiDict will be empty, instead of containing the key/value pairs as expected. This is because the generator is iterated over twice, and the first iteration discards all of the pairs.

multidicts are not picklable

Appveyor on master is broken

Missing codecov
@webknjaz please take a look

Large lag between Windows and Linux/OS X PyPI uploads

Hi!

Similar to the issues experienced with aiohttp (eg aio-libs/aiohttp#2347), there is a massive delay (currently 3 hours, but the other releases are still outstanding) between the Windows wheel for a new multidict release being uploaded to PyPI, and when the sdist/other wheels (manylinux/OS X) are available.

This is problematic, since tooling such as pyup.io see the new release in the PyPI feed, and so open a PR to update the dependency. However at that point only the Windows wheel exists, causing the Travis run to fail (eg mozilla/treeherder#2899).

For possible solutions, see:
aio-libs/aiohttp#2347 (comment)

Many thanks :-)

python 2 support?

Hi, I'm not sure if multidict has python2 support, at least there are some compatibility problems like FileNotFoundError is not in py2 and yield from is not valid construction in py2.

`pop()` is inconsistent

It now removes all keys but returns the last value
Should remove only first key.

pop() should be an alias for new method .popone() to mimic get/getone.

Issue with multidict installation with version 3.3.0

Issue with multidict installation with version 3.3.0

Even old versions have same issue.
Share what is missing?

multidict-3.3.0$ python --version
Python 3.6.1

multidict-3.3.0$ sudo python setup.py install
running install
running bdist_egg
running egg_info
creating multidict.egg-info
writing multidict.egg-info/PKG-INFO
writing top-level names to multidict.egg-info/top_level.txt
writing dependency_links to multidict.egg-info/dependency_links.txt
writing manifest file 'multidict.egg-info/SOURCES.txt'
reading manifest file 'multidict.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '.pyc' found anywhere in distribution
warning: no previously-included files found matching 'multidict/_multidict.html'
warning: no previously-included files found matching 'multidict/_multidict..so'
warning: no previously-included files found matching 'multidict/_multidict.pyd'
warning: no previously-included files found matching 'multidict/_multidict.*.pyd'
no previously-included directories found matching 'docs/_build'
writing manifest file 'multidict.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/multidict
copying multidict/_multidict_py.py -> build/lib.linux-x86_64-2.7/multidict
copying multidict/init.py -> build/lib.linux-x86_64-2.7/multidict
copying multidict/_abc.py -> build/lib.linux-x86_64-2.7/multidict
copying multidict/_compat.py -> build/lib.linux-x86_64-2.7/multidict
copying multidict/init.pyi -> build/lib.linux-x86_64-2.7/multidict
copying multidict/_istr.c -> build/lib.linux-x86_64-2.7/multidict
error: can't copy 'multidict/_multidict.c': doesn't exist or not a regular file

Cython and Python versions of CIMultiDict treat key case differently

First noticed this in 3.3.2 (probably existed before that), still exists in 4.0.0:

As one would expect (IMO) the Cython version uses the title-ized key for matching and preserves the original key for output/serialization. The Python version throws away the original key completely and uses the title-ized key for both purposes, breaking output. Again, that's my opinion, but in any case, the behavior should be the same.

In [1]: from multidict._multidict import CIMultiDict  # Cython

In [2]: md = CIMultiDict()
   ...: md['some_key'] = 1
   ...: list(md.items())
   ...:
Out[2]: [('some_key', 1)]  # outputs original key

In [3]: from multidict._multidict_py import CIMultiDict  # Python

In [4]: md = CIMultiDict()
   ...: md['some_key'] = 1
   ...: list(md.items())
   ...:
Out[4]: [('Some_Key', 1)]  # outputs title-ized key

Need to rewrite cython extensions in pure C

The reason is multidict.add(key, val) is ten times slower than dict[key] = val.

This is because multidict stores data internally as a list of cythonized _Item objects.
But creation of python (ever cythonized) object is too expensive for our use case.

The solution is using C structs for internal data but Cython has no support for visiting values stored in these structures: tp_visit and tp_clear slots.

Thus for sake of speed we need pure C implementation.

TODO: evaluate additional CIs

• Circle CI (in progress)
• Shippable (they have recently announced macOS and Windows support) / https://twitter.com/webKnjaZ/status/991415221834141696
• GitLab CI
• Wercker
• Semaphore / https://github.com/marketplace/semaphore/plan/MDIyOk1hcmtldHBsYWNlTGlzdGluZ1BsYW41MjA=#pricing-and-setup
• Codefresh / https://github.com/marketplace/codefresh
• Buddy / https://github.com/marketplace/buddy/plan/MDIyOk1hcmtldHBsYWNlTGlzdGluZ1BsYW45MDM=#pricing-and-setup / https://buddy.works/knowledge/automation/how-build-and-deploy-python

Misc

• https://github.com/menegazzo/travispy/blob/3bdbf9b0fa3b89b8efbdccef50e077ef8f27fad8/docs/getting_started.rst
• https://www.npmjs.com/package/github-auto-deploy
• github/github-services@ad96b97
• https://developer.github.com/v3/activity/events/types/#deploymentevent
• https://developer.github.com/v3/repos/deployments/#create-a-deployment
• https://buildkite.com/blog/github-deployments
• https://github.com/marketplace/category/deployment
• https://developer.github.com/v3/guides/automating-deployments-to-integrators/
• https://developer.github.com/changes/2014-01-09-preview-the-new-deployments-api/
• https://github.com/issc29/jenkins-deployment-api
• https://developer.github.com/v3/guides/delivering-deployments/

setup.py test failed: unrecognized arguments: --cov=multidict

DEBUG: + /usr/bin/python3.5 setup.py test
DEBUG: running pytest
DEBUG: running egg_info
DEBUG: writing multidict.egg-info/PKG-INFO
DEBUG: writing top-level names to multidict.egg-info/top_level.txt
DEBUG: writing dependency_links to multidict.egg-info/dependency_links.txt
DEBUG: reading manifest file 'multidict.egg-info/SOURCES.txt'
DEBUG: reading manifest template 'MANIFEST.in'
DEBUG: warning: no files found matching 'CHANGES.rst'
DEBUG: warning: no previously-included files matching '*.pyc' found anywhere in distribution
DEBUG: warning: no previously-included files found matching 'multidict/_multidict.html'
DEBUG: warning: no previously-included files found matching 'multidict/_multidict.pyd'
DEBUG: warning: no previously-included files found matching 'multidict/_multidict.*.pyd'
DEBUG: no previously-included directories found matching 'docs/_build'
DEBUG: writing manifest file 'multidict.egg-info/SOURCES.txt'
DEBUG: running build_ext
DEBUG: usage: setup.py [options] [file_or_dir] [file_or_dir] [...]
DEBUG: setup.py: error: unrecognized arguments: --cov=multidict
DEBUG:   inifile: /builddir/build/BUILD/multidict-4.2.0/pytest.ini
DEBUG:   rootdir: /builddir/build/BUILD/multidict-4.2.0

Support PEP-561

PEP-561 enables package maintainers to distribute their typing stub files and have mypy (as of version 0.590) use them from the installed package. Since multidict already includes stub files, the only thing that needs to be done is to distribute a file named py.typed in the package directory.

Replace istr()

Deriving from str is too expensive.
We need fast and simple way to figure out is key already titled (canonized) or not.

The alternative could be multidict.canonize(s: str) -> str function.
It converts parameter to s.title(), intern it by sys.intern(...) call and store the result into global dictionary.
Thus we have a very fast check for canonized strings.

The drawback or proposal is the canonized strings are never deleted (I think it's fine for aiohttp.hdrs usage). All canonize() calls should be done on module import stage, that's why I propose a new function instead of adding new tricks to istr.

istr will be supported for years anyway but canonize is the only way to keep multidicts fast.
Right now istr calls is 10-25% slower on MultiDict -- its not acceptable (but it's still much faster on CIMultiDict lookups than plain str).

Guard iterations

Iteration over keys, items and values should raise RuntimeError if dict is changed.

Force keys to str instances

Right now a key type is dark corner: it could be str, istr or something other.
For sake of further optimizations it should be str always.

Passing istr as key to any miltidict API is still allowed and encouraged: it saves execution time by getting rid of s.title() conversion.

But istr type is ambiguous by it nature: we can define equality for str and istr by comparing their canonical representations but ordering is cumbersome.
Ordering by canonical reprs leads to very tricky errors, the same for original reprs.
Also CPython is very optimized for str but not classes derived from it.

I believe the change doesn't touch our users but aiohttp might get speedup on implementing the issue.

ImportWarning on init

I'm using multidict 3.1.0 and python 3.6.1.

I get the following warning when my application starts:

/usr/lib/python3.6/importlib/_bootstrap.py:205: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

Running the same application with PYTHONWARNINGS=error, I can track down the issue to multidict:

...
  File "usr/lib/python3.6/site-packages/aiohttp/__init__.py", line 5, in <module>
  File "usr/lib/python3.6/site-packages/aiohttp/hdrs.py", line 2, in <module>
  File "usr/lib/python3.6/site-packages/multidict/__init__.py", line 24, in <module>
  File "multidict/_multidict.pyx", line 9, in init multidict._multidict (multidict/_multidict.c:16318)
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

Seems similar to #79. Apparently, #81 didn't fix the issue completely.

Upload linux wheels doesn't work

See
https://pypi.python.org/pypi?:action=display&name=multidict&version=3.3.0a1#downloads

Maybe handling travis deploy instruction has changed, don't know.
I'm inclined to use twine explicitly for uploading both tarball and manylinux wheels.

Moreover ./tools/run_docker.sh "multidict" generates extra _istr.xx.so for python 3.6 even it is executed for building 3.4 wheel.

I'll try to investigate what's going on.
@webknjaz if you have a time to take a look --- it would be awesome.
You are travis jedi, I suspect the deploy is broken after your changes in our build process.

Decorate _Pair by @cython.freelist

It could amortize a cost for adding new items into multidicts and as result improve performance.
Of course #97 would be much better but it implementation is much harder.

Implement dict's version

It could speedup headers caching

`istr` causes memory leak

Hi,

Long story short

# Warning: This script may cost about 100MB of memory.
import gc
import time

from multidict import istr


for i in range(1000000):
    # the memory of this temporary instance should be released after this loop,
    # but the result is not.
    istr('Content-Type')

gc.collect()
print('Done.')
time.sleep(300)

Environment

• Python 3.5.3
• multidict 3.1.0
• Mac OS 10.12.5 & Centos Linux 6.6

Thanks!

Multidict preserves order- document it somewhere

As it is implemented now, Multidict preserves order.
Because of the way it is used in aiohttp (parsing post items and HTTP headers), this seems a very desirable property, and not merely an implementation detail.

I think it should be documented both here and in aiohttp's docs that Multidict preserves order, so that one doesn't need to read the implementation to be sure.

Importing multidict raises ImportWarning on 3.6

$ python -We -c "import multidict as x; print(x.__version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages/multidict/__init__.py", line 24, in <module>
    from ._multidict import (MultiDictProxy,
  File "multidict/_multidict.pyx", line 1, in init multidict._multidict (multidict/_multidict.c:14817)
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

CIMultiDict.pop is case sensitive.

Repro Steps

>>> from multidict import CIMultiDict
>>> d = CIMultiDict(key='value')
>>> d['key']
'value'
>>> d.pop('key')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "multidict/_multidict.pyx", line 385, in multidict._multidict.MultiDict.pop (multidict/_multidict.c:7811)
KeyError: 'key'

Expected Behaviour

Calling pop should return 'value' and remove the item from the dict.

Actual Behaviour

KeyError

Environment

Python: Python 3.5.1 (default, Dec 14 2015, 09:21:15) [GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin

System: Mac OS X El Capitan 10.11.5

multidict version: 1.0.3, installed from PyPI

Refactor test suite

Now dependency tree for test_multidict.py is too complicated ever for me.
We need to slip it into several files, rewrite tests in pytest style and reduce test class dependencies.

Like:

READONLY_DICTS = [MultiDict, CIMultiDict, MultiDictProxy, CIMultiDictProxy]  # plus pure python classes

@pytest.parametrize(READONLY_DICTS)
def test_something(dct):
    pass

Implement `.popall(key[, default])`

Should remove all key occurrences and return a list with all removed values.

multidict 3.2.0 won't install properly on docker

Steps to reproduce

1. cat Dockerfile

FROM ubuntu:14.04

RUN apt-get update && apt-get -y install python3-pip
RUN bash -c "pip3 install multidict==3.2.0"

2. docker build -t multidict-test .

Result:

Downloading/unpacking multidict==3.2.0
  Downloading multidict-3.2.0.tar.gz
  Running setup.py (path:/tmp/pip_build_root/multidict/setup.py) egg_info for package multidict

    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files found matching 'multidict/_multidict.html'
    warning: no previously-included files found matching 'multidict/_multidict.*.so'
    warning: no previously-included files found matching 'multidict/_multidict.pyd'
    warning: no previously-included files found matching 'multidict/_multidict.*.pyd'
    no previously-included directories found matching 'docs/_build'
Installing collected packages: multidict
  Running setup.py install for multidict

    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files found matching 'multidict/_multidict.html'
    warning: no previously-included files found matching 'multidict/_multidict.*.so'
    warning: no previously-included files found matching 'multidict/_multidict.pyd'
    warning: no previously-included files found matching 'multidict/_multidict.*.pyd'
    no previously-included directories found matching 'docs/_build'
    error: can't copy 'multidict/_multidict.c': doesn't exist or not a regular file
    Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/multidict/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-rwpv1_j1-record/install-record.txt --single-version-externally-managed --compile:
    running install

running build

running build_py

creating build

creating build/lib.linux-x86_64-3.4

creating build/lib.linux-x86_64-3.4/multidict

copying multidict/__init__.py -> build/lib.linux-x86_64-3.4/multidict

copying multidict/_multidict_py.py -> build/lib.linux-x86_64-3.4/multidict

running egg_info

writing multidict.egg-info/PKG-INFO

writing dependency_links to multidict.egg-info/dependency_links.txt

writing top-level names to multidict.egg-info/top_level.txt

warning: manifest_maker: standard file '-c' not found



reading manifest file 'multidict.egg-info/SOURCES.txt'

reading manifest template 'MANIFEST.in'

warning: no previously-included files matching '*.pyc' found anywhere in distribution

warning: no previously-included files found matching 'multidict/_multidict.html'

warning: no previously-included files found matching 'multidict/_multidict.*.so'

warning: no previously-included files found matching 'multidict/_multidict.pyd'

warning: no previously-included files found matching 'multidict/_multidict.*.pyd'

no previously-included directories found matching 'docs/_build'

writing manifest file 'multidict.egg-info/SOURCES.txt'

copying multidict/__init__.pyi -> build/lib.linux-x86_64-3.4/multidict

copying multidict/_istr.c -> build/lib.linux-x86_64-3.4/multidict

error: can't copy 'multidict/_multidict.c': doesn't exist or not a regular file

Error running clean

rm .install-deps
rm: cannot remove '.install-deps': No such file or directory

These two lines in Makefile, under section clean::

    rm .install-deps
    rm .develop

should have a -f or -rf flag added to avoid such errors.