aio-libs / multidict Goto Github PK
View Code? Open in Web Editor NEWThe multidict implementation
Home Page: https://multidict.aio-libs.org
License: Other
The multidict implementation
Home Page: https://multidict.aio-libs.org
License: Other
The reason is multidict.add(key, val)
is ten times slower than dict[key] = val
.
This is because multidict stores data internally as a list of cythonized _Item
objects.
But creation of python (ever cythonized) object is too expensive for our use case.
The solution is using C structs for internal data but Cython has no support for visiting values stored in these structures: tp_visit
and tp_clear
slots.
Thus for sake of speed we need pure C implementation.
Pickled pure Python MultiDicts should be unpickled as Cythonized if unpicker is executed on machine with Cython support.
And vise versa.
The documentation's changelog reads currently:
- Accept multiple keys in
MultiDict.update()
andCIMultiDict.update()
(PR #199)
The link leads to https://github.com/aio-libs/multidict/pulls/199, which is broken; it should point to #199 (which GitHub is unhelpfully contracting here); that is s/pulls
/pull
. (If I knew / find where this is done, I would send a PR, but the source seems to simply say :pr:199:
)
Now dependency tree for test_multidict.py
is too complicated ever for me.
We need to slip it into several files, rewrite tests in pytest style and reduce test class dependencies.
Like:
READONLY_DICTS = [MultiDict, CIMultiDict, MultiDictProxy, CIMultiDictProxy] # plus pure python classes
@pytest.parametrize(READONLY_DICTS)
def test_something(dct):
pass
Iteration over keys, items and values should raise RuntimeError
if dict is changed.
Deriving from str
is too expensive.
We need fast and simple way to figure out is key already titled (canonized) or not.
The alternative could be multidict.canonize(s: str) -> str
function.
It converts parameter to s.title()
, intern it by sys.intern(...)
call and store the result into global dictionary.
Thus we have a very fast check for canonized strings.
The drawback or proposal is the canonized strings are never deleted (I think it's fine for aiohttp.hdrs
usage). All canonize()
calls should be done on module import stage, that's why I propose a new function instead of adding new tricks to istr
.
istr
will be supported for years anyway but canonize
is the only way to keep multidicts fast.
Right now istr
calls is 10-25% slower on MultiDict
-- its not acceptable (but it's still much faster on CIMultiDict
lookups than plain str
).
It could amortize a cost for adding new items into multidicts and as result improve performance.
Of course #97 would be much better but it implementation is much harder.
Python 3.5.2, multidict-2.0.0 from PyPI, following code fails with SIGSEGV:
from multidict import MultiDict
set() - MultiDict().keys()
If i understood correctly, error happens in PyAnySet_Check
call, since set substraction is basically difference
method call with type check:
if (!PyAnySet_Check(so) || !PyAnySet_Check(other)) {
Py_INCREF(Py_NotImplemented);
return Py_NotImplemented;
}
return set_difference(so, other);
It also can be reproduced with dict and list objects, so i think problem may be related to type checking.
Here's a small test case:
>>> import multidict
>>> m=multidict.CIMultiDict({multidict.istr('Foo'): 'bar'})
>>> m.copy()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "multidict/_multidict.pyx", line 325, in multidict._multidict.MultiDict.copy (multidict/_multidict.c:6830)
File "multidict/_multidict.pyx", line 251, in multidict._multidict.MultiDict.__init__ (multidict/_multidict.c:5495)
File "multidict/_multidict.pyx", line 289, in multidict._multidict.MultiDict._extend (multidict/_multidict.c:6115)
TypeError: Expected str, got istr
MultiDict._extend
expects every key to be a str. aiohttp.hdrs
creates istr
objects, so request.headers.copy()
doesn't work.
Usually you use v
prefix, but not in 2.1.4. Mistake or plan for further releases?
When updating existing key in multidict, it is moved to the end (original order is not preserved):
>>> d = CIMultiDict()
>>> d['1'] = '2'
>>> d['3'] = '4'
>>> d
<CIMultiDict('1': '2', '3': '4')>
>>> d['1'] = '10'
>>> d
<CIMultiDict('3': '4', '1': '10')>
In above example, guarantee of preserving insertion order should leave existing key '1'
on its original first position.
I'm using multidict 3.1.0 and python 3.6.1.
I get the following warning when my application starts:
/usr/lib/python3.6/importlib/_bootstrap.py:205: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Running the same application with PYTHONWARNINGS=error
, I can track down the issue to multidict:
...
File "usr/lib/python3.6/site-packages/aiohttp/__init__.py", line 5, in <module>
File "usr/lib/python3.6/site-packages/aiohttp/hdrs.py", line 2, in <module>
File "usr/lib/python3.6/site-packages/multidict/__init__.py", line 24, in <module>
File "multidict/_multidict.pyx", line 9, in init multidict._multidict (multidict/_multidict.c:16318)
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Seems similar to #79. Apparently, #81 didn't fix the issue completely.
As it is implemented now, Multidict preserves order.
Because of the way it is used in aiohttp (parsing post items and HTTP headers), this seems a very desirable property, and not merely an implementation detail.
I think it should be documented both here and in aiohttp's docs that Multidict preserves order, so that one doesn't need to read the implementation to be sure.
I was a little surprised by:
>>> import multidict
>>> foo = multidict.CIMultiDict()
>>> print(isinstance(foo, dict))
False
Hi!
Similar to the issues experienced with aiohttp (eg aio-libs/aiohttp#2347), there is a massive delay (currently 3 hours, but the other releases are still outstanding) between the Windows wheel for a new multidict release being uploaded to PyPI, and when the sdist/other wheels (manylinux/OS X) are available.
This is problematic, since tooling such as pyup.io see the new release in the PyPI feed, and so open a PR to update the dependency. However at that point only the Windows wheel exists, causing the Travis run to fail (eg mozilla/treeherder#2899).
For possible solutions, see:
aio-libs/aiohttp#2347 (comment)
Many thanks :-)
Issue with multidict installation with version 3.3.0
Even old versions have same issue.
Share what is missing?
multidict-3.3.0$ python --version
Python 3.6.1
multidict-3.3.0$ sudo python setup.py install
running install
running bdist_egg
running egg_info
creating multidict.egg-info
writing multidict.egg-info/PKG-INFO
writing top-level names to multidict.egg-info/top_level.txt
writing dependency_links to multidict.egg-info/dependency_links.txt
writing manifest file 'multidict.egg-info/SOURCES.txt'
reading manifest file 'multidict.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '.pyc' found anywhere in distribution
warning: no previously-included files found matching 'multidict/_multidict.html'
warning: no previously-included files found matching 'multidict/_multidict..so'
warning: no previously-included files found matching 'multidict/_multidict.pyd'
warning: no previously-included files found matching 'multidict/_multidict.*.pyd'
no previously-included directories found matching 'docs/_build'
writing manifest file 'multidict.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/multidict
copying multidict/_multidict_py.py -> build/lib.linux-x86_64-2.7/multidict
copying multidict/init.py -> build/lib.linux-x86_64-2.7/multidict
copying multidict/_abc.py -> build/lib.linux-x86_64-2.7/multidict
copying multidict/_compat.py -> build/lib.linux-x86_64-2.7/multidict
copying multidict/init.pyi -> build/lib.linux-x86_64-2.7/multidict
copying multidict/_istr.c -> build/lib.linux-x86_64-2.7/multidict
error: can't copy 'multidict/_multidict.c': doesn't exist or not a regular file
DEBUG: + /usr/bin/python3.5 setup.py test
DEBUG: running pytest
DEBUG: running egg_info
DEBUG: writing multidict.egg-info/PKG-INFO
DEBUG: writing top-level names to multidict.egg-info/top_level.txt
DEBUG: writing dependency_links to multidict.egg-info/dependency_links.txt
DEBUG: reading manifest file 'multidict.egg-info/SOURCES.txt'
DEBUG: reading manifest template 'MANIFEST.in'
DEBUG: warning: no files found matching 'CHANGES.rst'
DEBUG: warning: no previously-included files matching '*.pyc' found anywhere in distribution
DEBUG: warning: no previously-included files found matching 'multidict/_multidict.html'
DEBUG: warning: no previously-included files found matching 'multidict/_multidict.pyd'
DEBUG: warning: no previously-included files found matching 'multidict/_multidict.*.pyd'
DEBUG: no previously-included directories found matching 'docs/_build'
DEBUG: writing manifest file 'multidict.egg-info/SOURCES.txt'
DEBUG: running build_ext
DEBUG: usage: setup.py [options] [file_or_dir] [file_or_dir] [...]
DEBUG: setup.py: error: unrecognized arguments: --cov=multidict
DEBUG: inifile: /builddir/build/BUILD/multidict-4.2.0/pytest.ini
DEBUG: rootdir: /builddir/build/BUILD/multidict-4.2.0
Because the latest release does not have an python 3.6 wheel multidict, now breaks installing all of my discord bot's dependencies. The bad part about that is the bot is not supposed to be offline very ling but yet multidict and the latest yarl breaks it for python 3.6 Please fix these issues.
For more information see aio-libs/aiohttp#1727
The entire output of the console (including the failure) is this:
Collecting discord.py (from -r requirements.txt (line 1))
Using cached discord.py-0.16.7.tar.gz
Collecting aiohttp (from -r requirements.txt (line 2))
Using cached aiohttp-1.3.5.tar.gz
Collecting PyNacl (from -r requirements.txt (line 3))
Using cached PyNaCl-1.1.1-cp36-cp36m-win32.whl
Collecting pycares (from -r requirements.txt (line 4))
Using cached pycares-2.1.1.tar.gz
Collecting cchardet (from -r requirements.txt (line 5))
Using cached cchardet-1.1.3-cp36-cp36m-win32.whl
Collecting cffi (from -r requirements.txt (line 6))
Using cached cffi-1.9.1-cp36-cp36m-win32.whl
Collecting multidict (from -r requirements.txt (line 7))
Using cached multidict-2.1.4.tar.gz
Collecting yarl (from -r requirements.txt (line 8))
Using cached yarl-0.10.0.tar.gz
Collecting discord.webhooks (from -r requirements.txt (line 9))
Using cached discord.webhooks-0.0.2.tar.gz
Collecting websockets<4.0,>=3.1 (from discord.py->-r requirements.txt (line 1))
Using cached websockets-3.2-py33.py34.py35-none-any.whl
Collecting chardet (from aiohttp->-r requirements.txt (line 2))
Using cached chardet-2.3.0-py2.py3-none-any.whl
Collecting async_timeout>=1.1.0 (from aiohttp->-r requirements.txt (line 2))
Using cached async_timeout-1.2.0-py3-none-any.whl
Collecting six (from PyNacl->-r requirements.txt (line 3))
Using cached six-1.10.0-py2.py3-none-any.whl
Collecting pycparser (from cffi->-r requirements.txt (line 6))
Using cached pycparser-2.17.tar.gz
Installing collected packages: chardet, multidict, async-timeout, yarl, aiohttp,
websockets, discord.py, pycparser, cffi, six, PyNacl, pycares, cchardet, discor
d.webhooks
Running setup.py install for multidict ... error
Complete output from command E:\Python360\python.exe -u -c "import setuptool
s, tokenize;__file__='E:\\Users\\Elsword\\AppData\\Local\\Temp\\pip-build-j53pka
8u\\multidict\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read
().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" instal
l --record E:\Users\Elsword\AppData\Local\Temp\pip-61352pwp-record\install-recor
d.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build\lib.win32-3.6
creating build\lib.win32-3.6\multidict
copying multidict\_multidict_py.py -> build\lib.win32-3.6\multidict
copying multidict\__init__.py -> build\lib.win32-3.6\multidict
running egg_info
writing multidict.egg-info\PKG-INFO
writing dependency_links to multidict.egg-info\dependency_links.txt
writing top-level names to multidict.egg-info\top_level.txt
reading manifest file 'multidict.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.pyc' found anywhere in dis
tribution
warning: no previously-included files found matching 'multidict\_multidict.h
tml'
warning: no previously-included files found matching 'multidict\_multidict.*
.so'
warning: no previously-included files found matching 'multidict\_multidict.p
yd'
warning: no previously-included files found matching 'multidict\_multidict.*
.pyd'
no previously-included directories found matching 'docs\_build'
writing manifest file 'multidict.egg-info\SOURCES.txt'
copying multidict\__init__.pyi -> build\lib.win32-3.6\multidict
copying multidict\_multidict.c -> build\lib.win32-3.6\multidict
copying multidict\_multidict.pyx -> build\lib.win32-3.6\multidict
running build_ext
building 'multidict._multidict' extension
creating build\temp.win32-3.6
creating build\temp.win32-3.6\Release
creating build\temp.win32-3.6\Release\multidict
E:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\cl.exe /c /nologo
/Ox /W3 /GL /DNDEBUG /MD -IE:\Python360\include -IE:\Python360\include "-IE:\Pr
ogram Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IE:\Program Files (
x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IE:\Program Files (x86)\W
indows Kits\10\include\10.0.10586.0\ucrt" "-IE:\Program Files (x86)\Windows Kits
\NETFXSDK\4.6.1\include\um" "-IE:\Program Files (x86)\Windows Kits\10\include\10
.0.10586.0\shared" "-IE:\Program Files (x86)\Windows Kits\10\include\10.0.10586.
0\um" "-IE:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" "-IE
:\Program Files (x86)\Microsoft Visual Studio\VC98\atl\include" "-IE:\Program Fi
les (x86)\Microsoft Visual Studio\VC98\mfc\include" "-IE:\Program Files (x86)\Mi
crosoft Visual Studio\VC98\include" /Tcmultidict/_multidict.c /Fobuild\temp.win3
2-3.6\Release\multidict/_multidict.obj
_multidict.c
multidict/_multidict.c(11): fatal error C1083: Cannot open include file: 'Py
thon.h': No such file or directory
************************************************************
Cannot compile C accelerator module, use pure python version
************************************************************
running install
running build
running build_py
creating build\lib
creating build\lib\multidict
copying multidict\_multidict_py.py -> build\lib\multidict
copying multidict\__init__.py -> build\lib\multidict
running egg_info
writing multidict.egg-info\PKG-INFO
writing dependency_links to multidict.egg-info\dependency_links.txt
writing top-level names to multidict.egg-info\top_level.txt
reading manifest file 'multidict.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.pyc' found anywhere in dis
tribution
warning: no previously-included files found matching 'multidict\_multidict.h
tml'
warning: no previously-included files found matching 'multidict\_multidict.*
.so'
warning: no previously-included files found matching 'multidict\_multidict.p
yd'
warning: no previously-included files found matching 'multidict\_multidict.*
.pyd'
no previously-included directories found matching 'docs\_build'
writing manifest file 'multidict.egg-info\SOURCES.txt'
copying multidict\__init__.pyi -> build\lib\multidict
copying multidict\_multidict.c -> build\lib\multidict
copying multidict\_multidict.pyx -> build\lib\multidict
running install_lib
running install_egg_info
removing 'E:\Python360\Lib\site-packages\multidict-2.1.4-py3.6.egg-info' (an
d everything under it)
Copying multidict.egg-info to E:\Python360\Lib\site-packages\multidict-2.1.4
-py3.6.egg-info
running install_scripts
Traceback (most recent call last):
File "E:\Python360\lib\distutils\_msvccompiler.py", line 382, in compile
self.spawn(args)
File "E:\Python360\lib\distutils\_msvccompiler.py", line 501, in spawn
return super().spawn(cmd)
File "E:\Python360\lib\distutils\ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "E:\Python360\lib\distutils\spawn.py", line 38, in spawn
_spawn_nt(cmd, search_path, dry_run=dry_run)
File "E:\Python360\lib\distutils\spawn.py", line 81, in _spawn_nt
"command %r failed with exit status %d" % (cmd, rc))
distutils.errors.DistutilsExecError: command 'E:\\Program Files (x86)\\Micro
soft Visual Studio 14.0\\VC\\BIN\\cl.exe' failed with exit status 2
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "E:\Users\Elsword\AppData\Local\Temp\pip-build-j53pka8u\multidict\set
up.py", line 46, in build_extension
build_ext.build_extension(self, ext)
File "E:\Python360\lib\distutils\command\build_ext.py", line 533, in build
_extension
depends=ext.depends)
File "E:\Python360\lib\distutils\_msvccompiler.py", line 384, in compile
raise CompileError(msg)
distutils.errors.CompileError: command 'E:\\Program Files (x86)\\Microsoft V
isual Studio 14.0\\VC\\BIN\\cl.exe' failed with exit status 2
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "E:\Users\Elsword\AppData\Local\Temp\pip-build-j53pka8u\multidict\set
up.py", line 103, in <module>
setup(**args)
File "E:\Python360\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "E:\Python360\lib\distutils\dist.py", line 955, in run_commands
self.run_command(cmd)
File "E:\Python360\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "E:\Python360\lib\site-packages\setuptools\command\install.py", line
61, in run
return orig.install.run(self)
File "E:\Python360\lib\distutils\command\install.py", line 545, in run
self.run_command('build')
File "E:\Python360\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "E:\Python360\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "E:\Python360\lib\distutils\command\build.py", line 135, in run
self.run_command(cmd_name)
File "E:\Python360\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "E:\Python360\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "E:\Users\Elsword\AppData\Local\Temp\pip-build-j53pka8u\multidict\set
up.py", line 40, in run
build_ext.run(self)
File "E:\Python360\lib\distutils\command\build_ext.py", line 339, in run
self.build_extensions()
File "E:\Python360\lib\distutils\command\build_ext.py", line 448, in build
_extensions
self._build_extensions_serial()
File "E:\Python360\lib\distutils\command\build_ext.py", line 473, in _buil
d_extensions_serial
self.build_extension(ext)
File "E:\Users\Elsword\AppData\Local\Temp\pip-build-j53pka8u\multidict\set
up.py", line 49, in build_extension
raise BuildFailed()
__main__.BuildFailed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "E:\Users\Elsword\AppData\Local\Temp\pip-build-j53pka8u\multidict\set
up.py", line 110, in <module>
setup(**args)
File "E:\Python360\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "E:\Python360\lib\distutils\dist.py", line 955, in run_commands
self.run_command(cmd)
File "E:\Python360\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "E:\Python360\lib\site-packages\setuptools\command\install.py", line
61, in run
return orig.install.run(self)
File "E:\Python360\lib\distutils\command\install.py", line 557, in run
self.run_command(cmd_name)
File "E:\Python360\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "E:\Python360\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "E:\Python360\lib\site-packages\setuptools\command\install_scripts.py
", line 35, in run
bw_cmd = self.get_finalized_command("bdist_wininst")
File "E:\Python360\lib\distutils\cmd.py", line 298, in get_finalized_comma
nd
cmd_obj = self.distribution.get_command_obj(command, create)
File "E:\Python360\lib\distutils\dist.py", line 846, in get_command_obj
klass = self.get_command_class(command)
File "E:\Python360\lib\site-packages\setuptools\dist.py", line 489, in get
_command_class
self.cmdclass[command] = cmdclass = ep.load()
File "E:\Python360\lib\site-packages\pkg_resources\__init__.py", line 2291
, in load
return self.resolve()
File "E:\Python360\lib\site-packages\pkg_resources\__init__.py", line 2297
, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "E:\Python360\lib\site-packages\setuptools\command\bdist_wininst.py",
line 1, in <module>
import distutils.command.bdist_wininst as orig
ModuleNotFoundError: No module named 'distutils.command.bdist_wininst'
----------------------------------------
Command "E:\Python360\python.exe -u -c "import setuptools, tokenize;__file__='E:
\\Users\\Elsword\\AppData\\Local\\Temp\\pip-build-j53pka8u\\multidict\\setup.py'
;f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n')
;f.close();exec(compile(code, __file__, 'exec'))" install --record E:\Users\Elsw
ord\AppData\Local\Temp\pip-61352pwp-record\install-record.txt --single-version-e
xternally-managed --compile" failed with error code 1 in E:\Users\Elsword\AppDat
a\Local\Temp\pip-build-j53pka8u\multidict\
Press any key to continue . . .
btw how did you guys get the rst to show batdges properly and stuff? I could never get the rst to show properly in pypi:
https://pypi.python.org/pypi/discord.webhooks<-- one of my packages I could never get the rst to act right on.
PEP-561 enables package maintainers to distribute their typing stub files and have mypy (as of version 0.590) use them from the installed package. Since multidict already includes stub files, the only thing that needs to be done is to distribute a file named py.typed
in the package directory.
When using the pure Python implementation, if a MultiDict
is constructed with a generator argument:
headers = CIMultiDict(
(
k.decode('utf-8', 'surrogateescape'),
v.decode('utf-8', 'surrogateescape'),
)
for k, v in event.headers
)
then the resulting MultiDict
will be empty, instead of containing the key/value pairs as expected. This is because the generator is iterated over twice, and the first iteration discards all of the pairs.
Should remove all key occurrences and return a list with all removed values.
>>> from multidict import CIMultiDict
>>> d = CIMultiDict(key='value')
>>> d['key']
'value'
>>> d.pop('key')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "multidict/_multidict.pyx", line 385, in multidict._multidict.MultiDict.pop (multidict/_multidict.c:7811)
KeyError: 'key'
Calling pop
should return 'value'
and remove the item from the dict.
KeyError
Python: Python 3.5.1 (default, Dec 14 2015, 09:21:15) [GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
System: Mac OS X El Capitan 10.11.5
multidict version: 1.0.3, installed from PyPI
Good day.
I'm currently trying to rewrite multidict on top of OrderedDict/dict (4fun) and faced a behavior that is not clear for me.
So _ItemsView
returns a items in order they appear and that is correct. Nonetheless we are not able to retrieve values by keys in this example:
d2 = MultiDict([('1', 'foo'), ('2', 'bar'), ('1', 'baz')])
for k in d2:
print('%s => %s' % (k, d2[k]))
1 => foo
2 => bar
1 => foo
I know that it is a right behavior for MultiDict
but is it a most expected one?
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files found matching 'multidict/_multidict.html'
warning: no previously-included files found matching 'multidict/_multidict.*.so'
warning: no previously-included files found matching 'multidict/_multidict.pyd'
warning: no previously-included files found matching 'multidict/_multidict.*.pyd'
no previously-included directories found matching 'docs/_build'
clang: error: no such file or directory: 'multidict/_multidict.c'
clang: error: no input files
************************************************************
Cannot compile C accelerator module, use pure python version
************************************************************
Steps to reproduce:
It could speedup headers caching
Missing codecov
@webknjaz please take a look
Hi there! Great project! I was reviewing the implementation (love the cython bindings), but it occurred to me: have you considered using a linked-list + dict approach for the internals of the MultiDict? I may be wrong, but as it stands now, it seems like the access time may be O(n). It's good because it seems to preserve order, which I appreciate (some servers are header-order sensitive and query param order-sensitive).
I understand O(n) is probably ok most of the time for some protocols (HTTP probably has <10 headers on most requests), but I figured I'd ask anyway. Keep up the great work!
I am have a Flask app hosted in a gunicorn server behind a nginx reverse proxy. The app is not on the / URL but under a
Updating from 1.2.2 to latest cause the Flask app to return 404 for all routes. Downgrading back to 1.2.2 fixed the issues.
Not sure if this is a Flask issue or multidict.
To effective detect a group of classes
if isinstance(a, BaseMultidict):
# dosomething1
elif isinstances (a, Mapping):
# dosomething2
The simplest implementation is less effective
BaseMultidict = (CIMultiDict, MultiDict, MultiDictProxy)
Hi, I'm a distro packager and curious about the versioning with 'a'.
Are v3.1.4a0 โฆ v3.3.0a0 meant to be alpha releases that should not be distributed?
Or are those now "regular" version schemes and distros should pick up those versions?
thanks for clarifying and sorry for the noise
First noticed this in 3.3.2 (probably existed before that), still exists in 4.0.0:
As one would expect (IMO) the Cython version uses the title-ized key for matching and preserves the original key for output/serialization. The Python version throws away the original key completely and uses the title-ized key for both purposes, breaking output. Again, that's my opinion, but in any case, the behavior should be the same.
In [1]: from multidict._multidict import CIMultiDict # Cython
In [2]: md = CIMultiDict()
...: md['some_key'] = 1
...: list(md.items())
...:
Out[2]: [('some_key', 1)] # outputs original key
In [3]: from multidict._multidict_py import CIMultiDict # Python
In [4]: md = CIMultiDict()
...: md['some_key'] = 1
...: list(md.items())
...:
Out[4]: [('Some_Key', 1)] # outputs title-ized key
with gdb
Program received signal SIGILL, Illegal instruction.
0x00007ffff00b34aa in PyInit__multidict () from /opt/spot4/bin/lib/python3.6/multidict._multidict.so
I moved cx_freezed code to fresh installed debian. ldd
has no missed libs. But there is this trouble.
After deleting it All other libs loaded successfully. And previous version of aiohttp works well after moving.
There is an issue with CIMultiDict
copy method
import multidict
multidict.__version__ # '3.1.1'
a = multidict.CIMultiDict()
a['foo'] = 6
b = a.copy()
b['foo'] = 7
b # <CIMultiDict('foo': 7)>
a # <CIMultiDict('foo': 7)>
And using the copy
function
import copy
c = copy.copy(a)
I get
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../venv/lib/python3.6/copy.py", line 106, in copy
return _reconstruct(x, None, *rv)
File ".../venv/lib/python3.6/copy.py", line 274, in _reconstruct
y = func(*args)
File "multidict/_multidict.pyx", line 531, in multidict._multidict.CIMultiDict.__init__ (multidict/_multidict.c:9519)
File "multidict/_multidict.pyx", line 327, in multidict._multidict.MultiDict._extend (multidict/_multidict.c:6702)
TypeError: CIMultiDict takes either dict or list of (key, value) tuples
rm .install-deps
rm: cannot remove '.install-deps': No such file or directory
These two lines in Makefile
, under section clean:
:
rm .install-deps
rm .develop
should have a -f
or -rf
flag added to avoid such errors.
@webknjaz has given admin rights for the project.
He worked hard on travis and deployment setup, new access privileges will help him to do his duty faster and better.
He is a committer already BTW
I'm attempting to build a distribution package from the source on PyPi. However, the source contains the compiled shared library _istr.cpython-34m.so.
This causes problems if the arch and Python version don't match up with the library. Manually removing the file from the source solves the problem but it doesn't seem like this should exist in the source package?
I also noticed the source on PyPi does not match up with the source on Github. Github does not contain the _multidict.c file. Is this also a build artifact that was compiled from the pyx file?
The url I'm pulling source from is:
https://files.pythonhosted.org/packages/source/m/multidict/multidict-%{version}.tar.gz
$ python -We -c "import multidict as x; print(x.__version__)"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/antoine/miniconda3/envs/dask36/lib/python3.6/site-packages/multidict/__init__.py", line 24, in <module>
from ._multidict import (MultiDictProxy,
File "multidict/_multidict.pyx", line 1, in init multidict._multidict (multidict/_multidict.c:14817)
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Classes are not intended for inheritance.
For pure python implementation we could make a metaclass for the check easy.
See
https://pypi.python.org/pypi?:action=display&name=multidict&version=3.3.0a1#downloads
Maybe handling travis deploy instruction has changed, don't know.
I'm inclined to use twine
explicitly for uploading both tarball and manylinux wheels.
Moreover ./tools/run_docker.sh "multidict"
generates extra _istr.xx.so
for python 3.6
even it is executed for building 3.4
wheel.
I'll try to investigate what's going on.
@webknjaz if you have a time to take a look --- it would be awesome.
You are travis jedi, I suspect the deploy is broken after your changes in our build process.
Hi
would be great, if a tar-ball could be made available on pypi (useful for, e.g. linux distributions for creating a package). https://pypi.python.org/pypi/multidict currently only has whl packages.
Thanks
Arun
It now removes all keys but returns the last value
Should remove only first key.
pop()
should be an alias for new method .popone()
to mimic get/getone
.
Hi,
# Warning: This script may cost about 100MB of memory.
import gc
import time
from multidict import istr
for i in range(1000000):
# the memory of this temporary instance should be released after this loop,
# but the result is not.
istr('Content-Type')
gc.collect()
print('Done.')
time.sleep(300)
Thanks!
It looks like they've improved compatibility with CPython in v6.0
Ref: https://twitter.com/pypyproject/status/989392997300260865
Ref: travis-ci/travis-ci#9542
Steps to reproduce
FROM ubuntu:14.04
RUN apt-get update && apt-get -y install python3-pip
RUN bash -c "pip3 install multidict==3.2.0"
Result:
Downloading/unpacking multidict==3.2.0
Downloading multidict-3.2.0.tar.gz
Running setup.py (path:/tmp/pip_build_root/multidict/setup.py) egg_info for package multidict
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files found matching 'multidict/_multidict.html'
warning: no previously-included files found matching 'multidict/_multidict.*.so'
warning: no previously-included files found matching 'multidict/_multidict.pyd'
warning: no previously-included files found matching 'multidict/_multidict.*.pyd'
no previously-included directories found matching 'docs/_build'
Installing collected packages: multidict
Running setup.py install for multidict
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files found matching 'multidict/_multidict.html'
warning: no previously-included files found matching 'multidict/_multidict.*.so'
warning: no previously-included files found matching 'multidict/_multidict.pyd'
warning: no previously-included files found matching 'multidict/_multidict.*.pyd'
no previously-included directories found matching 'docs/_build'
error: can't copy 'multidict/_multidict.c': doesn't exist or not a regular file
Complete output from command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/multidict/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-rwpv1_j1-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.4
creating build/lib.linux-x86_64-3.4/multidict
copying multidict/__init__.py -> build/lib.linux-x86_64-3.4/multidict
copying multidict/_multidict_py.py -> build/lib.linux-x86_64-3.4/multidict
running egg_info
writing multidict.egg-info/PKG-INFO
writing dependency_links to multidict.egg-info/dependency_links.txt
writing top-level names to multidict.egg-info/top_level.txt
warning: manifest_maker: standard file '-c' not found
reading manifest file 'multidict.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files found matching 'multidict/_multidict.html'
warning: no previously-included files found matching 'multidict/_multidict.*.so'
warning: no previously-included files found matching 'multidict/_multidict.pyd'
warning: no previously-included files found matching 'multidict/_multidict.*.pyd'
no previously-included directories found matching 'docs/_build'
writing manifest file 'multidict.egg-info/SOURCES.txt'
copying multidict/__init__.pyi -> build/lib.linux-x86_64-3.4/multidict
copying multidict/_istr.c -> build/lib.linux-x86_64-3.4/multidict
error: can't copy 'multidict/_multidict.c': doesn't exist or not a regular file
As per https://hynek.me/articles/testing-packaging/ when testing --editable
(develop
) install you are not testing the package installed same way as it will be installed in by users.
@asvetlov what do you think about such approach? Sounds reasonable to me.
GitHub has publicly released feature of having several different templates for different issue types a few days ago. It looks pretty neat and I'm looking forward to integrating them into other projects.
Ref: maintainers/early-access-feedback#139
Right now a key type is dark corner: it could be str, istr or something other.
For sake of further optimizations it should be str
always.
Passing istr
as key to any miltidict API is still allowed and encouraged: it saves execution time by getting rid of s.title()
conversion.
But istr
type is ambiguous by it nature: we can define equality for str
and istr
by comparing their canonical representations but ordering is cumbersome.
Ordering by canonical reprs leads to very tricky errors, the same for original reprs.
Also CPython is very optimized for str
but not classes derived from it.
I believe the change doesn't touch our users but aiohttp might get speedup on implementing the issue.
Hi, I'm not sure if multidict has python2 support, at least there are some compatibility problems like FileNotFoundError
is not in py2 and yield from
is not valid construction in py2.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.