Merak is a package building toolkit.
This project is started as an attempt to implement a tool that builds a single Cython extension from a Python package, based on the discussion on StackOverflow - Collapse multiple submodules to one Cython extension. See the Idea section below.
More features and functionalities may be added in the future.
To install the current release:
$ pip install merakTo upgrade Merak to the latest version, add --upgrade flag to the above command.
Currently, Merak only supports the cythonize command for building binary extension from a Python package. More features and functionalities may be added in the future.
To build a binary extension from a Python package:
$ merak cythonize PACKAGE_PATH OUTPUT_PATHThe package built will be placed at <OUTPUT_PATH>/<PACKAGE_NAME>. If -f is specified, any existing file / directory at this path will be overwritten.
usage: merak cythonize [-h] [-v] [-k] [-s SEP] [-f] path output
positional arguments:
path Python package path
output Output directory
optional arguments:
-h, --help show this help message and exit
-v, --verbose Log verbosity level. Default -> WARNING, -v -> INFO, -vv
or above -> DEBUG.
-k, --color Display logging messages in colors.
-s SEP, --sep SEP Module layer separator, must be Python identifier.
Defaults to '_'
-f, --force Force overwrite if target path exists
--py-cmd PY_CMD Python interpreter to be used for building Cython
package. Defaults to value of environment variable
"PYTHON_CMD", or "python" if "PYTHON_CMD" not specified.
An example package foo is included in the examples/ directory. It consists of one subpackage bar with a module baz containing a function do() in it.
To build the foo package, run the following command in the project root:
$ merak cythonize examples/foo foo-buildThe foo binary package can then be found at foo-build/foo. Change directory to foo-build and use an interactive Python session to try it out:
>>> from foo.bar import baz
__main__:1: DeprecationWarning: Deprecated since Python 3.4. Use importlib.util.find_spec() instead.
>>> baz.do()
Running: foo.bar.baz.do()
The deprecation warning seems to originate from the import logic in the compiled __init__ extension by Cython. It should cause no execution problems at all.
The binary package can be built into a Python distribution via setuptools by simply adding a setup.py in the output directory that includes the cython extension as package data. For this example, add setup.py to foo-build/ with the following content:
import setuptools
setuptools.setup(
name="foo",
version="0.1.0",
packages=["foo"],
include_package_data=True,
package_data={"foo": ["*"]},
)and run
$ python setup.py bdist_wheelThe distribution can be found at foo-build/dist/.
Based on this answer, it appears that it is possible to build a single Cython extension with multiple modules included in it.
However, it does NOT work with multi-level packages. Cython builds a C source file for each module with an initializer named PyInit_xxx, which depends on the base name of the module. As the function is defined in the global scope, a name collision would happen if the same base name is used for different modules. For instance, the following package would have a name collision for __init__.py and base.py:
foo/
__init__.py
bar/
__init__.py
base.py
baz/
__init__.py
base.py
Here, we solve the problem in two steps:
- Module Flattening: We move all modules to the base layer, with name constructed from their original relative path:
path.replace(path_separator, sep), wheresepis a legal Python identifier. For example,foo/bar/base.py->foo/bar_sep_base.pyifsep="_sep_". - Import Redirection: We inject a finder inside the main
__init__.pythat redirects dotted-paths to their flattened counterparts. Using the above example, the finder redirects the importfoo.bar.basetofoo.bar_sep_base.
The injected finder is based on this answer with some modifications. See the template for implementation detail.
The result would contain a single __init__ extension inside the package folder. The package folder is still required for the builtin importer to load it as a package, rather than a module. The above example would result in a foo/ folder with a single __init__ Cython extension in it.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent