pebaz / nimporter

Compile Nim Extensions for Python On Import!

               

Nimporter

Directly import Nim extensions for Python and seamlessly package them for distribution in 1 line of code.

🍱 Benefits

• 🐆 Performance: Nim compiles to C

• 🚚 Distribution: Packaging Nimporter libraries is the primary use case

• 📦 Invisible: End users do not need to install Nim for source or binary distributions

• ♻️ Ecosystem: Leverage Python libraries for breadth and Nim libraries for performance.

• 🧣 Seamless: Integration with existing Nim code uses the Nimpy library.

• 🎈 Simple: Nimporter barely has a user interface at all

🐣 Installation

# 🐍 From Pypi:
$ pip install nimporter

# ⚙️ From GitHub:
$ pip install git+https://github.com/Pebaz/Nimporter

Library Author Dependencies:

1. Nim Compiler (for compiling Nim source files)
2. Nimpy library (installed automatically if nimporter init lib is used)
3. Nimporter library (distributed libraries will need access to Nimporter).

Nimporter can work seamlessly when Nim is installed via Choosenim or manually. No additional configuration is necessary once installed since Nimporter can find the Nim standard library and install Nimpy library if Nimble is on your path.

End User Dependencies:

Users of Nimporter libraries only need Nimporter! 🎉

📚 Documentation

To get started, first look at the Nimpy project as that is how Nim libraries are created that can be imported into Python. During development, Python can directly import the Nim file and build the public user-facing Python API in tandem with the Nim library extension. For assistance with the Nim language, look at Nim For Python Programmers as it is a great resource for getting up to speed quickly with Nim. Finally, Nimporter's unit tests all make use of a reference project that was designed to use each of Nimporter's features.

• 🦔 Nim For Python Programmers
• 🦊 Nimpy
• 🔅 Reference Project

Additionally, the Nimpy tests folder contains code examples on these topics:

• Passing/Returning None, booleans, integers, floats, strings, lists, tuples, dictionaries, JSON, and objects.
• Defining/Raising Python exceptions from Nim.
• Yielding values back to Python using iterators in Nim.
• Exposing Nim functions with custom names.
• Exposing Nim extension modules with customized names and docstrings.
• Using Python builtin functions in Nim.
• Using passed Python objects and accessing methods and fields.
• Passing keyword arguments to a Python function.

📋 Features

• Directly import Nim Extension Modules & Extension Libraries using Nimpy
• Cache build artifacts for quicker subsequent runs
• Invalidate cache using hash files
• Stores artifacts and hash files in __pycache__ to not clutter project
• Build Source & Binary Distributions using Nimporer with 1 line of code
• Command Line Interface for introspecting, initializing, and compiling projects
• Nimporter does not require that library end-users install a Nim compiler

🛠️ Usage

Nimporter is a library that allows the seamless import & packaging of Nim extensions for Python built with Nimpy. Nimpy is a library that is used on the Nim side for iteroperability with Python. All Nimporter libraries rely on Nimpy in order to expose Nim functions to Python. Nimporter's role in this is to formalize a method of distributing Nimpy libraries to ease the burden on library maintainers and end users so that they do not have to even have knowledge of Nim in order to use the library.

Nimpy is a complete library by itself. For information on how to integrate Nim and Python, look at the Nimpy documentation as it will be the Nim day-to-day development experience. Nimporter's role comes into play when a library is ready to be distributed. Nimporter handles the entire packaging for source and binary distributions in 1 line of code.

Important Considerations

Nimporter was designed to help bridge the ecosystem gap between Python and Nim while utilizing Nimpy so that library authors could seamlessly develop and distribute their libraries. Due to this fact, there are important limitations to consider when using Nimporter. They are described below:

1. Importing: Nimporter uses the C compiler that was used to build Python when importing a Nim module/library. This can be overridden in a <lib name>.nim.cfg but doing so means that the library will most likely not work on other platforms.

2. Distributing Sources: Nimporter sets the C compiler automatically by iterating through MSVC and GCC for each platform and architecture combo. This means that there will likely be several copies of the generated C source code for each supported platform (given in get_nim_extensions()).

3. Distributing Binaries: Nimporter uses the same process described for direct import of Nim code and will use the same C compiler that was used to build Python itself.

🎻 Instrumentation

To enable Nimporter debug traces, define NIMPORTER_INSTRUMENT in the environment and Nimporter will use IceCream to show output from Nim and other interesting bits necessary for debugging any issues that could arise.

🦓 Extension Modules & Extension Libraries

Extension Modules are distinct from Extension Libraries. Nimporter (not Nimpy) makes a distinction here. However, it is of special note that distribution of either extension type is the same (nimporter.get_nim_extensions()).

🦄 Extension Libraries

Extension Libraries are entire Nim projects exposed as a single module from the perspective of Python. They are comprised of a single folder containing all code and configuration for the extension. It is important to note that they are a concept formalized by the Nimporter project and must accept some limitations.

These limitations (and capabilities) are listed below:

• ✔️ Can have external Nim dependencies: inside the Extension Library folder, use a <library name>.nimble in order to depend upon other Nim libraries.

• ✔️ Can be split up into any number of Nim modules: the Extension Library folder can contain any desired inner folder structure.

• ✔️ CLI switches used by Nim & the C compiler can be customized: this can be useful but be cognizant about cross-platform compatibility. Remember, if the C compiler used by Python is different than the one used by Nim, there will definitely without a doubt be strange issues arising from this. Note that choosing a different C compiler may result in the setup.py not being able to compile the extension. Use a <library name>.nim.cfg for this use case.

• ❌ Must use folder structure known to Nimporter: the below folder structure is generated when nimporter init lib is used:

  the_library_name/
      the_library_name.nim  # Must be present
      the_library_name.nim.cfg  # Must be present even if empty
      the_library_name.nimble  # Must contain `requires "nimpy"`
  

🐴 Extension Modules

Extension Modules are the simplest form of using Nimpy libraries with existing Python code. Once Nimporter is imported, Nimpy libraries can be directly imported like normal Python modules. However, there are a few restrictions on what is supported when importing a Nim module in this way. It is important to remember that Nim compiles to C and therefore could theoretically integrate with a build system that is extremely brittle. To completely solve this, Nimporter disallows certain use cases that are technically possible but would otherwise prevent widespread use of the resulting technology.

Below are the restrictions present when importing a Nim Extension Module:

• ❌ Cannot have any dependencies other than Nimpy: this is due to the fact that Nimporter disallows multiple *.nimble files strewn about in a Python project. Use an Extension Library for this use case.

• ❌ Cannot import other Nim modules in same directory: this is because there is no way to tell which files pertain to each extension and knowing this is a prerequisite to packaging the extension up for distribution. Additionally, Nimporter moves extensions to temporary directories during compilation in order to control where the Nim compiler places the resultant C sources.

• ❌ Cannot customize Nim or C compiler switches: proliferating a Python package with these extra files would be unsightly and it is possible to have two different Nim modules with custom configurations collide in awkward ways. If CLI configuration is required, use an Extension Library.

• ❌ Cannot override the C compiler used to build the extension: Although this practice is certainly and technically possible, it is unequivocally a bad decision when integrating software originating from a different compilers. If an expert user is in need of this capability, use an Extension Library.

Although these restrictions limit the number of possible use cases for the integration of Nim & Python, portability, compatibility, and stability were chosen as the guiding principles for Nimporter.

📦 Distribution

There are a few ways to use Nimporter to integrate Nim & Python code:

1. 🥇 Library uses Nim code internally but exposes a Python API: this is the reason why Nimporter was built. It was built to allow Python library authors to use Nim to speed up their library.

2. 🥈 Application uses Nim code: this is very possible but it is recommended to pull out the Nim code into a Python library that imports that Nim code in order to take advantage of the amazing distribution features of Nimporter. Having a separately-updatable library that the application imports greatly streamlines development and reduces packaging difficulty (the Python library dependency that imports Nim code behaves exactly like a pure-Python dependency).

3. 🥉 Docker: this is a possible application of Nimporter, but it requires the use of nimporter compile in order to let the Docker container not have to contain a Nim & C compiler and to ensure that the builds are cached.

Amazingly, Nimporter allows the end user installing a library built with Nimporter to not have to install Nim! 🥳 This is incredible and is accomplished by recompiling the same Nim extension to every desired platform, architecture, and C compiler that the library is supported on. Specifically, Nimporter tells the Nim compiler to compile the extension to C once for Windows, MacOS, and Linux and and then bundles all of the resulting C source files into the source distribution. At the time of the installation on the end user's machine, the appropriate set of C source files is selected that matches the user's environment! 🙂

For binary distributions, this process just skips to the one set of C source files that matches the host's environment. One binary distribution per supported platform must then be built.

This might sound complicated but Nimporter accomplishes this by requesting that the setup.py contain 1 line of code to find, compile, and bundle all of the C files necessary to be portable across platform, architecture, and C compilers.

📧 Source Distributions

To create a source distribution, it is assumed that the setup.py contains a dependency on Nimporter as well as a call to get_nim_extensions().

# Example setup.py

import setuptools
from nimporter import get_nim_extensions, WINDOWS, MACOS, LINUX

setuptools.setup(
    name='calculatorlib',
    install_requires=['nimporter'],
    py_modules=['calculatorlib.py'],
    ext_modules=get_nim_extensions(platforms=[WINDOWS, LINUX, MACOS])
)

The below command will create a source distribution in the dist/ directory and can be easily uploaded to PyPI.

$ python setup.py sdist  # Contains entire matrix of supported platforms, etc.

Note: when an end-user tries to install a Nimporter library from GitHub directly, it is required that the Nim compiler and a compatible C compiler is installed because setup.py install is invoked which is equivalent to a binary distribution but does require the Nim & C compilers to be installed.

💿 Binary Distributions

Binary distributions use the same setup.py structure mentioned above.

The below command will create a Python Wheel in the dist/ directory that can be easily uploaded to PyPI.

$ python setup.py bdist_wheel  # Contains a single supported platform, etc.

Note: A Nim compiler and C compiler is required when creating a binary distribution.

Special note for Linux users: Unfortunately, PyPi will not allow you to upload just any Linux wheel. There is a special compilation process that can be explained here. Interestingly enough, I got around this by simply renaming the resulting Linux build according to the manylinux1 naming convention. You can see my solution in the examples/github_actions_template.yml file for the build-linux job. I expect that there could be many downsides of using this hack but it worked for me on 2 different Linux platforms.

⭕ Publish Build Artifacts to PyPi Automatically

For a dead-simple way to publish Windows, MacOS, and Linux packages to PyPi automatically, use the github_actions_template.yml template found in the examples/ directory. This template integrates with your repository's GitHub Actions runner to build, package, and deploy your library on Windows, MacOS, and Linux automatically when you create a new "Release" is created.

💽 Computer Hardware Actually Exists

Dynamic, safe programming languages are great, but naturally, when integrating with native code, there are limitations to what is possible to accomplish in certain situations. On Windows, a DLL that has been loaded into a process cannot be deleted while it is in use. Additionally, Windows has a path length limit of 260 characters by default (and therefore relying on the user having disabled this limit in the system registry is not possible). This severely limits how deep a Nim extension can be placed into a Python package hierarchy. Furthermore, generously-named Nim extensions may fail to compile with a message that resembles:

failed to open compiler generated file: ''

If this message occurs, it is due to the path length limit of 260 characters. Shorten the name of the Nim extension and make the package hierarchy shallower. More information about the 260 character path limit can be found here.

Nimporter comes with a defense against this behavior by automatically renaming the generated C sources to not contain the @m and @s symbols that proliferate the filename and soak up most of the 260 character budget. For instance, the filename:

@m..@s..@s..@s..@s..@s..@sUsers@s<USERNAME>@s.nimble@[email protected]@snimpy@spy_utils.nim.c

Gets turned into:

[email protected]@nimpy@py_utils.nim.c

Much shorter! 🚀

🧑‍💻 Nimporter Command Line Interface

Nimporter provides a CLI that you can use to easily clean all cached build and hash files from your project recursively. This can be very useful for debugging situations arising from stale builds.

Usage example:

# Removes all __pycache__ directories with .hash, .pyd/.so, and .pyc files
$ nimporter clean

The Nimporter CLI can also precompile all extensions within a project without needing to run the project. This is useful in situations where you do not want to package your application using a setup.py (such as a zip file) or for use within Docker containers.

# Recursively compile all Nim extension modules and libraries:
$ nimporter compile

Finally, the CLI has provisions for listing out the extensions that it can auto-detect. This is useful to identify if an extension folder structure is properly setup.

# List all extensions that Nimporter will find when handling imports
$ nimporter list

⚓ Usage with Docker

Nimporter can easily be used within a Docker container. To prevent the need for a Nim compiler toolchain to be installed into the container to run Nim code, the extensions can be precompiled and copied into the container. This process is roughly as follows:

1. Create a project that uses Python and Nim
2. Run nimporter compile to recursively-compile all extensions in the project
3. Ensure that in your Dockerfile that the __pycache__ directories are included as they will contain the Nim shared objects as well as the Nimporter hash files to prevent a recompilation (which would fail without a Nim & C compiler installed in the container).

🧪 Running The Tests

To run Nimporter's test suite on your local machine, you will need to install a Nim compiler. This example will assume you are cloning the GitHub repository.

$ git clone https://github.com/Pebaz/Nimporter
$ cd Nimporter
$ pip install -r requirements_dev.txt
$ pip install .  # Nimporter is needed for the integration tests
$ pytest --cov=. --cov-report=html tests

❓ How Does Nimporter Work?

Nimporter provides essentially two capabilities:

• The ability to directly import Nim code
• The ability to bundle Python-compatible extensions for any supported platform

The way it accomplishes the ability to import Nim code is by adding two custom importers to the Python import machinery. This is why it is required to import Nimporter before importing any Nim code because the Python import machinery must be amended with the custom importers.

The first one is for the ability to search and import Nim modules. When a Nim module is found, Nimporter first looks in the __pycache__ directory to see if there is already a built version of the module. If there is not, it builds a new one and stores it in the __pycache__ directory.

If one is found, it could be stale, meaning the Nim file could have been modified since it was built. To keep track of this, a hash of the source file is also kept in the __pycache__ directory and is consulted whenever there is a possibility that a stale build could be imported.

When a Nim module and a Python module have the same name and reside in the same folder, the Python module is given precedence. Please don't do this.

The second custom importer has the exact same purpose of the first one except it is used to import Nim extension libraries. A library is any folder within a Python project that contains a <lib name>.nim, a <lib name>.nimble, and a <lib name>.nim.cfg.

These files mark that the folder should be treated as one unit. It also makes it so that Nimble dependencies can be installed.

As for the second capability, Nimporter helps you bundle and distribute Nim code as part of a source or binary distribution extremely easily.

The way it works is by iterating through your entire project and identifying any Nim module and Nim library that it finds and compiling them to C using a feature of Nim that specifically supports this.

Why compile to C? Because Python already has extensive infrastructure to support the compilation and distribution of C extensions.

Once each Nim module and library is compiled to C, Python deals with them the exact same way as a typical C extension. These extensions are then bundled into the resulting binary distribution and can be uploaded to PyPi or similar.

For source distributions, Nimporter instructs the Nim compiler to output a copy of the generated C code for each platform, architecture, and C compiler that is supported by the library author (some libraries only make sense to work on Windows for example like DirectX). It then bundles all of these as individual C extensions into the source distribution. At installation time, Nimporter then selects the C extension that matches the end-user's host machine target triple.

👷 Contributing

Pull requests are welcome, especially for fixing bugs! 😁

Feel free to open an issue if something seems to be broken but please look through the README first if time allows.

👏 Special Thanks

Nimporter would not be possible without Nimpy. Thank you Yuriy Glukhov for making this project possible!

🌠 Stargazers Over Time

Made using https://starchart.cc/

nimporter's Issues

Prevent warnings from crashing compilation

Suggestion: make it easier to turn on dangerous mode (or support nims)

Right now, Nimporter offers great low level control over the compiler via switches.py. However, as a developer, I don't want to be worrying about maintaining my own copy of switches.py for changing simple settings.

Suppose, after #24, you decide that using the default GC is better. Now, in order to get that update, I can't just update Nimporter; I have to modify my copy-pasted-with-one-added-line switches.py file. For the more standard compiler options (danger, checks, opt), I would suggest reading from a config file or taking them as args for build_nim_extensions:

ext_modules=nimporter.build_nim_extensions(danger=True, floatChecks=True),

Or something like a .nims file:

--opt:size
--define:foo
--forceBuild

Given that the one of the major use cases for Nimporter is moving computationally intensive tasks to Nim, I imagine many users will want to use -d:danger to speed up their pipelines. For me, it makes a 3x difference, so I definitely want to use it.

Is it possible to create CLIs in Nim and distribute using Nimporter?

When compiling a CLI created in Nim using docopt, I get the following error:

Obtaining file:///Users/BenjaminLee/Desktop/Python/Research/librtd
    ERROR: Command errored out with exit status 1:
     command: /Users/BenjaminLee/.virtualenvs/librtd/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/BenjaminLee/Desktop/Python/Research/librtd/setup.py'"'"'; __file__='"'"'/Users/BenjaminLee/Desktop/Python/Research/librtd/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/gw/sk03t1cn3wlgclyc55yst44w0000gn/T/pip-pip-egg-info-3e_yn9f1
         cwd: /Users/BenjaminLee/Desktop/Python/Research/librtd/
    Complete output (19 lines):
    /Users/BenjaminLee/.nimble/pkgs/nimpy-0.1.0/nimpy.nim(972, 32) Warning: Cannot prove that 'arg1k' is initialized. This will become a compile time error in the future. [ProveInit]
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/BenjaminLee/Desktop/Python/Research/librtd/setup.py", line 11, in <module>
        ext_modules=nimporter.build_nim_extensions(),
      File "/Users/BenjaminLee/.virtualenvs/librtd/lib/python3.7/site-packages/nimporter.py", line 1070, in build_nim_extensions
        ext = cls._build_nim_extension(extension, root)
      File "/Users/BenjaminLee/.virtualenvs/librtd/lib/python3.7/site-packages/nimporter.py", line 969, in _build_nim_extension
        path, root, library=path.is_dir()
      File "/Users/BenjaminLee/.virtualenvs/librtd/lib/python3.7/site-packages/nimporter.py", line 465, in compile_nim_extension
        raise NimCompileException(errors[0])
    nimporter.NimCompileException:  commandLineParams() unsupported by dynamic libraries; usage of 'commandLineParams' is an {.error.} defined at /usr/local/Cellar/nim/1.2.0/nim/lib/pure/os.nim(2774, 3)
     |
     |
     -> proc docopt*(doc: string, argv: seq[string] = command_line_params(), help = true,
     |               version: string = "", options_first = false, quit = true
     |              ): Table[string, Value] {.gcsafe.} =

    At /Users/BenjaminLee/.nimble/pkgs/docopt-0.6.8/docopt.nim 608:47
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This appears to be because functions like paramCount don't work in dynamic libraries. So, is there way to have pure Nim CLIs distributed in Python? Or do I need to rewrite the CLI in Python?

pragmas ignored, not in nimporter.build_nim_extensions()

The extension objects in nimporter.build_nim_extensions() do not contain the values specified by:

.passC
.passL
.compile
.link

Of note, the pragma values do carry through to nim-extensions/nim-extensions/src./.json. No effect on the execution of setup.py though.

To workaround, one may add the necessary values after calling nimporter.build_nim_extensions():

modules = nimporter.build_nim_extensions()
modules[0].sources.append("src/capture_c.c")
modules[0].extra_compile_args.append("-w")
setup (
    name = "capture",
    ext_modules = modules
)

Document shipping just .nim files

I am interested in trying all ways of shipping that nimporter can do, one of the options is not documented along with the others,
thats the option of just shipping the .nim files, I made a PIP package that install Nim and other that can install Nimpy,
so would be nice to document on the readme how users can package the .nim files.
For new users it might not be so obvious.

You can make your package depend on it install_requires = ["choosenim_install"], heres an example on Windows:

Commands in order to copy & paste:

• pip install choosenim_install
• pip install nimble_install --install-option="--nimble=nimpy"

Feel free to use the screenshot if is useful for Docs.
:)

How to bypass compiling tests?

For background, since I've filed a few issues, I'm working on a library to analyze DNA and using this library to study the coronavirus genome. The pure Python version was quite slow, so I rewrote it in Nim and have achieved great (1000x+) speedups. I followed your advice in #20 and am just about ready to publish the librtd library and CLI.

Currently, I'm getting an error when testing my CLI (which is done through the Python interface) on Windows. What is weird is that the error is coming from my tests:

LINK : error LNK2001: unresolved external symbol PyInit_tests
220
    build\temp.win-amd64-3.8\Release\nim-extensions\tests.tests\tests.cp38-win_amd64.lib : fatal error LNK1120: 1 unresolved externals
221
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\link.exe' failed with exit status 1120

Since the tests don't need to be compiled, is there anyway to turn that off? Alternatively, if you have suggestions for fixing the error, I would really appreciate it. Thanks again for such a great library!

nimporter.NimporterException: Error importing faster_than_requests.pyd

`std/db_sqlite(184, 1) Warning: Circular dependency detected. 'codeReordering' pragma may not be able to reorder some nodes properly [User]
ws.nim(329, 32) Warning: conversion to enum with holes is unsafe: Opcode(b0 and 0x0000000F) [HoleEnumConv]
nimpy.nim(557, 32) Warning: Cannot prove that 'arg0timeout' is initialized. This will become a compile time error in the future. [ProveInit]
faster_than_requests.nim(293, 132) Warning: Deprecated since v1.5; Use auto instead.; any is deprecated [Deprecated]
faster_than_requests.nim(298, 101) Warning: Deprecated since v1.5; Use auto instead.; any is deprecated [Deprecated]
Traceback (most recent call last):
File "C:\Users\Nandi\AppData\Local\Programs\Python\Python39\lib\site-packages\nimporter.py", line 979, in __validate_spec
util.module_from_spec(spec)
File "", line 565, in module_from_spec
File "", line 1108, in create_module
File "", line 228, in _call_with_frames_removed
ImportError: DLL load failed while importing faster_than_requests: The specified module could not be found.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\Nandi\Desktop\python\speed.py", line 1, in
import faster_than_requests
File "C:\Users\Nandi\AppData\Local\Programs\Python\Python39\lib\site-packages\faster_than_requests_init_.py", line 7, in
from . faster_than_requests import *
File "", line 1007, in _find_and_load
File "", line 982, in _find_and_load_unlocked
File "", line 925, in _find_spec
File "C:\Users\Nandi\AppData\Local\Programs\Python\Python39\lib\site-packages\nimporter.py", line 1269, in find_spec
return Nimporter.import_nim_code(fullname, path, library=False)
File "C:\Users\Nandi\AppData\Local\Programs\Python\Python39\lib\site-packages\nimporter.py", line 957, in import_nim_code
cls.__validate_spec(spec)
File "C:\Users\Nandi\AppData\Local\Programs\Python\Python39\lib\site-packages\nimporter.py", line 1025, in _validate_spec
raise NimporterException(error_message) from import_error
nimporter.NimporterException: Error importing C:\Users\Nandi\AppData\Local\Programs\Python\Python39\Lib\site-packages\faster_than_requests_pycache\faster_than_requests.pyd
Error Message:

DLL load failed while importing faster_than_requests: The specified module could not be found.

Python Version:

3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)]

Nim Version:

Nim Compiler Version 1.6.6 [Windows: amd64]

CC Version:

vccUndefined: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat

Installed CCs:

{'msc': WindowsPath('C:/nim/bin/vccexe.EXE'), 'clang': WindowsPath('C:/Users/Nandi/AppData/Local/Nuitka/Nuitka/gcc/x86_64/10.2.0-11.0.0-8.0.0-r5/mingw64/bin/clang.EXE'), 'gcc': WindowsPath('C:/Users/Nandi/AppData/Local/Nuitka/Nuitka/gcc/x86_64/10.2.0-11.0.0-8.0.0-r5/mingw64/bin/gcc.EXE')}

Please help improve Nimporter by opening a bug report at: https://github.com/Pebaz/nimporter/issues/new and submit the above information along with your description of the issue.`

documentation

we need documentation

setup_test fails with No such file or directory: 'command line'

Python 3.8.2

Traceback (most recent call last):
  File "setup.py", line 7, in <module>
    ext_modules=nimporter.build_nim_extensions()
  File "/home/phdyex/.local/lib/python3.8/site-packages/nimporter.py", line 1070, in build_nim_extensions
    ext = cls._build_nim_extension(extension, root)
  File "/home/phdyex/.local/lib/python3.8/site-packages/nimporter.py", line 968, in _build_nim_extension
    return NimCompiler.compile_nim_extension(
  File "/home/phdyex/.local/lib/python3.8/site-packages/nimporter.py", line 465, in compile_nim_extension
    raise NimCompileException(errors[0])
  File "/home/phdyex/.local/lib/python3.8/site-packages/nimporter.py", line 56, in __init__
    with open(nim_module, 'r') as mod:
FileNotFoundError: [Errno 2] No such file or directory: 'command line'

dynamic module does not define module export function (PyInit_commands)

I'm trying to make a simple program using Python and Nim where the Python file calls a procedure that opens YouTube from the Nim file. And I get this error when importing the Nim file:

commands.nim:

import browsers

proc youtube* =
  openDefaultBrowser("https://www.youtube.com/")

main.py:

import nimporter
from commands import youtube

youtube()

And I get this error when importing the Nim file:

Error Message:

dynamic module does not define module export function (PyInit_commands)

Python Version:

3.10.1 (tags/v3.10.1:2cd268a, Dec  6 2021, 19:10:37) [MSC v.1929 64 bit (AMD64)]

Nim Version:

Nim Compiler Version 1.6.2 [Windows: amd64]

CC Version:

vccUndefined: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvarsall.bat

Installed CCs:

{'msc': WindowsPath('C:/Nim/nim-1.6.2/bin/vccexe.EXE'), 'gcc': WindowsPath('C:/Program Files/mingw-w64/x86_64-8.1.0-posix-seh-rt_v6-rev0/mingw64/bin/gcc.EXE')}

Also, if I just call something simple like an echo, or anything else, it throws the same error! In addition, this might not be important but I installed pipwin before running this code and nimporter used to work before so i wonder if those two are connected.

building wheel with build_nim_extensions()

Hi,

I am trying to build a binary wheel from the examples in the example folder. The setup.py file looks as follows:

from setuptools import setup
import nimporter

setup(
    ...,
    ext_modules=nimporter.build_nim_extensions()
)

But when I run python setup.py bdist_wheel, I get the following error message:

AttributeError: module 'nimporter' has no attribute 'build_nim_extensions'. Did you mean: 'get_nim_extensions'?

I am on a macOS BigSur, version=11.6.5, and have everything that I need installed.
I have also tried with get_nim_extensions() in the setup.py file, but with no success.

Thank you very much for this package!

Clarification request concerning `choosenim_install`

The readme has this example to auto-install the Nim compiler for source distributions, which is nice for providing code to those who don't have Nim installed.

setup(
    ...,                            # Keep your existing arguments
    package_data={'': ['*.nim*']},  # Distribute *.nim & *.nim.cfg source files
    # include_package_data=True,    # <- This line cannot work with package_data
    install_requires=[
        'nimporter',  # Must depend on Nimporter
        'choosenim_install'  # Optional. Auto-installs Nim compiler
    ]
)

However, I cannot seem to get this to work (in a fresh environment without the Nim compiler).

A pip install -e . or python setup.py develop and nimporter build [package name] results in:

FileNotFoundError: [Errno 2] No such file or directory: 'nimble'

This is on both Windows and Linux (Ubuntu).

Am I doing something wrong here?

[Question] How to dockerize a nimporter application?

When deploying I'd like the nimporter to:

• use the *.so files
• don't require the original .nim files if possible
• don't require nim to be present

Is this possible? I am having an difficulity understanding how to "freeze" nimporter for production.

Documentation error

This is about https://github.com/Pebaz/nimporter/tree/nimporter-v2.0.0rc

On this image, on the right code block, at the top,
it says calculatorlib.nim but has Python code inside, I think it should be calculatorlib.py ?.

• Which versions of Python it supports ?.

Quality Of Life - Better error message for when Nim/Nimble is not installed or can't be found

Add better error messages for the 2.0.0rc branch.

ImportError: Failed to import test module: faster_than_requests
Traceback (most recent call last):
  File "D:\soft\py39\lib\unittest\loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "D:\soft\py39\lib\unittest\loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "D:\code\me\faster-than-requests\faster_than_requests\__init__.py", line 7, in <module>
    from . faster_than_requests import *
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 945, in _find_spec
  File "D:\soft\py39\lib\site-packages\nimporter\nimporter.py", line 253, in <lambda>
    lambda fullname, path, _: nimport(fullname, path, library=False)
  File "D:\soft\py39\lib\site-packages\nimporter\nimporter.py", line 217, in nimport
    compile_extension_to_lib(ext)
  File "D:\soft\py39\lib\site-packages\nimporter\nimporter.py", line 62, in compile_extension_to_lib
    ensure_nimpy()
  File "D:\soft\py39\lib\site-packages\nimporter\lib.py", line 213, in ensure_nimpy
    code, *_ = run_process(shlex.split('nimble path nimpy'), show_output)
  File "D:\soft\py39\lib\site-packages\nimporter\lib.py", line 155, in run_process
    process = subprocess.run(
  File "D:\soft\py39\lib\subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "D:\soft\py39\lib\subprocess.py", line 969, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "D:\soft\py39\lib\subprocess.py", line 1438, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

When srcDir = "src", package name should not be prefixed by "src."

When the .nimble specifies a srcDir, the package name should not include the directory name.

Can you create a Discussion section for the project for easier collaboration.

Hi @Pebaz ,

I received your invitation to the repository, but I think it would be easier to communicate through the discussion pages.

@SekouDiaoNlp

'-d:lto' flag breaks macOS

    NIM_CLI_ARGS = [
        '--opt:speed',
        '--parallelBuild:0',
        '--gc:refc',
        '--threads:on',
        '--app:lib',
        '-d:release',
        '-d:strip',
        # '-d:lto',
        '-d:ssl',

I have to comment out the above line to achieve build on macOS.

It's here: https://github.com/Pebaz/nimporter/blob/master/nimporter.py#L130

The root seems to be nim-lang/Nim#15614

This issue is blocking juancarlospaco/faster-than-requests#156

Maybe needs something like:

if not user supplied(--os: macOS or macOSX):
    NIM_CLI_ARGS += '-d:lto'

But there's a bigger issue: These inbuilt flags can't be removed by the user's .cfg file.

This makes the library brittle.

Need some way to 'RESET' these internal defaults from the .cfg file, maybe? e.g. REMOVE='-d:lto'...

Don't assume vcc availability on Windows

OS: Windows 10 Version 2004 (Build 19041.572)
Python: Python 3.8.4 (tags/v3.8.4:dfa645a, Jul 13 2020, 16:46:45) [MSC v.1924 64 bit (AMD64)]
Nim: 1.4.0
C Compiler: gcc version 8.1.0 (i686-posix-dwarf-rev0, Built by MinGW-W64 project)

On Windows, nimporter doesn't work if the Visual Studio C Compiler is not installed. See the following stacktrace:

c:\Temp>python test.py
Traceback (most recent call last):
  File "C:\Temp\test.py", line 2, in <module>
    import nimporter, nim_math
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 914, in _find_spec
  File "O:\app\python3\lib\site-packages\nimporter.py", line 1123, in find_spec
    return Nimporter.import_nim_code(fullname, path, library=False)
  File "O:\app\python3\lib\site-packages\nimporter.py", line 806, in import_nim_code
    NimCompiler.compile_nim_code(
  File "O:\app\python3\lib\site-packages\nimporter.py", line 566, in compile_nim_code
    raise NimCompileException(errors[0])
  File "O:\app\python3\lib\site-packages\nimporter.py", line 48, in __init__
    nim_module = nim_module.splitlines()[-1]
IndexError: list index out of range

Alas, this stacktrace is not very meaningful. But the root cause is quite simple. The class NimCompiler in the file nimporter.py defines a standard set of Nim cli args. On Windows (if sys.platform == 'win32') another option is added: --cc:vcc. This is where the problem arises. IMO it is wrong to unconditionally assume the availability of vcc on Windows boxes. Nim could well be using gcc or tcc instead.

Maybe you could check if vcc is at all installed before adding this cli argument?

setup.cfg instead of setup.py

Use setup.cfg instead of setup.py on the documentation and the setup.py generator.

• https://gist.github.com/juancarlospaco/b91d7736281e2090fce84ed399c68972

.cfg can be easy to generate/parse from Nim using parsecfg module, builtin on Python.
🙂

[Bug] dynamic module does not define module export function

hello , im trying to run simple nim code inside python script with nimporter but i always got an error about importing error from GCC

• nscript.nim

proc main() =
   echo "Hello World From NIm"

• test.py

import nimporter
import nscript

print("Word :D")

☰ Error

$ python test.py

gcc: fatal error: no input files
compilation terminated.
Traceback (most recent call last):
  File "/home/knassar702/.local/lib/python3.9/site-packages/nimporter.py", line 863, in __validate_spec
    util.module_from_spec(spec)
  File "<frozen importlib._bootstrap>", line 565, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1173, in create_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
ImportError: dynamic module does not define module export function (PyInit_nscript)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/knassar702/nim_test/test.py", line 2, in <module>
    import nscript
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 982, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 925, in _find_spec
  File "/home/knassar702/.local/lib/python3.9/site-packages/nimporter.py", line 1150, in find_spec
    return Nimporter.import_nim_code(fullname, path, library=False)
  File "/home/knassar702/.local/lib/python3.9/site-packages/nimporter.py", line 841, in import_nim_code
    cls.__validate_spec(spec)
  File "/home/knassar702/.local/lib/python3.9/site-packages/nimporter.py", line 909, in __validate_spec
    raise NimporterException(error_message) from import_error
nimporter.NimporterException: Error importing /home/knassar702/nim_test/__pycache__/nscript.so
Error Message:

    dynamic module does not define module export function (PyInit_nscript)

Python Version:

    3.9.6 (default, Jun 30 2021, 15:31:20) [GCC 11.1.0]

Nim Version:

    Nim Compiler Version 1.4.8 [Linux: amd64]

CC Version:

    <Error getting version>

Installed CCs:

    {'gcc': PosixPath('/usr/bin/gcc')}

Please help improve Nimporter by opening a bug report at: https://github.com/Pebaz/nimporter/issues/new and submit the above information along with your description of the issue.

Env

• OS : Arch Linux (64)
• Nim Compiler : 1.4.8
• Python : 3.9.6

Running nimporter clean followed by setup.py adds duplicate lines to MANIFEST.in

I should ask first, is running nimporter clean followed by pip install -e . the correct way to recompile the project on changes? I haven't had success otherwise but am probably doing something wrong.

Each time I rerun nimporter clean followed by pip install -e ., I get duplicate entries in MANIFEST.in. Is this intentional?

MANIFEST.in not deleted from root directory

When run with nimporter clean this does not remove the MANIFEST.in in the root dir which results in duplicate lines being appended each time a bundle is created.

how to keep so files inside __pycache__?

This repo manages to keep all artifacts under pycache; how would I place the .so file under __pycache__ when using nimpy without nimporter?
eg, I'd like to run nim c --app:lib --out:__pycache__/mymodule.so --threads:on mymodule instead of nim c --app:lib --out:mymodule.so --threads:on mymodule in https://github.com/yglukhov/nimpy, but as is, placing the so inside the pychache doesn't work.

Thanks, I know it's a tangential question to this repo.

Bugfix - Add `--threads:on` to ALWAYS_ARGS

nimporter/nimporter/lib.py

Line 49 in 7c7aac9

ALWAYS_ARGS = [

This is regarding 2.0.0rc.

Release GH Action works on Mac but not Linux/Windows

I'm a bit stumped as to what's going wrong since Windows and Linux give different errors. On Mac, it seems to be working.

Here's the GH action. The parts that I modified are the installation steps in which I first install my package and then the dependencies for the CLI package.

The errors I'm getting can be seen here for Linux and here for Windows. Briefly, on Linux, I'm getting this:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
IndexError: list index out of range

On Windows, I'm not able to use the python3 function:

python3: D:\a\_temp\ea2d772e-0efd-40be-942c-5d46b8d41f17.ps1:8
Line |
   8 |  python3 setup.py bdist_wheel
     |  ~~~~~~~
     | The term 'python3' is not recognized as the name of a cmdlet, function, script file, or operable
     | program. Check the spelling of the name, or if a path was included, verify that the path is correct
     | and try again.

Do you have any suggestions? I'm so happy that librtd is finally close to release!

use python with mingw on windows

The closed issues made a change as

    - (['--cc:vcc'] if sys.platform == 'win32' else [])
    + (['--cc:vcc'] if shutil.which('vccexe') else [])

which looks reasonable, but should it be( sorry I have not use MSVC for a long time, cl.exe is valid for VC 6.0)

    - (['--cc:vcc'] if sys.platform == 'win32' else [])
    + (['--cc:vcc'] if shutil.which('cl.exe') else [])

however in current code, this change is lost again:

    NIM_CLI_ARGS = [
                       '--opt:speed',
                       '--parallelBuild:0',
                       '--gc:refc',
                       '--threads:on',
                       '--app:lib',
                       '-d:release',
                       '-d:strip',
                       '-d:ssl',

                       # https://github.com/Pebaz/nimporter/issues/41
                       '--warning[ProveInit]:off',

                   ] + (['--cc:vcc'] if 'MSC' in sys.version else [])

which is definitely wrong because sys.version tells which C compiler is used to compile the python.exe, but can not tell whether python.exe runs in MSVC or mingw circumstance

IndexError: list index out of range

This line explodes very frequently, without useful debug information:

  File "/home/juan/.local/lib/python3.9/site-packages/nimporter.py", line 48, in __init__
    nim_module = nim_module.splitlines()[-1]
IndexError: list index out of range

Full traceback

>>> import faster_than_walk
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/juan/.local/lib/python3.9/site-packages/faster_than_walk/__init__.py", line 7, in <module>
    from . faster_than_walk import *
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 982, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 925, in _find_spec
  File "/home/juan/.local/lib/python3.9/site-packages/nimporter.py", line 1150, in find_spec
    return Nimporter.import_nim_code(fullname, path, library=False)
  File "/home/juan/.local/lib/python3.9/site-packages/nimporter.py", line 828, in import_nim_code
    NimCompiler.compile_nim_code(
  File "/home/juan/.local/lib/python3.9/site-packages/nimporter.py", line 588, in compile_nim_code
    raise NimCompileException(errors[0])
  File "/home/juan/.local/lib/python3.9/site-packages/nimporter.py", line 48, in __init__
    nim_module = nim_module.splitlines()[-1]
IndexError: list index out of range

>>> exit()

$ ls /home/juan/.local/lib/python3.9/site-packages/faster_than_walk/
faster_than_walk.nim  faster_than_walk.nim.cfg  __init__.py

$ head /home/juan/.local/lib/python3.9/site-packages/faster_than_walk/faster_than_walk.nim
import os, strutils, nimpy


proc walk*(folderpath: string, extensions: seq[string] = @[""], followlinks : bool = false, yieldfiles: bool = true, debugs: bool = false, check_folders: bool = false): seq[string] {.exportpy.} =
  ## Faster os.walk(), followlinks follows SymLinks, yieldfiles yields files else folders, return 1 list of strings.
  let extused {.noalias.} = create(bool, sizeOf(bool))  # Optimization.
  extused[] = extensions != @[""] and extensions.len > 0
  for item in walkDirRec(folderpath, {if yieldfiles: pcFile else: pcDir}, {if followlinks: pcLinkToDir else: pcDir}, checkDir=check_folders):
    if unlikely(debugs): echo item
    if unlikely(extused[]):

$

• https://github.com/Pebaz/nimporter/blob/master/nimporter.py#L49

It is possible to add some defensive programing just on that line ?,
something like a try except and print a useful message,
manually checking like
if not len(nim_module) > 0: print("ERROR: File not found, Nimporter can not find the nim files") or something.

Wrong link

nimporter/setup.py

Line 10 in 2cd3d2e

as the [Nimpy](https://github.com/yglukhov/nimporter) library.

nimporter ignores nim.cfg

Hi,

I'm totally new to Nim and nimporter and tried to build a little test application. I changed Nim to use clang, which is compatible with CPython on Windows 10. Nim file:

import nimpy
import math, random
import times

let time = cpuTime()

proc mcpi(nthrows: float): float {.exportpy.} =
  randomize()
  var inside = 0.0
  for i in 1..int64(nthrows):
    if hypot(rand(1.0), rand(1.0)) < 1:
      inside += 1
  result = 4 * inside / nthrows
 
echo mcpi(250000000)

echo cpuTIme()-time  

Python file:

import nimporter, mcpi
import time

time0 = time.time()
print("pi:",mcpi.mcpi(250000000))
print("tm:",time.time()-time0)

Running the nim-file using Nim works perfectly. Compiling to library (nimpy only) and importing to Python also works perfectly.

Running the Python program "hangs" the execution and while trying to start Visual C++. Both files are in the same directory, i.e. there is no directory structure.

What is wrong here?

Add `nimporter compile` CLI command

Add a CLI command that will simplify the distribution of applications using Nimporter by lifting the requirement of using a setup.py for packaging.

Todo

• nimporter compile CLI command
• Run nimporter_cli.clean() before building all modules/libraries
• Add documentation to README explaining use case behind the new command
• Add a hello-world example to examples/ for using Nimporter with Docker

package_dir setting in setup is not respected

I have

package_dir={'mypackagename': 'src'},

in my setup.py

But nim compiled (.so) files are installed under .../site-packages/src instead of .../site-packages/mypackagename

Install works as editable but not regular

I'm in a different directory as my main library in a different virtualenv. It works when I do a pip install -e install but not a regular (no -e flag) install:

~/D/P/R/coronavirus_genome_phylogeny ❯❯❯ pip install ../librtd/librtd-py
Processing /Users/BenjaminLee/Desktop/Python/Research/librtd/librtd-py
Processing /Users/BenjaminLee/Library/Caches/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e/docopt-0.6.2-py2.py3-none-any.whl
Building wheels for collected packages: librtd
  Building wheel for librtd (setup.py) ... done
  Created wheel for librtd: filename=librtd-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whl size=141098 sha256=549d98793215bc354b3faf5eb7a37b10ac530be44e11da67a6b0d5315b978dab
  Stored in directory: /Users/BenjaminLee/Library/Caches/pip/wheels/29/7f/c4/4fba79502ea438c55740cfddce1b8f317ce3078f8913bae1fc
Successfully built librtd
Installing collected packages: docopt, librtd
Successfully installed docopt-0.6.2 librtd-0.0.1
WARNING: You are using pip version 20.1; however, version 20.2 is available.
You should consider upgrading via the '/Users/BenjaminLee/.virtualenvs/corona/bin/python3.7 -m pip install --upgrade pip' command.
~/D/P/R/coronavirus_genome_phylogeny ❯❯❯ rtd
Traceback (most recent call last):
  File "/Users/BenjaminLee/.virtualenvs/corona/bin/rtd", line 5, in <module>
    from cli_wrapper import cli_wrapper
ModuleNotFoundError: No module named 'cli_wrapper'
~/D/P/R/coronavirus_genome_phylogeny ❯❯❯ pip install ../librtd/librtd-py^C
~/D/P/R/coronavirus_genome_phylogeny ❯❯❯ python
Python 3.7.7 (default, Mar 18 2020, 19:38:56)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import librtd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'librtd'
>>>
KeyboardInterrupt
>>>
~/D/P/R/coronavirus_genome_phylogeny ❯❯❯ pip uninstall ../librtd/librtd-py
ERROR: You must give at least one requirement to uninstall (see "pip help uninstall")
~/D/P/R/coronavirus_genome_phylogeny ❯❯❯ pip uninstall librtd
Found existing installation: librtd 0.0.1
Uninstalling librtd-0.0.1:
  Would remove:
    /Users/BenjaminLee/.virtualenvs/corona/bin/rtd
    /Users/BenjaminLee/.virtualenvs/corona/lib/python3.7/site-packages/cli.cpython-37m-darwin.so
    /Users/BenjaminLee/.virtualenvs/corona/lib/python3.7/site-packages/librtd-0.0.1.dist-info/*
    /Users/BenjaminLee/.virtualenvs/corona/lib/python3.7/site-packages/librtdpy.cpython-37m-darwin.so
Proceed (y/n)? y
  Successfully uninstalled librtd-0.0.1
~/D/P/R/coronavirus_genome_phylogeny ❯❯❯ pip install -e ../librtd/librtd-py
Obtaining file:///Users/BenjaminLee/Desktop/Python/Research/librtd/librtd-py
Requirement already satisfied: docopt in /Users/BenjaminLee/.virtualenvs/corona/lib/python3.7/site-packages (from librtd==0.0.1) (0.6.2)
Installing collected packages: librtd
  Running setup.py develop for librtd
Successfully installed librtd
WARNING: You are using pip version 20.1; however, version 20.2 is available.
You should consider upgrading via the '/Users/BenjaminLee/.virtualenvs/corona/bin/python3.7 -m pip install --upgrade pip' command.
~/D/P/R/coronavirus_genome_phylogeny ❯❯❯ rtd
Usage:
rtd <k> <input> [<output>] [--reverse-complement|--pairwise]
rtd (-h | --help)
rtd --version
~/D/P/R/coronavirus_genome_phylogeny ❯❯❯

Any idea why that would be?

Does not work with methods

import nimpy


type
  My* = ref object of RootObj
    xx: string


proc myHelp*(my: My): string {.exportpy.} =
  return "mine"

  
method mmyHelp*(my: My): string {.base,exportpy.} =
  return "mine"

ttt.nim(13, 1) Error: Expected one of {nnkProcDef, nnkIteratorDef, nnkFuncDef}, got nnkMethodDef

Add nimporter commands to build binary and source distributions

Even though it should not be very difficult to generate a binary or source distribution by following the nimoorter README, it would be nice if the nimporter command line tool included a command to do it for you. IMHO this would make the tool make the whole process fully “idiot proof”.

Basically it would look for a setup.py file and create it if it didn’t exist. It would check if the necessary code is there already and if not it would add it. Finally it would make the appropriate call to “python setup.py”.

This

is the best thing I've ever seen. You win the Nimternet

Why use --gc:markAndSweep as default?

Is there a reason to use this over refc for Nimporter? Nimpy suggested compiler invocation makes no mention of it:

# Compile on Windows:
nim c --threads:on --app:lib --out:mymodule.pyd mymodule
# Compile on everything else:
nim c --threads:on --app:lib --out:mymodule.so mymodule

Additionally, microbenchmarking suggests is it is slower than refc (nimr is just an alias for nim c -r --hints:off --verbosity:0):

~/D/P/R/l/librtd-py ❯❯❯ nimr -d:danger --opt:speed --gc:markAndSweep --threads:on call_cli.nim; rm test.jsonl
Success: Analyzed RTD for 50 sequences totaling 1492590 bp in 19.9 seconds.
~/D/P/R/l/librtd-py ❯❯❯ nimr -d:danger --opt:speed --gc:refc --threads:on call_cli.nim; rm test.jsonl
Success: Analyzed RTD for 50 sequences totaling 1492590 bp in 15.1 seconds.
~/D/P/R/l/librtd-py ❯❯❯ nimr -d:danger --opt:speed --gc:arc --threads:on call_cli.nim; rm test.jsonl
Success: Analyzed RTD for 50 sequences totaling 1492590 bp in 12.5 seconds.

exclude_dirs, cryptic error when exclude_dirs specified as a set

While exclude_dirs supplied to build_nim_extensions() clearly should be a list, it would seem excusable for a user to specify it as a set in the light that _find_extensions() does:

def _find_extensions(cls, path, exclude_dirs=set()):  

Unfortunately, one then gets the following cryptic error:
Traceback (most recent call last):
File "setup.py", line 9, in
modules = nimporter.build_nim_extensions(exclude_dirs={})
File "/home/phdyex/.local/lib/python3.8/site-packages/nimporter.py", line 1081, in build_nim_extensions
ext = cls._build_nim_extension(extension, root)
File "/home/phdyex/.local/lib/python3.8/site-packages/nimporter.py", line 979, in _build_nim_extension
return NimCompiler.compile_nim_extension(
File "/home/phdyex/.local/lib/python3.8/site-packages/nimporter.py", line 465, in compile_nim_extension
raise NimCompileException(errors[0])
nimporter.NimCompileException: type mismatch: got <int literal(5), int literal(5)>
| if exp[i].kind in nnkCallKinds + {nnkDotExpr, nnkBracketExpr}:
| let callVar = newIdentNode(":c" & $counter)
-> result.assigns.add getAst(asgn(callVar, paramAst))
| result.check[i] = callVar
| result.printOuts.add getAst(print(argStr, callVar))

At /home/phdyex/.choosenim/toolchains/nim-1.2.0/lib/pure/unittest.nim 650:43

I will create a PR to catch this scenario and raise an informative exception.

Cleanup Sheer Amount of Options To Set Nim Compile Switches

This would represent a breaking change and would require a major version bump.

One of the main things that needs to happen is to have hopefully 1 place to set Nim compile args. This should probably be in the form of directly setting nimporter.NimCompiler.NIM_CLI_ARGS but any better ideas are definitely appreciated.

Here are the ways that compile options can be configured:

• Directly setting nimporter.NimCompiler.NIM_CLI_ARGS
• Using *.nim.cfg
• Using *.nims
• Using switches.py (deprecated, needs to be officially removed)
• Using the danger=True flag in build_nim_extensions()

These use cases need to be kept in mind with any new designs:

• Local import during development
• Precompiling for Docker
• Source distributions that require Nim
• Binary distributions that only require Nimporter

Some random todo items would also be nice during this major update:

• Update test coverage numbers
• Remove switches.py support
• Add clear documentation on the difference and use cases of an extension library and an extension module
• Refactor the codebase to be a bit flatter (bunch of static classes are acting basically like namespaces, module namespaces could be used instead)

Warning ProveInit

When using Nimporter it complains:

~/.nimble/pkgs/nimpy-0.1.1/nimpy.nim(1053, 32) Warning: Cannot prove that 'arg0timeout' is initialized. This will become a compile time error in the future. [ProveInit]

• yglukhov/nimpy#163

Maybe Nimporter can pass the argument --warning[ProveInit]:off ?.

Python users probably do not know what it means anyway. 🤷

Nimporter fails to import due to undefined symbol "__builtin_ssubll_overflow" when using GCC on CentOS 7.5

So, I got this really ugly stack trace when trying to import the basic nim_math.nim example code into my python project. The relevant sections of the project tree look like:

root/
    nim/
        nim_math.nim
        __pycache__/
            nim_math.nim.hash
            nim_math.so
    src/
        app.py

And the relevant contents of app.py are:

...
import nimporter
import nim.nim_math
...

Here's some more data about the environment that isn't available in the stack trace: I'm using a CentOS 7.5 base docker image with GCC version 4.8.5. I've also replicated this same failure with the same docker image on GCC versions 7.x and 8.x.

/pyenv/versions/3.8.0/envs/my-new-project/lib/python3.8/site-packages/nimporter.py:1117: in find_spec
    return Nimporter.import_nim_code(fullname, path, library=False)
/pyenv/versions/3.8.0/envs/my-new-project/lib/python3.8/site-packages/nimporter.py:819: in import_nim_code
    cls.__validate_spec(spec)
/pyenv/versions/3.8.0/envs/my-new-project/lib/python3.8/site-packages/nimporter.py:887: in __validate_spec
    raise NimporterException(error_message) from import_error
E   nimporter.NimporterException: Error importing /app/nim/__pycache__/nim_math.so
E   Error Message:
E
E       /app/nim/__pycache__/nim_math.so: undefined symbol: __builtin_ssubll_overflow
E
E   Python Version:
E
E       3.8.0 (default, Dec 16 2019, 22:06:06) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
E
E   Nim Version:
E
E       Nim Compiler Version 1.2.0 [Linux: amd64]
E
E   CC Version:
E
E       <Error getting version>
E
E   Installed CCs:
E
E       {'gcc': PosixPath('/usr/bin/gcc')}
E
E   Please help improve Nimporter by opening a bug report at: https://github.com/Pebaz/nimporter/issues/new and submit the above information along with your description of the issue.

I'm not sure if the issue is with nimporter, nimpy, the nim compiler, or gcc itself. It looks like this is an issue with the compiled output, so nimporter might be off the hook here. If that's the case, I'll have to file this same issue one layer deeper down the stack - which I guess would involve pestering the folks who develop nimpy.

I'm hoping this is an easy fix though - maybe there's some compiler switch I'm missing that could make this all go away. Thanks for the help (and the awesome project!)

nimporter can't find file on Windows 10

Here is a working example to reproduce the problem:

C:\Users\dave.sisk\math_speedup_example>type pmath.py
def fib(n):
if n == 0:
return 0
elif n < 3:
return 1
return fib(n - 1) + fib(n - 2)

C:\Users\dave.sisk\math_speedup_example>type nmath.nim
import nimpy

proc fib(n: int): int {.exportpy.} =
if n == 0:
return 0
elif n < 3:
return 1
return fib(n-1) + fib(n-2)

C:\Users\dave.sisk\math_speedup_example>type main.py
import nimporter
from time import perf_counter
import nmath # Nim imports!
import pmath
print('Measuring Python...')
start_py = perf_counter()
for i in range(0, 40):
print(pmath.fib(i), end=" ")
end_py = perf_counter()

print('\n\nMeasuring Nim...')
start_nim = perf_counter()
for i in range(0, 40):
print(nmath.fib(i), end=" ")
end_nim = perf_counter()

print('\n---------')
print('Python Elapsed: {:.2f}'.format(end_py - start_py))
print('Nim Elapsed: {:.2f}'.format(end_nim - start_nim))

C:\Users\dave.sisk\math_speedup_example>python main.py
Traceback (most recent call last):
File "C:\Users\dave.sisk\math_speedup_example\main.py", line 3, in
import nmath # Nim imports!
File "", line 1027, in _find_and_load
File "", line 1002, in _find_and_load_unlocked
File "", line 945, in _find_spec
File "C:\Users\dave.sisk\AppData\Local\Programs\Python\Python310\lib\site-packages\nimporter.py", line 1269, in find_spec
return Nimporter.import_nim_code(fullname, path, library=False)
File "C:\Users\dave.sisk\AppData\Local\Programs\Python\Python310\lib\site-packages\nimporter.py", line 944, in import_nim_code
NimCompiler.compile_nim_code(
File "C:\Users\dave.sisk\AppData\Local\Programs\Python\Python310\lib\site-packages\nimporter.py", line 705, in compile_nim_code
cls.check_compile_errors(errors, library, nim_args, output, tmp_cwd, warnings)
File "C:\Users\dave.sisk\AppData\Local\Programs\Python\Python310\lib\site-packages\nimporter.py", line 492, in check_compile_errors
raise NimCompileException(errors[0])
nimporter.NimCompileException: Error: unhandled exception: The system cannot find the file specified.

Issue running example

I was trying to follow along with the tutorial but ran into the follow error

[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import nimporter
>>> import nim_marh
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'nim_marh'
>>> import nim_math
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 894, in _find_spec
  File "/usr/local/lib/python3.6/dist-packages/nimporter.py", line 234, in find_spec
    NimCompiler.hash_changed(module_path),
  File "/usr/local/lib/python3.6/dist-packages/nimporter.py", line 119, in hash_changed
    return cls.get_hash(module_path) != NimCompiler.hash_file(module_path)
  File "/usr/local/lib/python3.6/dist-packages/nimporter.py", line 126, in hash_file
    return importlib.util.source_hash(module_path.read_bytes())
AttributeError: module 'importlib.util' has no attribute 'source_hash'

Seems to be related to https://stackoverflow.com/questions/39660934/error-when-using-importlib-util-to-check-for-library/39661116

Tag v1.0.2 on GitHub

Thanks for the release on PyPI! Can you also tag the release on GitHub?

Multithreading Issue

It seems that when trying to use a Nimpy library from multiple Python threads, an error crashes the Python interpreter:

SIGSEGV: Illegal storage access. (Attempt to read from nil?)

This error originates from compiled Nim code. The information I have on it so far:

• If a Nim extension is imported in the global scope, it can only be used from the main thread
• If an extension has been imported into the global scope, you cannot use it from another Python thread
• If an extension is imported into a running function's scope, it will still be bound to sys.modules so when another thread tries to access it it will crash with the above error
• When using an extension in a non-main thread only, it will work (since it is only one thread)
• When using a Nim extension with Flask within a route handler, it will crash with the above error since each handler is executed in its own thread. Even handling subsequent requests after the initial one will crash even though it is on the same thread ID
• I tried duplicating a Nim extension .so per thread, creating a unique Spec and Loader object and it successfully imports, but for some reason it still crashes likely due to the underlying C extension exposing the same module name (PyInit_<mod name>)
• The only solution I can come up with is to compile a separate Nim extension with a unique name per thread that should be used. This is just straight up hacky, however. A better solution probably exists.

Use this Nim module as a reference for the next 2 Python examples:

# calc.nim

import nimpy

proc add(a, b: int): int {.exportpy.} =
    return a + b

Here is a code example that you can use the reproduce the problem:

import threading, nimporter

def foo():
    import calc
    print(calc.add(2, 20))

threading.Thread(target=foo).start()
threading.Thread(target=foo).start()

print('Here!')

Another example:

from flask import Flask
import nimporter

app = Flask(__name__)

@app.route('/')
def hello_world():
    # Importing here instead of in global scope works for 1 request
    import calc
    return 'Hello ' + str(calc.add(2, 20))

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

Unfortunately, it should be noted that this issue is beyond my current (2020) skillset so any assistance is appreciated if it will contribute towards a resolution.

Basic example fails on windows with "ImportError: DLL load failed: %1 is not a valid Win32 application"

I just tried to follow the example on the readme file (creating a simple nim_math.nim file and a python nimporter_test.py file to use nim_math). I got the following error on my Windows computer:

python nimporter_test.py
Traceback (most recent call last):
File "nimporter_test.py", line 2, in
import nimporter, nim_math
ImportError: DLL load failed: %1 is not a valid Win32 application.

I'm using python 3.7.1 and nim 1.2.0. I installed nimporter using "pip install nimporter". I installed nimpy using "nimble install nimpy".

Cannot compile even if GCC is found

Hi,
I'm trying to use nimporter in Windows10 and I'm getting the following error:

Traceback (most recent call last):
  File "c:\users\thiba\appdata\local\programs\python\python38\lib\site-packages\nimporter.py", line 979, in __validate_spec
    util.module_from_spec(spec)
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1101, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed while importing engine: %1 is not a valid Win32 application.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\thiba\AppData\Local\Programs\Python\Python38\Scripts\pyxalchemy-script.py", line 11, in <module>
    load_entry_point('pyxalchemy', 'console_scripts', 'pyxalchemy')()
  File "c:\users\thiba\appdata\local\programs\python\python38\lib\site-packages\pkg_resources\__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "c:\users\thiba\appdata\local\programs\python\python38\lib\site-packages\pkg_resources\__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "c:\users\thiba\appdata\local\programs\python\python38\lib\site-packages\pkg_resources\__init__.py", line 2443, in load
    return self.resolve()
  File "c:\users\thiba\appdata\local\programs\python\python38\lib\site-packages\pkg_resources\__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "d:\src\python_projects\pyxalchemy\pyxalchemy\game.py", line 12, in <module>
    from . import engine
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 914, in _find_spec
  File "c:\users\thiba\appdata\local\programs\python\python38\lib\site-packages\nimporter.py", line 1269, in find_spec
    return Nimporter.import_nim_code(fullname, path, library=False)
  File "c:\users\thiba\appdata\local\programs\python\python38\lib\site-packages\nimporter.py", line 957, in import_nim_code
    cls.__validate_spec(spec)
  File "c:\users\thiba\appdata\local\programs\python\python38\lib\site-packages\nimporter.py", line 1025, in __validate_spec
    raise NimporterException(error_message) from import_error
nimporter.NimporterException: Error importing D:\src\Python_Projects\Pyxalchemy\pyxalchemy\__pycache__\engine.pyd
Error Message:

    DLL load failed while importing engine: %1 is not a valid Win32 application.

Python Version:

    3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)]

Nim Version:

    Nim Compiler Version 1.6.4 [Windows: i386]

CC Version:

    <No compatible C compiler installed>

Installed CCs:

    {'gcc': WindowsPath('C:/tools/mingw-w64/mingw32/bin/gcc.EXE')}

Please help improve Nimporter by opening a bug report at: https://github.com/Pebaz/nimporter/issues/new and submit the above information along with your description of the issue.

Problems implementing nimporter

• I am trying to migrate https://github.com/juancarlospaco/faster-than-requests
• The project is older than Nimporter and I didnt have the time before. I need to build for Windows, Mac, Linux.
• I have made a PR were I deleted everything and only left the basic files, the .py and .nim files.

I have several errors but I dont know whats the problem, they all just exit on the GitHub Actions CI with:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
IndexError: list index out of range

It does not fully finish correctly, always fails on 1 of the 3 operating systems.
I guess something is failing to build the files, but the error message is confusing.
Maybe can you take a quick look at the branch and help me with the errors?:

• https://github.com/juancarlospaco/faster-than-requests/tree/next
• juancarlospaco/faster-than-requests#133
• https://github.com/juancarlospaco/faster-than-requests/blob/next/setup.py
• https://github.com/juancarlospaco/faster-than-requests/blob/next/.github/workflows/build.yml

🙂

What's the expected "workflow"?

Hello! New to Nim, not-so-new to Python, but trying to figure out how to use this tool to get the best of both.

I setup the following repo to mess around with some stuff based on what I'm reading in the README: https://github.com/macintacos/nimport-playground

The project is pretty "rough", I would say. I'm not really understanding what the expected "workflow" is supposed to be. Right now I:

• Create a virtualenv and activate it. (python -m venv venv && source venv/bin/activate.fish)
• Call python setup.py install twice, because the first time nimporter isn't installed and yet it seemingly needs to be used in the call to setup() (?)
• Now I can edit the Python code and call python -m play.main as I see fit.

However, if I want to edit the Nim code, it seems like I need to recompile it completely with nimporter compile and then python setup.py install again. At first I thought running python setup.py install again would work (after changing some of the Nim code), but that doesn't seem to do anything.

Is this the expected workflow? Or am I missing something about my setup? Sorry if I'm missing something obvious 😬

Basic example does not work on Windows

I tried running the readme example code on Windows and it fails as soon as you import an external nim modele. I do not get any error messages, python just seems to quietly die. It does work on macos.

The example, copied verbatim from the readme.md is as follows:

nim_math.nim:

import nimpy

proc add(a: int, b: int): int {.exportpy.} =
    return a + b

nimporter_test.py:

import nimporter, nim_math

print(nim_math.add(2, 4))  # 6

In order to check that the nim code worked, I added a "when is_main_module" clause:

import nimpy

proc add(a: int, b: int): int {.exportpy.} =
    return a + b

when is_main_module:
    echo add(2, 3)

When I run this directly with nim it prints 5 as expected.