Python packages using Nim

This blog post describes a way to create Python packages using Nim.

Build example Nim module

We start by implementing a simple compiled Python module using nimpy. First you must Install Nim then install nimpy with nimble install nimpy.

Create file named mymodule.nim . Note, this filename will match the module name that you are going to import from Python.

import nimpy

proc greet(name: string): string {.exportpy.} =
  return "Hello, " & name & "!"

Then compile the module.

# Compile on Windows:
nim c --threads:on --app:lib --out:mymodule.pyd mymodule
# Compile on everything else:
nim c --threads:on --app:lib --out:mymodule.so mymodule

Try calling this Nim module

We can now try this module using the following Python example. Copy this text to a file called try.py and then run with python try.py.

import mymodule
print(mymodule.greet("World"))

Build Python package

Now we want to package this module, for publishing to PyPI. First, create the directory structure required for the module publishing project.

  • example_nim_pkg/
    • __init__.py is simply a blank file.
    • mymodule.nim
    • mymodule.so or mymodule.pyd depending on platform.
  • LICENSE
  • README.md
  • setup.py

The content of the LICENSE and README.md are up to you. Example content for setup.py is as follows:

import setuptools
from setuptools import Extension

with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="example-nim-pkg-stever",
    version="0.0.1",
    author="Steven Robertson",
    author_email="stever@hey.com",
    description="A small example package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/pypa/sampleproject",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    package_data={
    	"": ["*.so", "*.pyd"]
    },
    include_package_data=True,
    zip_safe=False,
    ext_modules = [
        Extension(
            name = 'dummy',
            sources = ['dummy.c']
        )
    ]
)

Change the highlighted lines and replace with your own details. The package name should include your own PyPI username.

At this point we will create and activate a virtualenv environment, install the required packages for publishing, and then build the package.

Prepare your virtualenv environment:

# pip install virtualenv
virtualenv venv
source ./venv/bin/activate # on Linux and macOS etc.
# On Windows activate with 'venv\Scripts\activate'

Build the package:

python setup.py sdist bdist_wheel

The above command will require a build environment. On Windows this will probably require Visual Studio with C build tools. The reason for this is that we create a dummy C extension in the process to create a platform specific wheel. This is important because the Nim extension is built for a single platform, and using this technique we can create and include wheels for other platforms.

Publish the Python package

It must be noted here that uploading a wheel for Windows and uploading this to PyPI will only work for Windows. You can use Azure Artifacts to publish packages that include an additional wheel built for Linux doing what is described here without any additional steps. Otherwise, packages including wheels for Linux may need to use manylinux, which is out of the scope of this article. That can done using a Docker image to prepare the Linux package. This might be covered in a later post here. I initially used Azure Artifacts, then wrote these instructions with PyPI as the example package host.

Upload the package

We use twine to upload the package:

pip install twine
twine upload dist/*

The last command will prompt you for your PyPI username & password. Following this your package will be published to the public index.

Including wheels for multiple platforms

Generating the package using setup.py on each platform creates a platform-specific wheel in the dist/ sub-folder. You can copy these wheels into the same sub-folder from the platform from which you publish the package. This should then include all wheels for each of the platforms.

Using the published package

Now, we will create a new shell window and a new virtualenv environment to install this package and try it out.

# activate virtualenv as shown above
pip install example-nim-pkg-stever

The following Python file can be used to call out to the Nim module:

from example_nim_pkg import nimgreet
print(nimgreet.greet("World"))

Conclusion

This article demonstrates a way to create multi-platform packages using Nim modules. This seems like a useful way to package Nim modules and consume them in Python scripts.

Source-code for this project is available here.

Rich is a Python library for rich text

https://github.com/willmcgugan/rich

This is a screenshot of the output in Windows Terminal (PowerShell)

Output in screenshot above from the following commands, where we have Python and virtualenv previously installed.

virtualenv venv
.\venv\Scripts\activate
pip install rich
python -m rich

Very nice. I will use this in Python scripts.

Neat, this also works in Jupyter Notebooks. Read more about it here.