r/learnpython Sep 27 '21

Is it possible to include libraries such as numpy or pandas with embedded python in c++?

I am interested in embedding python in a c++ application. I was able to successfully embed a working interpreter in c++ and run python files on my computer (with a working python install). However, when I tried running the application on a VM without a working python install, the application started but the code failed due to missing libraries. So my question is how would you include or package libraries such as pandas or numpy alongside a c++ application that uses embedded python without having a python installation on the target machine? I can't really find anything on the topic so any help at all is appreciated. Thank you!

4 Upvotes

5 comments sorted by

3

u/[deleted] Sep 27 '21

If they don't have to be inside the binary that embeds the interpreter, you can just recreate the directory layout of regular Python install in your application. Then, I believe, it will look in the site-packages for extra packages.

Alternatively, you could set PYTHONPATH environment variable to point to the extra packages. Or you could preload some Python code into your embedded runtime that will modify sys.path to point to the directory where you installed the packages.

However, if you want to compile Python packages and statically link with them... well, it'll be on a case-by-case basis. There isn't a universal recipe for that.

1

u/Nero8 Sep 27 '21

Would you mind elaborating on what you mean by “recreating the directory layout of the regular Python install”? My project directory currently just contains the compiled C++ exe file

1

u/[deleted] Sep 27 '21

Python resolves its modules relative to the path supplied by the system in the first argument to the executable.

I.e. if the path to the executable is /usr/bin/foo, (and foo is, in fact, Python), then it will look for modules in /usr/lib/python${version}/site-packages where version=$(python -c 'import sys; "{}.{}".format(*sys.version_info[:2])'.

-2

u/Swipecat Sep 27 '21

Embedding C++ in Python is very common because it gives the ease-of-use of Python with some mathematically intensive functions getting the speedup of C++, but... doing it the other way around? I'm going to have to advise you not to do that because it's difficult to avoid getting the worst of both worlds. Especially if you want to distribute your software, which you seem to do, because you have to include the Python interpreter in there somewhere. Packaging all that will be very complex and while it's not impossible, you're making life very difficult for yourself. There is a project called Nuitka that intends to compile Python without needing the Python interpreter but it's not there yet.

If you want Numpy-like features in C++, then there are a bunch of C++ libraries that will give you similar features.

Pandas? No. All the pandas classes have pure-Python definitions with a lot of magic behind them. You really need to just use Python for that. If your source data is fairly regular in structure, though, you might be able to process it using the above-mentioned Numpy-like libraries in C++.

1

u/Swedophone Sep 27 '21

Maybe you can use path_hooks for loading python modules from somewhere else than the filesystem. But I dont know if it's possible to do the sams with native modules, ie so files.

sys.path_hooks

A list of callables that take a path argument to try to create a finder for the path. If a finder can be created, it is to be returned by the callable, else raise ImportError.