Boa is the Python Bridge Layer in Pipcook, it lets you call Python functions seamlessly in Node.js, it delivers any Python module for Node.js developer in lower-cost to learn or use.
Install Boa from npm:
$ npm install @pipcook/boaLet's have a glance on how to call to Python's function:
const boa = require('@pipcook/boa');
const os = boa.import('os');
console.log(os.getpid()); // prints the pid from python.
// using keyword arguments namely `kwargs`
os.makedirs('..', boa.kwargs({
mode: 0x777,
exist_ok: false,
}));
// using bult-in functions
const { range, len } = boa.builtins();
const list = range(0, 10); // create a range array
console.log(len(list)); // 10
console.log(list[2]); // 2By default, Boa will install a conda virtual environment under the path of the Boa package. To make it easier to install python libraries, you can run:
$ ./node_modules/.bin/bip install <package-name>
bipis an alias of pip that points to the correct Python environment.
For Python beginners, writing JavaScript code based on Boa can be quite challenging, as we may not always be familiar with Python's ecosystem. Therefore, this project has prepared a prompt powered by ChatGPT. With it, developers can simply send the corresponding Python source code to ChatGPT and receive the corresponding Boa code, along with related explanations.
Now we only have Chinese and English versions prepared:
(中文版本提示词)
你作为一个 Boa 代码转换器用于将 Python 的代码转换为依赖 Boa 的 JavaScript 代码，在输出的代码中首先会依赖 @pipcook/boa，然后将 Python import 语法转换为 boa.import(x)，x 即 Python 中的包名，接下来的转换规则如下：
1. 遇到 keyword arguments，使用 boa.kwargs(map) 包装，map 中的 key 表示 Python 函数定义的 key，map 中的 value 即参数值
2. Python 中的函数调用可以直接转换为 JavaScript 的同名函数调用
3. Python 中的内置对象通过 boa.builtins() 获取
如果明白以上规则，请回复我明白了。
(English prompt)
As a Boa code converter, you are responsible for converting Python code into JavaScript code that depends on Boa. In the output code, you will first rely on "@pipcook/boa". Then, Python's import syntax is converted to "boa.import(x)", where "x" represents the package name in Python. The following conversion rules apply:
When encountering keyword arguments, wrap them with "boa.kwargs(map)", where "map" represents the key-value pairs of the arguments.
Python function calls can be directly converted to JavaScript function calls with the same name.
Python built-in objects can be accessed through "boa.builtins()".
If you understand the above rules, please reply with "I understand".
An example in Chinese:
An example in English:
A Connection between 2 languages(ecosystems) has huge works to be done, even though this package is working only on the unilateral from Python to JavaScript. The most difficult part is that for developers, they need to understand the correspondence between the two languages and ecosystems. Therefore, a good design principle will make developers reduce learning costs.
require('@pipcook/boa') returns the root object, which will be your entry point to all Python functions, and it provides these methods:
Gets the Python's built-in functions, for example:
const { len, range } = boa.builtins();
len([1, 2, 3]); // 3
len(range(0, 10)); // 10Imports a Python module in your current environment, the module includes:
- system modules like
os,stringandre. - third-party modules like
numpyandrequestvia pip.
To call the function, you should pass a mod argument for the module that you want to import.
const os = boa.import('os');
const str = boa.import('string');
const numpy = boa.import('numpy');This returns an instance of PythonObjectWrapper or a JavaScript primitive value for some special cases.
Creates a Python's keyword arguments, Python provides a way to map arguments with names:
fs.open('./a-file-to-open', mode=0e777)Correspondingly, this function is used to represent a keyword arguments, and the specific usage is very easy to understand:
const fs = boa.import('fs');
fs.open('./a-file-to-open', boa.kwargs({ mode: 0e777 }));It's equivalent to the with-statement in Python, this would be called with an object ctx that supports the context management protocol (that is, has __enter__() and __exit__() methods). And 2nd fn is corresponding to the execution block, A simple example is as follows:
boa.with(localcontext(), (ctx) => {
// execution
// the ctx is localcontext().__enter().
});Execute Python expression in the context specified, here is a simple call:
boa.eval('len([10, 20])');
// 2Alternatively, developers can use tagged template literal to pass variables that have been defined in JavaScript:
const elem = np.array([[1, 2, 3], [4, 5, 6]], np.int32);
boa.eval`${elem} + 100`; // do matrix + operation
boa.eval`len(${elem})`; // equivalent to `len(elem)`For multi-line code, Python 3 does not provide a mechanism to return a value, so the eval function can only handle single-line Python expressions.
A shortcut to create a Python's bytes literal from JavaScript string, it's equivalent to b'foobar' in Python.
const { bytes } = boa;
bytes('foobar'); // "b'foobar'"The bytes(str) function simply creates a plain object that is used to pass a string to a Python function as a bytes literal, but does not correspond to any Python object itself. Alternatively, you could use Python's builtin class bytes for creating a real object:
const { bytes } = boa.builtins();
const foobar = Buffer.from('foobar');
bytes.fromhex(foobar.toString('hex'));
// "b'foobar'"This class represents a wrapper for the corresponding object in Python runtime, it must be returned only from boa methods like boa.builtins and boa.import.
In order for developers to use Python objects seamlessly, creating a PythonObjectWrapper requires some necessary steps.
First, check the type of the Python object under instance. If it is one of the following types, it will be converted to the corresponding primitive type.
| python type | primitive |
|---|---|
int,float |
number |
int64 |
bigint |
float64 |
bigdecimal |
bool |
boolean |
str |
string |
NoneType |
null |
If the type of the object that needs to be wrapped is not in the above primitive, a temporary object will be created, and methods and properties will be defined through Object.defineProperties.
On an instance of PythonObjectWrapper, developers can directly obtain values through the property way, just like using those in Python. This is because we use ES6 Proxy, so the last step, we created a Proxy Object, configured with 3 trap handlers, get, set, apply, and finally returns this proxy object.
At Python language, an object has attr and item accessors, and they use different expressions:
x.yis for attr accessorm[n]is for item accessor
Unfortunately, ES6 Proxy does not distinguish the above things. Therefore, it's needed to define an algorithm to confirm their priority in a time of operation.
- given a
namevariable which is passed by ES6 Proxy'sgethandler. - check the
nameproperty is owned by the JavaScript object via.hasOwnProperty().- return the property if it's truthy.
- check the
nameproperty is owned by the object's class via.constructor.prototype.hasOwnProperty().- return the property if it's truthy.
- check if
nameis a numeric representation.- if it's truthy, call the internal method
.__getitem__(i)for item accessor. - otherwise
- try to access the attr via the internal method
.__getattr__().- if no exceptions, create the new instance from the value and return.
- try to access the item via the internal method
.__getitem__().- if no exceptions, create the new instance from the value and return.
- otherwise, return
undefined.
- try to access the attr via the internal method
- if it's truthy, call the internal method
To better understand the algorithm above, let's look at some examples:
const boa = require('@pipcook/boa');
const { abs, tuple } = boa.builtins();
{
console.log(abs(-100)); // 100
console.log(abs(100)); // 100
}
{
const re = boa.import('re');
const m = re.search('(?<=abc)def', 'abcdef');
console.log(m.group(0)); // 'def'
}
{
// make sure the `numpy` is in your current python env.
const np = boa.import('numpy');
const x0 = np.array([[1, 2, 3], [4, 5, 6]], np.int32);
const x1 = np.arange(15).reshape(3, 5);
const x2 = np.zeros(tuple([3, 4]));
}As mentioned above, in addition to dynamically obtaining objects from the Python runtime, the class PythonObjectWrapper also defines the following public methods built into JavaScript.
Returns a string for representing the object, internally it calls the CPython's PyObject_Str.
console.log(boa.import('os').toString());
// "<module 'os' from '/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/os.py'>"Returns a new wrapped slice object, it's equivalent to s[start:stop:step]. For example:
const { range } = boa.builtins();
const mlist = range(0, 10); // [0...10]
const s = mlist.slice(2, 10, 1); // [2...10]Note: a new tc39 proposal slice notation attempts to add this kind of syntax, it'll be merged when it's land on v8 engine. Or try with
evalin Python's syntax:boa.eval`${mlist}[0...10]` boa.eval`${mlist}[1:10:2]`
Returns the hash value of this object, internally it calls the CPython's PyObject_Hash.
Magic methods there are some others like
__getitem__,__setitem__,__getattr__,__setattr__which are used internally in this library, it's not recommended to actively use them at user-land.
Returns a corresponding primitive value for this object, see Symbol.toPrimitive on MDN for more details.
Requires Node.js >=
v12.11.1
Use Node.js custom loader for better import-statement.
// app.mjs
import { range, len } from 'py:builtins';
import os from 'py:os';
import {
array as NumpyArray,
int32 as NumpyInt32,
} from 'py:numpy';
console.log(os.getpid()); // prints the pid from python.
const list = range(0, 10); // create a range array
console.log(len(list)); // 10
console.log(list[2]); // 2
const arr = NumpyArray([1, 2, 3], NumpyInt32); // Create an array of int32 using ndarry constructor
console.log(arr[0]); // 1In Node.js v14.x you can specify only --experimental-loader to launch your application:
$ node --experimental-loader @pipcook/boa/esm/loader.mjs app.mjsIn Node.js version < v14.x, you also need to add the --experimental-modules option:
$ node --experimental-modules --experimental-loader @pipcook/boa/esm/loader.mjs app.mjsThe package is able to handle the Python generator in JavaScript directly:
# Write a Python Generator in count_down.py
def count_down(count):
while count >= 0:
yield count
count -= 1The above code will take a number and keep yielding and decreasing the value until 0.
const boa = require('@pipcook/boa')
const countDown = boa.import('count_down')
const generator = countDown(3); // Generator
// You can use typical next syntax
let curr = generator.next();
while (curr.done) {
console.log(curr.value); // 3 2 1 0
curr = generator.next()
}Or use the syntactic suger:
const boa = require('@pipcook/boa')
const countDown = boa.import('count_down')
const generator = countDown(3); // Generator
// Or access data via the following syntax
for (const element of generator) {
console.log(element) // 3 2 1 0
}The @pipcook/boa package calls Python function in blocking way, which is because of the Python(CPython)'s object model is not thread-safe, thus Python limits to run Python functions in different threads.
However the @pipcook/boa package allows running Python function in another thread with Node.js worker_threads, an non-blocking example is:
const { Worker, isMainThread, workerData, parentPort } = require('worker_threads');
const boa = require('@pipcook/boa');
const pybasic = boa.import('tests.base.basic'); // a Python example
const { SharedPythonObject, symbols } = boa;
class Foobar extends pybasic.Foobar {
hellomsg(x) {
return `hello <${x}> on ${this.test}(${this.count})`;
}
}
if (isMainThread) {
const foo = new Foobar();
const worker = new Worker(__filename, {
workerData: {
foo: new SharedPythonObject(foo),
},
});
let alive = setInterval(() => {
const ownership = foo[symbols.GetOwnershipSymbol]();
console.log(`ownership should be ${expectedOwnership}.`);
}, 1000);
worker.on('message', state => {
if (state === 'done') {
console.log('task is completed');
setTimeout(() => {
clearInterval(alive);
console.log(foo.ping('x'));
}, 1000);
}
});
} else {
const { foo } = workerData;
console.log(`worker: get an object${foo} and sleep 5s in Python`);
foo.sleep(); // this is a blocking function which is implemented at Python to sleep 1s
console.log('python sleep is done, and sleep in nodejs(thread)');
setTimeout(() => parentPort.postMessage('done'), 1000);
}In the new sub-thread created by worker_threads, the @pipcook/boa won't create the new interrupter, it means all the threads by Node.js shares the same Python interpreter to avoid the Python GIL.
To make sure the thread-safty works, we introduce a SharedPythonObject class to share the Python objects between threads via the following:
// main thread
const foo = new Foobar(); // Python object
const worker = new Worker(__filename, {
workerData: {
foo: new SharedPythonObject(foo),
},
});
// worker thread
const { workerData } = require('worker_threads');
const boa = require('@pipcook/boa');
console.log(workerData.foo);The SharedPythonObject accepts a Python object created by @pipcook/boa, once created, the original object won't be used util the worker thread exits, this is to make sure the thread-safty of the shared objects, it means an object could only be used by worker or main thread at the same time.
# clone this project firstly.
$ npm install
$ npm run buildVerify if the generated library is linked to correct Python version
When buidling finished, use objdump -macho -dylibs-used ./build/Release/boa.node to check if your linked libs are correct as:
/build/Release/boa.node:
/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/Python (compatibility version 3.7.0, current version 3.7.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)See ./tests for more testing details.
To run the full tests:
$ npm testIf you are using virtualenv or conda, you can just set up your system environment PYTHONPATH to point to your site-packages folder. For instance:
$ export PYTHONPATH = /Users/venv/lib/python3.7/site-packages