Skip to content
forked from discord/erlpack

High Performance Erlang Term Format Packer

License

MIT, Unknown licenses found

Licenses found

MIT
LICENSE
Unknown
COPYING
Notifications You must be signed in to change notification settings

hzmifork/erlpack

 
 

Repository files navigation

Erlpack

Erlpack is a fast encoder and decoder for the Erlang Term Format (version 131) for Python and JavaScript.

JavaScript

Things that can be packed:

  • Null
  • Booleans
  • Strings
  • Atoms
  • Unicode Strings
  • Floats
  • Integers
  • Longs
  • Longs over 64 bits
  • Objects
  • Arrays
  • Tuples
  • PIDs
  • Ports
  • Exports
  • References

How to pack:

let erlpack = require("erlpack");

packed = erlpack.pack({'a': true, 'list': ['of', 3, 'things', 'to', 'pack']});

How to unpack:

Note: Unpacking requires the binary data be a Uint8Array or Buffer. For those using electron/libchromium see the gotcha below.

let erlpack = require("erlpack");

let unpacked = null;
let packed = new Buffer('', 'binary');
try  {
    unpacked = erlpack.unpack(packed);
}
catch (e) {
    // got an exception parsing
}

Libchromium / Electron Gotcha

Some versions of libchromium replace the native data type backing TypedArrays with a custom data type called blink::WebArrayBuffer. To keep erlpack' dependencies simple this data type is not supported directly. If you're using Electron / Libchromium you need to convert the blink::WebArrayBuffer into a node::Buffer before passing to erlpack. You will need to add this code into your native package somewhere:

v8::Local<v8::Value> ConvertToNodeBuffer(const v8::Local<v8::Object>& blinkArray)
{
    if (node::Buffer::HasInstance(blinkArray)) {
        return blinkArray;
    }
    else if (blinkArray->IsArrayBufferView()) {
        auto byteArray = v8::ArrayBufferView::Cast(*blinkArray);
        return node::Buffer::Copy(v8::Isolate::GetCurrent(), (const char*)byteArray->Buffer()->GetContents().Data(), byteArray->ByteLength()).ToLocalChecked();
    }
    
    return v8::Local<v8::Primitive>(v8::Null(v8::Isolate::GetCurrent()));
}

Then in JavaScript something like:

let packed = NativeUtils.convertToNodeBuffer(new Uint8Array(binaryPayload));
// unpack now using erlpack.unpack(packed)

Python

Things that can be packed:

  • None
  • Booleans
  • Strings
  • Atoms
  • Unicode Strings
  • Floats
  • Integers
  • Longs
  • Longs over 64 bits
  • Dictionaries
  • Lists
  • Tuples
  • User Types (via an encode hook)
  • PIDs
  • Ports
  • Exports
  • References

How to pack:

from erlpack import pack

packed = pack(["thing", "to", "pack"])

How to unpack:

from erlpack import unpack

unpacked = unpack(packed)

How to pack an atom:

from erlpack import Atom, pack

packed = pack(Atom('hello'))

How to use an encode hook.

from erlpack import ErlangTermEncoder

def encode_hook(obj):
    if isinstance(obj, datetime.datetime):
        return obj.isoformat()

encoder = ErlangTermEncoder(encode_hook=encode_hook)
packed = encoder.pack(datetime.datetime(2015, 12, 25, 12, 23, 55))

How to make custom types packable.

from erlpack import pack, Atom

class User(object):
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __erlpack__(self):
        return {
            Atom('name'): self.name,
            Atom('age'): self.age
        }

u = User(name='Jake', age=23)
packed = pack(u)

Go (golang)

Discord has moved away from Go internally and so we do not maintain a version of erlpack in Go ourselves. However, all is not lost!, please check out: https://github.com/JakeMakesStuff/go-erlpack

Building

Python

Generating the new .cpp files can be accomplished by running python setup.py --use-cython build_ext --inplace.

cython must be installed for this to work.

Testing

  1. Install the development version of erlpack with python setup.py develop.
  2. Install pytest.
  3. Execute pytest py/tests.

About

High Performance Erlang Term Format Packer

Resources

License

MIT, Unknown licenses found

Licenses found

MIT
LICENSE
Unknown
COPYING

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Cython 23.7%
  • Python 23.7%
  • C++ 21.8%
  • JavaScript 14.8%
  • C 13.1%
  • Elixir 2.8%
  • Shell 0.1%