Manual

[stevengj]

At some point we will need a manual, which will have both a tutorial and documentation for individual functions. (Ideally, the latter will be generated from the docstrings, much like in the Julia manual. Can the Lexicon.jl package do this?)

[JaredCrean2]

Docs are now hosted and there is a badge in the readme, but I am waiting on the Documenter package to add a new feature that will enable much better organization.

[lascott]

@stevengj , @JaredCrean2
Gents, for the benefit of those considering wrapping other libraries, any comments, warnings and general help on the clang wrap side would be very helpful. Particularly rewrite.jl.

[JaredCrean2]

Sure, here is a brief overview:

The purpose of Clang.jl is to map C types to the Julia C interface types (double* -> Ptr{Float64}). It also reproduces typedefs and struct declarations in Julia. The purpose of the rewriter function (rewrite.jl) is to map C interfaces types to higher level types (Ptr{Float64} -> Array, Ptr{UInt8} -> String etc.).

With this model in mind, here are a few things I learned in the process of wrapping PETSc. First, the real challenge is that the map of Julia types to C types is a many to one, and mapping C types to Julia types in one to many. This makes it difficult to generalize the C interface types because it is not clear which of the several possible Julia types a C type represents (ie. is a `Ptr{Float64} an array or a Float64 passed by reference). As a result, the PETSc wrappers have two layers: a layer produced by Clang.jl and a handwritten layer that uses the Clang.jl generated layer.

As for what the rewriter does, it generalizes the signatures of the Clang.jl generated functions so they can accept anything that might map to the C type (rather than only the C type). This makes it much easier write the second layer of wrappers. The remapping is accomplished by searching through the Expr objects produced by Clang.jl, locating the function signature and modifying the type annotations. It became convenient to have two types of replacement, recursive and non-recursive. The non-recursive replace finds type annotations that exactly match a list of expressions and replaces them. The recursive replace does a recursive descent search through the expression tree and replaces any expressions that match. The lists of what to replace are defined in Dictionaries at the top of rewrite.jl.

An interesting observation about PETSc is that all the objects (PetscVec, PetscMat, etc.) are actually Ptr{Void}s that point to the objects that contain the data. Clang.jl determined that these are typedefs in C and reproduced them in Julia. This creates a type safety problem. A function that takes an argument of type PetscMat, which typealiased to Ptr{Void} can accept any Ptr{Void}, not just one that points to a PetscMat. To deal with this, we replace the typealiases with definitions of immutable types with a single Ptr{Void} field. This has the same memory layout as a Ptr{Void}, but enables the compiler to distinguish between the different types of objects and throw errors if users pass the wrong type of object.

One warning: Clang.jl is fickle. Sometimes it can't figure out what the arguments to a function are and inserts a single argument of type Cint. Over the course of this project, Clang has gone back and forth between correctly wrapping functions and the Cint argument several times, and I haven't been able to figure out why. PETSc's source code is quite involved and uses lots of macros, so its possible this won't be a problem for simple code bases.