Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update calling-c-and-fortran docs #31700

Merged
merged 7 commits into from
Apr 17, 2019
Merged

Update calling-c-and-fortran docs #31700

Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
59743f1
Update calling-c-and-fortran docs
kmsquire Apr 12, 2019
63129e5
Re-clarify Ref/Ptr documentation
kmsquire Apr 16, 2019
886d289
Apply suggestions from code review
vtjnash Apr 16, 2019
a1d7122
Trailing whitespace
kmsquire Apr 16, 2019
eb221b4
Apply suggestions from code review
kmsquire Apr 17, 2019
5c577a2
Minor additional cleanup in response to code (well, doc) review
kmsquire Apr 17, 2019
697ed8d
Wrap long lines
kmsquire Apr 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
183 changes: 87 additions & 96 deletions doc/src/manual/calling-c-and-fortran-code.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,12 @@ The code to be called must be available as a shared library. Most C and Fortran
compiled as shared libraries already, but if you are compiling the code yourself using GCC (or
Clang), you will need to use the `-shared` and `-fPIC` options. The machine instructions generated
by Julia's JIT are the same as a native C call would be, so the resulting overhead is the same
as calling a library function from C code. (Non-library function calls in both C and Julia can
be inlined and thus may have even less overhead than calls to shared library functions. When both
libraries and executables are generated by LLVM, it is possible to perform whole-program optimizations
that can even optimize across this boundary, but Julia does not yet support that. In the future,
however, it may do so, yielding even greater performance gains.)
as calling a library function from C code. [^1]

Shared libraries and functions are referenced by a tuple of the form `(:function, "library")`
or `("function", "library")` where `function` is the C-exported function name. `library` refers
to the shared library name: shared libraries available in the (platform-specific) load path will
be resolved by name, and if necessary a direct path may be specified.
or `("function", "library")` where `function` is the C-exported function name, and `library` refers
to the shared library name. Shared libraries available in the (platform-specific) load path will
be resolved by name. The full path to the library may also be specified.

A function name may be used alone in place of the tuple (just `:function` or `"function"`). In
this case the name is resolved within the current process. This form can be used to call C library
Expand All @@ -37,30 +33,33 @@ arrays and other mutable objects which are normally heap-allocated, but also to
scalar values such as integers and floats which are normally stack-allocated and
commonly passed in registers when using C or Julia calling conventions.

Finally, you can use [`ccall`](@ref) to actually generate a call to the library function. Arguments
to [`ccall`](@ref) are as follows:
Finally, you can use [`ccall`](@ref) to actually generate a call to the library function. The arguments
to [`ccall`](@ref) are:

1. A `(:function, "library")` pair, which must be written as a literal constant,
1. A `(:function, "library")` pair,

OR

a `:function` name symbol or `"function"` name string, which is resolved in the
current process,
a `:function` name symbol or `"function"` name string,

OR

a function pointer (for example, from `dlsym`).

2. Return type (see below for mapping the declared C type to Julia)
2. The function's return type

* This argument will be evaluated at compile-time, when the containing method is defined.
3. A tuple of input types, corresponding to the function signature

3. A tuple of input types. The input types must be written as a literal tuple, not a tuple-valued
variable or expression.
4. The actual argument values to be passed to the function, if any; each is a separate parameter.

* This argument will be evaluated at compile-time, when the containing method is defined.
!!! note
The `(:function, "library")` pair, return type, and input types must be literal constants
(i.e., they can't be variables, but see [Non-constant Function Specifications](@ref) below).

The remaining parameters are evaulated at compile time.
kmsquire marked this conversation as resolved.
Show resolved Hide resolved

4. The following arguments, if any, are the actual argument values passed to the function.
!!! note
See below for how to [map C types to Julia types](@ref mapping-c-types-to-julia).

As a complete but simple example, the following calls the `clock` function from the standard C
library:
Expand Down Expand Up @@ -164,20 +163,20 @@ typedef returntype (*functiontype)(argumenttype, ...)
```

The macro [`@cfunction`](@ref) generates the C-compatible function pointer for a call to a
Julia function. Arguments to [`@cfunction`](@ref) are as follows:
Julia function. The arguments to [`@cfunction`](@ref) are:

1. A Julia Function
2. Return type
2. The return type
kmsquire marked this conversation as resolved.
Show resolved Hide resolved
3. A literal tuple of input types

Like ccall, all of these arguments will be evaluated at compile-time, when the containing method is defined.
As with `ccall`, all of these arguments will be evaluated at compile-time when the containing method is defined.

Currently, only the platform-default C calling convention is supported. This means that
`@cfunction`-generated pointers cannot be used in calls where WINAPI expects `stdcall`
function on 32-bit windows, but can be used on WIN64 (where `stdcall` is unified with the
C calling convention).

A classic example is the standard C library `qsort` function, declared as:
As a classic example, consider the standard C library `qsort` function, declared as:
kmsquire marked this conversation as resolved.
Show resolved Hide resolved

```c
void qsort(void *base, size_t nmemb, size_t size,
Expand All @@ -187,10 +186,11 @@ void qsort(void *base, size_t nmemb, size_t size,
The `base` argument is a pointer to an array of length `nmemb`, with elements of `size` bytes
each. `compare` is a callback function which takes pointers to two elements `a` and `b` and returns
an integer less/greater than zero if `a` should appear before/after `b` (or zero if any order
is permitted). Now, suppose that we have a 1d array `A` of values in Julia that we want to sort
is permitted).

Now, suppose that we have a 1d array `A` of values in Julia that we want to sort
using the `qsort` function (rather than Julia's built-in `sort` function). Before we worry about
calling `qsort` and passing arguments, we need to write a comparison function that works for some
arbitrary objects (which define `<`):
calling `qsort` and passing arguments, we need to write a comparison function:

```jldoctest mycompare
julia> function mycompare(a, b)::Cint
Expand All @@ -199,8 +199,8 @@ julia> function mycompare(a, b)::Cint
mycompare (generic function with 1 method)
```

Notice that we have to be careful about the return type: `qsort` expects a function returning
a C `int`, so we annotate the return type of the function to be sure it returns a `Cint`.
``qsort`` expects a comparison function that return a C ``int``, so we annotate the return type
to be ``Cint``.

In order to pass this function to C, we obtain its address using the macro `@cfunction`:

Expand Down Expand Up @@ -234,23 +234,24 @@ julia> A
```

As can be seen, `A` is changed to the sorted array `[-2.7, 1.3, 3.1, 4.4]`. Note that Julia
knows how to convert an array into a `Ptr{Cdouble}`, how to compute the size of a type in bytes
(identical to C's `sizeof` operator), and so on. For fun, try inserting a `println("mycompare($a, $b)")`
line into `mycompare`, which will allow you to see the comparisons that `qsort` is performing
(and to verify that it is really calling the Julia function that you passed to it).
[takes care of converting the array to a `Ptr{Cdouble}`](@ref automatic-type-conversion)), computing
the size of the element type in bytes, and so on.

For fun, try inserting a `println("mycompare($a, $b)")` line into `mycompare`, which will allow
you to see the comparisons that `qsort` is performing (and to verify that it is really calling
the Julia function that you passed to it).

## Mapping C Types to Julia
## [Mapping C Types to Julia](@id mapping-c-types-to-julia)

It is critical to exactly match the declared C type with its declaration in Julia. Inconsistencies
can cause code that works correctly on one system to fail or produce indeterminate results on
a different system.

Note that no C header files are used anywhere in the process of calling C functions: you are responsible
for making sure that your Julia types and call signatures accurately reflect those in the C header
file. (The [Clang package](https://github.com/ihnorton/Clang.jl) can be used to auto-generate
Julia code from a C header file.)
file.[^2]

### Auto-conversion:
### [Automatic Type Conversion](@id automatic-type-conversion)

Julia automatically inserts calls to the [`Base.cconvert`](@ref) function to convert each argument
to the specified type. For example, the following call:
Expand All @@ -276,9 +277,9 @@ For example, this is used to convert an `Array` of objects (e.g. strings) to an
converting an object to a native pointer can hide the object from the garbage collector, causing
it to be freed prematurely.

### Type Correspondences:
### Type Correspondences

First, a review of some relevant Julia type terminology:
First, let's review some relevant Julia type terminology:

| Syntax / Keyword | Example | Description |
|:----------------------------- |:------------------------------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
Expand Down Expand Up @@ -347,7 +348,7 @@ This can help for writing portable code (and remembering that an `int` in C is n
an `Int` in Julia).


**System Independent:**
**System Independent Types**

| C name | Fortran name | Standard Julia Alias | Julia Base Type |
|:------------------------------------------------------- |:------------------------ |:-------------------- |:-------------------------------------------------------------------------------------------------------------- |
Expand Down Expand Up @@ -388,7 +389,7 @@ to skip the check, you can use `Ptr{UInt8}` as the argument type. `Cstring` can
the [`ccall`](@ref) return type, but in that case it obviously does not introduce any extra
checks and is only meant to improve readability of the call.

**System-dependent:**
**System Dependent Types**

| C name | Standard Julia Alias | Julia Base Type |
|:--------------- |:-------------------- |:-------------------------------------------- |
Expand Down Expand Up @@ -471,7 +472,7 @@ checks and is only meant to improve readability of the call.
!!! note
A C function declared to return `Cvoid` will return the value `nothing` in Julia.

### Struct Type correspondences
### Struct Type Correspondences

Composite types, aka `struct` in C or `TYPE` in Fortran90 (or `STRUCTURE` / `RECORD` in some variants
of F77), can be mirrored in Julia by creating a `struct` definition with the same
Expand All @@ -485,24 +486,27 @@ are not possible in the translation to Julia.

Packed structs and union declarations are not supported by Julia.

You can get a near approximation of a `union` if you know, a priori, the field that will have
You can get an approximation of a `union` if you know, a priori, the field that will have
the greatest size (potentially including padding). When translating your fields to Julia, declare
the Julia field to be only of that type.

Arrays of parameters can be expressed with `NTuple`:
Arrays of parameters can be expressed with `NTuple`. For example, this struct in C,
kmsquire marked this conversation as resolved.
Show resolved Hide resolved

in C:
```c
struct B {
int A[3];
};

b_a_2 = B.A[2];
```
in Julia:

can be written in Julia as

```julia
struct B
A::NTuple{3, Cint}
end

b_a_2 = B.A[3] # note the difference in indexing (1-based in Julia, 0-based in C)
```

Expand Down Expand Up @@ -602,7 +606,7 @@ In Julia code wrapping calls to external C routines, ordinary (non-pointer) data
to be of type `T` inside the [`ccall`](@ref), as they are passed by value. For C code accepting
pointers, [`Ref{T}`](@ref) should generally be used for the types of input arguments, allowing the use
of pointers to memory managed by either Julia or C through the implicit call to [`Base.cconvert`](@ref).
In contrast, pointers returned by the C function called should be declared to be of output type
In contrast, pointers returned by the C function called should be declared to be of output type
[`Ptr{T}`](@ref), reflecting that the memory pointed to is managed by C only. Pointers contained in C
structs should be represented as fields of type `Ptr{T}` within the corresponding Julia struct
types designed to mimic the internal structure of corresponding C structs.
Expand Down Expand Up @@ -718,37 +722,7 @@ ccall(:foo, Cvoid, (Ref{Cint}, Ref{Cfloat}), width, range)
Upon return, the contents of `width` and `range` can be retrieved (if they were changed by `foo`)
by `width[]` and `range[]`; that is, they act like zero-dimensional arrays.

### Special Reference Syntax for ccall (deprecated):

The `&` syntax is deprecated, use the `Ref{T}` argument type instead.

A prefix `&` is used on an argument to [`ccall`](@ref) to indicate that a pointer to a scalar
argument should be passed instead of the scalar value itself (required for all Fortran function
arguments, as noted above). The following example computes a dot product using a BLAS function.

```julia
function compute_dot(DX::Vector{Float64}, DY::Vector{Float64})
@assert length(DX) == length(DY)
n = length(DX)
incx = incy = 1
product = ccall((:ddot_, "libLAPACK"),
Float64,
(Ref{Int32}, Ptr{Float64}, Ref{Int32}, Ptr{Float64}, Ref{Int32}),
n, DX, incx, DY, incy)
return product
end
```

The meaning of prefix `&` is not quite the same as in C. In particular, any changes to the referenced
variables will not be visible in Julia unless the type is mutable (declared via `mutable struct`). However,
even for immutable structs it will not cause any harm for called functions to attempt such modifications
(that is, writing through the passed pointers). Moreover, `&` may be used with any expression,
such as `&0` or `&f(x)`.

When a scalar value is passed with `&` as an argument of type `Ptr{T}`, the value will first be
converted to type `T`.

## Some Examples of C Wrappers
## C Wrapper Examples

Here is a simple example of a C wrapper that returns a `Ptr` type:
kmsquire marked this conversation as resolved.
Show resolved Hide resolved

Expand Down Expand Up @@ -778,12 +752,12 @@ function `gsl_permutation_alloc`. As user code never has to look inside the `gsl
struct, the corresponding Julia wrapper simply needs a new type declaration, `gsl_permutation`,
that has no internal fields and whose sole purpose is to be placed in the type parameter of a
`Ptr` type. The return type of the [`ccall`](@ref) is declared as `Ptr{gsl_permutation}`, since
the memory allocated and pointed to by `output_ptr` is controlled by C (and not Julia).
the memory allocated and pointed to by `output_ptr` is controlled by C.

The input `n` is passed by value, and so the function's input signature is
simply declared as `(Csize_t,)` without any `Ref` or `Ptr` necessary. (If the
wrapper was calling a Fortran function instead, the corresponding function input
signature should instead be `(Ref{Csize_t},)`, since Fortran variables are
signature would instead be `(Ref{Csize_t},)`, since Fortran variables are
passed by pointers.) Furthermore, `n` can be any type that is convertible to a
`Csize_t` integer; the [`ccall`](@ref) implicitly calls [`Base.cconvert(Csize_t,
n)`](@ref).
Expand All @@ -804,13 +778,9 @@ end
```

Here, the input `p` is declared to be of type `Ref{gsl_permutation}`, meaning that the memory
that `p` points to may be managed by Julia or by C. A pointer to memory allocated by C should
that `p` points to may be managed by Julia or by C. A pointer to memory allocated by C should generally
kmsquire marked this conversation as resolved.
Show resolved Hide resolved
be of type `Ptr{gsl_permutation}`, but it is convertible using [`Base.cconvert`](@ref) and therefore
kmsquire marked this conversation as resolved.
Show resolved Hide resolved
can be used in the same (covariant) context of the input argument to a [`ccall`](@ref). A pointer
to memory allocated by Julia must be of type `Ref{gsl_permutation}`, to ensure that the memory
address pointed to is valid and that Julia's garbage collector manages the chunk of memory pointed
to correctly. Therefore, the `Ref{gsl_permutation}` declaration allows pointers managed by C or
Julia to be used.
can be used in the same (covariant) context of the input argument to a [`ccall`](@ref).
kmsquire marked this conversation as resolved.
Show resolved Hide resolved

If the C wrapper never expects the user to pass pointers to memory managed by Julia, then using
kmsquire marked this conversation as resolved.
Show resolved Hide resolved
`p::Ptr{gsl_permutation}` for the method signature of the wrapper and similarly in the [`ccall`](@ref)
Expand Down Expand Up @@ -841,19 +811,29 @@ end
```

The C function wrapped returns an integer error code; the results of the actual evaluation of
the Bessel J function populate the Julia array `result_array`. This variable can only be used
with corresponding input type declaration `Ref{Cdouble}`, since its memory is allocated and managed
by Julia, not C. The implicit call to [`Base.cconvert(Ref{Cdouble}, result_array)`](@ref) unpacks
the Bessel J function populate the Julia array `result_array`. This variable is declared as a
`Ref{Cdouble}`, since its memory is allocated and managed by Julia. The implicit call to
[`Base.cconvert(Ref{Cdouble}, result_array)`](@ref) unpacks
the Julia pointer to a Julia array data structure into a form understandable by C.

Note that for this code to work correctly, `result_array` must be declared to be of type `Ref{Cdouble}`
and not `Ptr{Cdouble}`. The memory is managed by Julia and the `Ref` signature alerts Julia's
garbage collector to keep managing the memory for `result_array` while the [`ccall`](@ref) executes.
If `Ptr{Cdouble}` were used instead, the [`ccall`](@ref) may still work, but Julia's garbage
collector would not be aware that the memory declared for `result_array` is being used by the
external C function. As a result, the code may produce a memory leak if `result_array` never gets
freed by the garbage collector, or if the garbage collector prematurely frees `result_array`,
the C function may end up throwing an invalid memory access exception.
kmsquire marked this conversation as resolved.
Show resolved Hide resolved
## Fortran Wrapper Example

The following example computes a dot product using a Fortran BLAS function. Note the use
kmsquire marked this conversation as resolved.
Show resolved Hide resolved
of `Ref` and `Ptr`.
kmsquire marked this conversation as resolved.
Show resolved Hide resolved

```julia
function compute_dot(DX::Vector{Float64}, DY::Vector{Float64})
@assert length(DX) == length(DY)
n = length(DX)
incx = incy = 1
product = ccall((:ddot_, "libLAPACK"),
Float64,
(Ref{Int32}, Ptr{Float64}, Ref{Int32}, Ptr{Float64}, Ref{Int32}),
n, DX, incx, DY, incy)
return product
end
```


## Garbage Collection Safety

Expand Down Expand Up @@ -1079,3 +1059,14 @@ For more details on how to pass callbacks to C libraries, see this [blog post](h

For direct C++ interfacing, see the [Cxx](https://github.com/Keno/Cxx.jl) package. For tools to create C++
bindings, see the [CxxWrap](https://github.com/JuliaInterop/CxxWrap.jl) package.



[^1]: Non-library function calls in both C and Julia can be inlined and thus may have
even less overhead than calls to shared library functions. When both libraries and executables are
kmsquire marked this conversation as resolved.
Show resolved Hide resolved
generated by LLVM, it is possible to perform whole-program optimizations that can even optimize
kmsquire marked this conversation as resolved.
Show resolved Hide resolved
across this boundary, but Julia does not yet support that. In the future, however, it may do so,
kmsquire marked this conversation as resolved.
Show resolved Hide resolved
yielding even greater performance gains.
kmsquire marked this conversation as resolved.
Show resolved Hide resolved

[^2]: The [Clang package](https://github.com/ihnorton/Clang.jl) can be used to auto-generate Julia code
from a C header file.