Backend API
Backend API is defined by One Runtime. It is about actual computation of operations and memory management for operands. In order to allow different kinds of computation units or libraries, Backend API is exposed to support user defined operation kernels and memory manager. It contains several C++ interface classes which are subject to change.
How backends are loaded
When a backend ID is given to a session, the compiler module tries to load libbackend_{BACKEND_ID}.so
. If it is successful, the runtime looks up for C API functions in it and makes use of those.
C and C++ API
C API
We have two C API functions which are used as the entrypoint and the exitpoint. Here are the definitions of those.
onert::backend::Backend *onert_backend_create();
void onert_backend_destroy(onert::backend::Backend *backend);
What they do is creating a C++ object and destroying it, respectively. These two functions are the only ones that are dynamically resolved at runtime.
C++ API
NOTE C++ API is subject to change so it may change in every release
C API above is just an entrypoint and it delegates core stuff to the C++ API.
Major classes are described below. One must implement these classes (and some more classes) to create a backend.
Backend
: Responsible for creating a backend context which is a set of backend componentsBackendContext
: Holds data for the current session and also responsible for creation of tensor objects and kernelsBackendContext::genTensors
: Creates tensor objectsBackendContext::genKernels
: Creates kernels
IConfig
: Configurations and miscellaneous stuff (global, not session based)ITensorRegistry
: A set of tensor (ITensor
) objects that are used by the current backend
Please refer to each class document for details. You may refer to Bundle Backends for actual implementation samples.
Provided Backend Implementations
We provide some backends along with the runtime. There is the special backend builtin
which is part of runtime core, and some bundle backends which are baseline backends and samples of backend implementation.
builtin
Backend
builtin
is a special backend that is always loaded (statically linked, part of runtime core). It is implemented just like other backends, but there are some things that it does exclusively.
Has kernels for If, While and Permute operations (Kernels from other backends are never used)
The runtime core directly creates
builtin
’s tensor objects to accept user-given input and output buffersThe runtime core gives the executor a context to
builtin
backend which lets control flow ops properly change the execution flow
Bundle Backends
Without actual implementation of backends, we cannot run any model. So we provide 3 bundle backends which support dozens of operations.
cpu
This backend is written in C++ and all the computation is done exclusively on a CPU.
acl_neon
acl_neon
is a backend that is an adaptation layer of ARM ComputeLibrary NE (NEON) part. So it’s CPU-only and restricted to ARM.
acl_cl
acl_cl
is a backend that is an adaptation layer of ARM ComputeLibrary CL (OpenCL) part. OpenCL support (libOpenCL.so
) is also necessary in the running environment to be able to use this backend. Also, it works only on ARM.