vendor: OpenCV 5.0.0 snapshot at 40738fb16ceddb5fb3fea747585f7ce6abb0605b
This commit is contained in:
Gitea Mirror Bot
@@ -0,0 +1,236 @@
|
||||
# Custom deep learning layers support {#tutorial_dnn_custom_layers}
|
||||
|
||||
@tableofcontents
|
||||
|
||||
@prev_tutorial{tutorial_dnn_javascript}
|
||||
@next_tutorial{tutorial_dnn_OCR}
|
||||
|
||||
| | |
|
||||
| -: | :- |
|
||||
| Original author | Dmitry Kurtaev |
|
||||
| Compatibility | OpenCV >= 3.4.1 |
|
||||
|
||||
## Introduction
|
||||
Deep learning is a fast-growing area. New approaches to building neural networks
|
||||
usually introduce new types of layers. These could be modifications of existing
|
||||
ones or implementation of outstanding research ideas.
|
||||
|
||||
OpenCV allows importing and running networks from different deep learning frameworks.
|
||||
There is a number of the most popular layers. However, you can face a problem that
|
||||
your network cannot be imported using OpenCV because some layers of your network
|
||||
can be not implemented in the deep learning engine of OpenCV.
|
||||
|
||||
The first solution is to create a feature request at https://github.com/opencv/opencv/issues
|
||||
mentioning details such as a source of a model and a type of new layer.
|
||||
The new layer could be implemented if the OpenCV community shares this need.
|
||||
|
||||
The second way is to define a **custom layer** so that OpenCV's deep learning engine
|
||||
will know how to use it. This tutorial is dedicated to show you a process of deep
|
||||
learning model's import customization.
|
||||
|
||||
## Define a custom layer in C++
|
||||
Deep learning layer is a building block of network's pipeline.
|
||||
It has connections to **input blobs** and produces results to **output blobs**.
|
||||
There are trained **weights** and **hyper-parameters**.
|
||||
Layers' names, types, weights and hyper-parameters are stored in files are
|
||||
generated by native frameworks during training. If OpenCV encounters unknown
|
||||
layer type it throws an exception while trying to read a model:
|
||||
|
||||
```
|
||||
Unspecified error: Can't create layer "layer_name" of type "MyType" in function getLayerInstance
|
||||
```
|
||||
|
||||
To import the model correctly you have to derive a class from cv::dnn::Layer with
|
||||
the following methods:
|
||||
|
||||
@snippet dnn/custom_layers.hpp A custom layer interface
|
||||
|
||||
And register it before the import:
|
||||
|
||||
@snippet dnn/custom_layers.hpp Register a custom layer
|
||||
|
||||
@note `MyType` is a type of unimplemented layer from the thrown exception.
|
||||
|
||||
Let's see what all the methods do:
|
||||
|
||||
- Constructor
|
||||
|
||||
@snippet dnn/custom_layers.hpp MyLayer::MyLayer
|
||||
|
||||
Retrieves hyper-parameters from cv::dnn::LayerParams. If your layer has trainable
|
||||
weights they will be already stored in the Layer's member cv::dnn::Layer::blobs.
|
||||
|
||||
- A static method `create`
|
||||
|
||||
@snippet dnn/custom_layers.hpp MyLayer::create
|
||||
|
||||
This method should create an instance of you layer and return cv::Ptr with it.
|
||||
|
||||
- Output blobs' shape computation
|
||||
|
||||
@snippet dnn/custom_layers.hpp MyLayer::getMemoryShapes
|
||||
|
||||
Returns layer's output shapes depending on input shapes. You may request an extra
|
||||
memory using `internals`.
|
||||
|
||||
- Run a layer
|
||||
|
||||
@snippet dnn/custom_layers.hpp MyLayer::forward
|
||||
|
||||
Implement a layer's logic here. Compute outputs for given inputs.
|
||||
|
||||
@note OpenCV manages memory allocated for layers. In the most cases the same memory
|
||||
can be reused between layers. So your `forward` implementation should not rely on that
|
||||
the second invocation of `forward` will have the same data at `outputs` and `internals`.
|
||||
|
||||
- Optional `finalize` method
|
||||
|
||||
@snippet dnn/custom_layers.hpp MyLayer::finalize
|
||||
|
||||
The chain of methods is the following: OpenCV deep learning engine calls `create`
|
||||
method once, then it calls `getMemoryShapes` for every created layer, then you
|
||||
can make some preparations depend on known input dimensions at cv::dnn::Layer::finalize.
|
||||
After network was initialized only `forward` method is called for every network's input.
|
||||
|
||||
@note Varying input blobs' sizes such height, width or batch size make OpenCV
|
||||
reallocate all the internal memory. That leads to efficiency gaps. Try to initialize
|
||||
and deploy models using a fixed batch size and image's dimensions.
|
||||
|
||||
## Example: custom layer from TensorFlow
|
||||
This is an example of how to import a network with [tf.image.resize_bilinear](https://www.tensorflow.org/versions/master/api_docs/python/tf/image/resize_bilinear)
|
||||
operation. This is also a resize but with an implementation different from OpenCV's built-in resize.
|
||||
|
||||
Let's create a single layer network:
|
||||
~~~~~~~~~~~~~{.py}
|
||||
inp = tf.placeholder(tf.float32, [2, 3, 4, 5], 'input')
|
||||
resized = tf.image.resize_bilinear(inp, size=[9, 8], name='resize_bilinear')
|
||||
~~~~~~~~~~~~~
|
||||
OpenCV sees that TensorFlow's graph in the following way:
|
||||
|
||||
```
|
||||
node {
|
||||
name: "input"
|
||||
op: "Placeholder"
|
||||
attr {
|
||||
key: "dtype"
|
||||
value {
|
||||
type: DT_FLOAT
|
||||
}
|
||||
}
|
||||
}
|
||||
node {
|
||||
name: "resize_bilinear/size"
|
||||
op: "Const"
|
||||
attr {
|
||||
key: "dtype"
|
||||
value {
|
||||
type: DT_INT32
|
||||
}
|
||||
}
|
||||
attr {
|
||||
key: "value"
|
||||
value {
|
||||
tensor {
|
||||
dtype: DT_INT32
|
||||
tensor_shape {
|
||||
dim {
|
||||
size: 2
|
||||
}
|
||||
}
|
||||
tensor_content: "\t\000\000\000\010\000\000\000"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
node {
|
||||
name: "resize_bilinear"
|
||||
op: "ResizeBilinear"
|
||||
input: "input:0"
|
||||
input: "resize_bilinear/size"
|
||||
attr {
|
||||
key: "T"
|
||||
value {
|
||||
type: DT_FLOAT
|
||||
}
|
||||
}
|
||||
attr {
|
||||
key: "align_corners"
|
||||
value {
|
||||
b: false
|
||||
}
|
||||
}
|
||||
}
|
||||
library {
|
||||
}
|
||||
```
|
||||
Custom layers import from TensorFlow is designed to put all layer's `attr` into
|
||||
cv::dnn::LayerParams but input `Const` blobs into cv::dnn::Layer::blobs.
|
||||
In our case resize's output shape will be stored in layer's `blobs[0]`.
|
||||
|
||||
@snippet dnn/custom_layers.hpp ResizeBilinearLayer
|
||||
|
||||
Next we register a layer and try to import the model.
|
||||
|
||||
@snippet dnn/custom_layers.hpp Register ResizeBilinearLayer
|
||||
|
||||
## Example: custom layer from ONNX
|
||||
ONNX groups operators into **domains**. The standard operators live in the
|
||||
default domain `ai.onnx`; vendors and exporters often place their own ops in a
|
||||
named domain such as `my.namespace`. When OpenCV imports an ONNX node, it looks
|
||||
the op up in cv::dnn::LayerFactory by:
|
||||
|
||||
- the op_type alone, for nodes in the default `ai.onnx` domain (or no domain), and
|
||||
- `"<domain>.<op_type>"`, for nodes in any non-default domain.
|
||||
|
||||
Node attributes are passed through to the layer constructor as
|
||||
cv::dnn::LayerParams entries with the same names. Consider an op `MyCustomOp`
|
||||
with attributes `scale` and `bias` that computes `y = scale * x + bias`. The
|
||||
implementation can look like:
|
||||
|
||||
@snippet dnn/custom_layer_onnx.cpp CustomScaleBiasLayer
|
||||
|
||||
To import a model that uses this op, register the layer **before** calling
|
||||
cv::dnn::readNetFromONNX. Use cv::dnn::LayerFactory::registerLayer for runtime
|
||||
registration (and cv::dnn::LayerFactory::unregisterLayer when done) — pick the
|
||||
right key for the domain of the op as described above:
|
||||
|
||||
@snippet dnn/custom_layer_onnx.cpp Register CustomScaleBiasLayer
|
||||
|
||||
A complete runnable example is available at
|
||||
[samples/dnn/custom_layer_onnx.cpp](https://github.com/opencv/opencv/tree/5.x/samples/dnn/custom_layer_onnx.cpp).
|
||||
Tiny ONNX models exercising both the default-domain and custom-domain registration
|
||||
paths can be generated with
|
||||
[generate_custom_layer_models.py](https://github.com/opencv/opencv_extra/tree/5.x/testdata/dnn/onnx/generate_custom_layer_models.py)
|
||||
in the opencv_extra repository.
|
||||
|
||||
## Define a custom layer in Python
|
||||
The following example shows how to customize OpenCV's layers in Python.
|
||||
|
||||
Let's consider the [Holistically-Nested Edge Detection](https://arxiv.org/abs/1504.06375)
|
||||
model. Its `Crop` layers receive two input blobs and crop the first one to match the
|
||||
spatial dimensions of the second. OpenCV's built-in `Crop` layer trims from the
|
||||
top-left corner, whereas this model expects cropping from the center, so using the
|
||||
built-in behaviour directly would produce shifted results with filled borders.
|
||||
|
||||
Next we're going to replace OpenCV's `Crop` layer that makes top-left cropping by
|
||||
a centric one.
|
||||
|
||||
- Create a class with `getMemoryShapes` and `forward` methods
|
||||
|
||||
@snippet dnn/custom_layer.py CropLayer
|
||||
|
||||
@note Both methods should return lists.
|
||||
|
||||
- Register a new layer.
|
||||
|
||||
@snippet dnn/custom_layer.py Register
|
||||
|
||||
That's it! We have replaced an implemented OpenCV's layer to a custom one.
|
||||
You may find a full script in the [source code](https://github.com/opencv/opencv/tree/5.x/samples/dnn/edge_detection.py).
|
||||
|
||||
<table border="0">
|
||||
<tr>
|
||||
<td></td>
|
||||
<td></td>
|
||||
</tr>
|
||||
</table>
|
||||
Reference in New Issue
Block a user