vendor: OpenCV 5.0.0 snapshot at 755e50675d97db9b7d449d8bd6b09888646f6c6e
|
After Width: | Height: | Size: 76 KiB |
|
After Width: | Height: | Size: 1.2 KiB |
|
After Width: | Height: | Size: 212 KiB |
|
After Width: | Height: | Size: 2.5 KiB |
@@ -0,0 +1,187 @@
|
||||
Processing images causing optical illusions {#tutorial_bioinspired_retina_illusion}
|
||||
=============================================================
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
I will show here how the bioinspired module can reproduce a well-known optical illusion that
|
||||
our eyes perceive in certain light condition: The Adelson checkerboard.
|
||||
|
||||
The Adelson checkerboard
|
||||
------------------------
|
||||
|
||||
Looking at the checkerboard image below, human eyes perceive the "B" square lighter than the
|
||||
"A" square, although they are pictured in the very same RGB color.
|
||||
Of course in the physical world, checkerboard has a "B" square which is lighter than "A", but in this image the
|
||||
shadow of the green cylinder casting over the "B" square ends up in making the "A" and "B"
|
||||
squares actually having the same luminance.
|
||||
|
||||

|
||||
|
||||
Our visual system does "compensate" for the shadow, making us perceive the "B" square lighter,
|
||||
as the shadow wouldn't be there. This is due to local adaptation process that is performed in the
|
||||
foveal area.
|
||||
|
||||
You may find the original Adelson's explanation [here](http://web.mit.edu/persci/people/adelson/checkershadow_description.html).
|
||||
|
||||
Proof: You can convince yourself by using an image manipulation program, cutting out a portion
|
||||
of the two squares, and looking at them without any background. You can also measure the RGB
|
||||
values of the two squares with the picker tool.
|
||||
|
||||
In this image I've cropped a little piece of the A and B squares and I've put them side-by-side.
|
||||
It should be quite evident they have the same luminance.
|
||||

|
||||
|
||||
It's worth to know that this illusion works because the checkerboard image, as you may see it
|
||||
on your laptop, casts on your retina with dimensions that cause the retina local adaptation to take
|
||||
into account both the two squares at the same time.
|
||||
|
||||
The foveal vision area is something like one inch at one meter (and because your eye moves
|
||||
continuously, with the so called "saccades", your brain is able to reconstruct the entire
|
||||
color scene in real time). This means that one single letter, either A or B, can hit
|
||||
your fovea at any time.
|
||||
|
||||
The point is that, even if you can't see both letters at the same time in a single eye fixation,
|
||||
when looking at one letter your fovea also takes into account light information from what is around it.
|
||||
This means that the fovea actually perceives also the neighboring cells.
|
||||
|
||||
The net effect is that when looking at one area, your eye locally adapts to luminance, filters noise,
|
||||
enforces contours, etc. considering what *surrounds* this area, and this makes the illusion work. We
|
||||
say that *the retina works in a "center surround" manner*.
|
||||
|
||||
So, the "A" cell being surrounded by lighter cells can be perceived darker. As a comparison, cell "B" 's
|
||||
neighborhood is darker and the cell "B" is then perceived lighter.
|
||||
|
||||
Finally, since shadow edges are soft, retina eliminates this information. Then shadows do not disrupt the overall chessboard observation making possible to "confidently being fooled" by the perceived cells luminance.
|
||||
|
||||
Reproducing the illusion
|
||||
------------------------
|
||||
The bioinspired module does mimic (also) the parvocellular retina process, that is our foveal
|
||||
vision, and it does reproduce our eyes' local adaptation.
|
||||
|
||||
This means we can expect the parvo channel output to really contain luminance values
|
||||
similar to those we perceive with our eyes. Specifically, in this case we expect the "B" square
|
||||
RGB values to be actually lighter than the "A" ones.
|
||||
|
||||
To correctly mimic what our eye does we need opencv to do the local adaptation on the right
|
||||
image portion. This means we have to ensure that the opencv's notion of "local" does match with our
|
||||
image's dimensions, otherwise the local adaptation wouldn't work as expected.
|
||||
|
||||
For this reason we may have to adjust the **hcellsSpatialConstant** parameter (that technically
|
||||
specifies the low spatial cut frequency, or slow luminance changes sensitivity) depending by
|
||||
the image resolution.
|
||||
|
||||
For the image in this tutorial, the default retina parameters should be fine.
|
||||
|
||||
In order to feed the image to the bioinspired module, you can use either your own code or
|
||||
the *example_bioinspired_retinaDemo* example that comes with the bioinspired module.
|
||||
|
||||
Running
|
||||
@code{.sh}
|
||||
example_bioinspired_retinaDemo -image checkershadow_illusion4med.jpg
|
||||
@endcode
|
||||
|
||||
will cause our image to be processed in both parvocellular and magnocellular channels (we are interested
|
||||
just in the first one).
|
||||
|
||||
If you choose to use your own code, please note that the parvocellular (and magnocellular)
|
||||
channel does require some iterations (frames to be processed) before actually getting steady.
|
||||
|
||||
Actually parvo (and magno) channel do cares about temporal information. That is, when you start
|
||||
feeding frames, it is similar to you with closed eyes; then you open them and you see the chessboard.
|
||||
|
||||
This is a static image but your retina just starts moving to a new context (eyes opening) and
|
||||
has to adapt.
|
||||
|
||||
While in this transient state the luminance information do matters, and you see more or less
|
||||
the absolute luminance values. Absolute luminance is exactly what you need **not** to look at in
|
||||
order to reproduce the illusion..
|
||||
|
||||
As soon as steady state is reached, you receive more contextual luminance information. Your eyes work
|
||||
in a center-surround manner and take into account the neighborhood luminance to evaluate the
|
||||
region of interest luminance level. And that's when our illusion comes out !
|
||||
|
||||
This is something that you don't need to worry about when you process videos, because you are
|
||||
naturally feeding the virtual retina with several frames, but you have to take care of it in
|
||||
order to process a single frame.
|
||||
|
||||
What you will actually need to do when processing a single frame, and you only need steady state response,
|
||||
is to repeatedly feed the retina with the same frame (this is what the example code does), as you
|
||||
would do with a still video. Alternatively you can set retina temporal parameters to 0 to get steady state immediately
|
||||
(**photoreceptorsTemporalConstant** and **hcellsTemporalConstant** parameters of the xml file); however
|
||||
in this case you should be aware that you are now making experiments with something that is
|
||||
deliberately less accurate in reproducing the behaviour of a real retina!
|
||||
|
||||
Here there is a small fragment of python code we used to process the image. It does 20
|
||||
iterations. This is an arbitrary number that we found experimentally to be (more than)
|
||||
enough
|
||||
|
||||
@code{.py}
|
||||
import cv2 as cv
|
||||
|
||||
inputImage = cv.imread('checkershadow_illusion4med.jpg', 1)
|
||||
retina = cv.bioinspired.createRetina((inputImage.shape[1], inputImage.shape[0]))
|
||||
|
||||
# the retina object is created with default parameters. If you want to read
|
||||
# the parameters from an external XML file, uncomment the next line
|
||||
#retina.setup('MyRetinaParameters.xml')
|
||||
|
||||
# feed the retina with several frames, in order to reach 'steady' state
|
||||
for i in range(20):
|
||||
retina.run(inputImage)
|
||||
|
||||
# get our processed image :)
|
||||
retinaOut_parvo = retina.getParvo()
|
||||
|
||||
# show both the original image and the processed one
|
||||
cv.imshow('image', inputImage)
|
||||
cv.imshow('retina parvo out', retinaOut_parvo)
|
||||
|
||||
# wait for a key to be pressed and exit
|
||||
cv.waitKey(0)
|
||||
cv.destroyAllWindows()
|
||||
|
||||
# write the output image on a file
|
||||
cv.imwrite('checkershadow_parvo.png', retinaOut_parvo)
|
||||
@endcode
|
||||
|
||||
Whatever method you used to process the image, you should end up
|
||||
with something like this:
|
||||
|
||||

|
||||
|
||||
Analyzing the results
|
||||
----------------------
|
||||
|
||||
We expected that the "B" pixels in the parvo channel output are lighter than "A" ones.
|
||||
|
||||
.. And in fact that is!
|
||||
|
||||
Looking at the resulting image might not tell us so much at a first glance: the "B" square looks
|
||||
lighter than "A" to our eyes, as it did in the input image. The difference is that, contrarily to
|
||||
the input image, now the RGB values of the pixels are actually lighter; note that when looking at
|
||||
the output image, we are actually applying the parvocellular process
|
||||
two times: first in the bioinspired module, then in our eyes.
|
||||
We can convince ourselves that the illusion appeared
|
||||
in the computed image by measuring the squares' luminance with the image manipulation program
|
||||
and the picker tool, or by cropping pieces of the squares and putting them side-by-side.
|
||||
|
||||
In the following image I cropped a portion of square "A" and a portion of square "B", and I placed
|
||||
them side-by-side, as I did for the original Adelson image.
|
||||
|
||||

|
||||
|
||||
It should be quite evident that the "B" square is really lighter than the "A" square! Congratulations: you have
|
||||
just reproduced the Adelson illusion with the Bioinspired module!
|
||||
|
||||
Credits
|
||||
-------
|
||||
|
||||
I want to thank:
|
||||
|
||||
**Alexandre Benoit** - for being so kind of explaining me how this whole thing works, for giving me the
|
||||
opportunity of writing this tutorial, and for reviewing it.
|
||||
|
||||
**Edward Adelson** - for allowing me to freely use his checkerboard image.
|
||||
|
||||
**Antonio Cuni** - for reviewing this tutorial and for writing the Python code.
|
||||
|
After Width: | Height: | Size: 147 KiB |
|
After Width: | Height: | Size: 49 KiB |
|
After Width: | Height: | Size: 78 KiB |
|
After Width: | Height: | Size: 28 KiB |
|
After Width: | Height: | Size: 68 KiB |
@@ -0,0 +1,479 @@
|
||||
Retina and real-world vision {#tutorial_bioinspired_retina_model}
|
||||
=============================================================
|
||||
|
||||
@tableofcontents
|
||||
|
||||
Goal
|
||||
----
|
||||
|
||||
I present here a model of human retina that shows some interesting properties for image
|
||||
preprocessing and enhancement. In this tutorial you will learn how to:
|
||||
|
||||
- discover the main two channels outing from your retina
|
||||
- see the basics to use the retina model
|
||||
- discover some parameters tweaks
|
||||
|
||||
General overview
|
||||
----------------
|
||||
|
||||
The proposed model originates from Jeanny Herault's research @cite Herault2010 at
|
||||
[Gipsa](http://www.gipsa-lab.inpg.fr). It is involved in image processing applications with
|
||||
[Listic](http://www.listic.univ-savoie.fr) (code maintainer and user) lab. This is not a complete
|
||||
model but it already present interesting properties that can be involved for enhanced image
|
||||
processing experience. The model allows the following human retina properties to be used :
|
||||
|
||||
- spectral whitening that has 3 important effects: high spatio-temporal frequency signals
|
||||
canceling (noise), mid-frequencies details enhancement and low frequencies luminance energy
|
||||
reduction. This *all in one* property directly allows visual signals cleaning of classical
|
||||
undesired distortions introduced by image sensors and input luminance range.
|
||||
- local logarithmic luminance compression allows details to be enhanced even in low light
|
||||
conditions.
|
||||
- decorrelation of the details information (Parvocellular output channel) and transient
|
||||
information (events, motion made available at the Magnocellular output channel).
|
||||
|
||||
The first two points are illustrated below :
|
||||
|
||||
In the figure below, the OpenEXR image sample *CrissyField.exr*, a High Dynamic Range image is
|
||||
shown. In order to make it visible on this web-page, the original input image is linearly rescaled
|
||||
to the classical image luminance range [0-255] and is converted to 8bit/channel format. Such strong
|
||||
conversion hides many details because of too strong local contrasts. Furthermore, noise energy is
|
||||
also strong and pollutes visual information.
|
||||
|
||||

|
||||
|
||||
In the following image, applying the ideas proposed in @cite Benoit2010, as your retina does, local
|
||||
luminance adaptation, spatial noise removal and spectral whitening work together and transmit
|
||||
accurate information on lower range 8bit data channels. On this picture, noise in significantly
|
||||
removed, local details hidden by strong luminance contrasts are enhanced. Output image keeps its
|
||||
naturalness and visual content is enhanced. Color processing is based on the color
|
||||
multiplexing/demultiplexing method proposed in @cite Chaix2007 .
|
||||
|
||||

|
||||
|
||||
*Note :* image sample can be downloaded from the [OpenEXR website](http://www.openexr.com).
|
||||
Regarding this demonstration, before retina processing, input image has been linearly rescaled
|
||||
within 0-255 keeping its channels float format. 5% of its histogram ends has been cut (mostly
|
||||
removes wrong HDR pixels). Check out the sample
|
||||
*opencv/samples/cpp/OpenEXRimages_HighDynamicRange_Retina_toneMapping.cpp* for similar
|
||||
processing. The following demonstration will only consider classical 8bit/channel images.
|
||||
|
||||
The retina model output channels
|
||||
--------------------------------
|
||||
|
||||
The retina model presents two outputs that benefit from the above cited behaviors.
|
||||
|
||||
- The first one is called the Parvocellular channel. It is mainly active in the foveal retina area
|
||||
(high resolution central vision with color sensitive photo-receptors), its aim is to provide
|
||||
accurate color vision for visual details remaining static on the retina. On the other hand
|
||||
objects moving on the retina projection are blurred.
|
||||
- The second well known channel is the Magnocellular channel. It is mainly active in the retina
|
||||
peripheral vision and send signals related to change events (motion, transient events, etc.).
|
||||
These outing signals also help visual system to focus/center retina on 'transient'/moving areas
|
||||
for more detailed analysis thus improving visual scene context and object classification.
|
||||
|
||||
**NOTE :** regarding the proposed model, contrary to the real retina, we apply these two channels on
|
||||
the entire input images using the same resolution. This allows enhanced visual details and motion
|
||||
information to be extracted on all the considered images... but remember, that these two channels
|
||||
are complementary. For example, if Magnocellular channel gives strong energy in an area, then, the
|
||||
Parvocellular channel is certainly blurred there since there is a transient event.
|
||||
|
||||
As an illustration, we apply in the following the retina model on a webcam video stream of a dark
|
||||
visual scene. In this visual scene, captured in an amphitheater of the university, some students are
|
||||
moving while talking to the teacher.
|
||||
|
||||
In this video sequence, because of the dark ambiance, signal to noise ratio is low and color
|
||||
artifacts are present on visual features edges because of the low quality image capture tool-chain.
|
||||
|
||||

|
||||
|
||||
Below is shown the retina foveal vision applied on the entire image. In the used retina
|
||||
configuration, global luminance is preserved and local contrasts are enhanced. Also, signal to noise
|
||||
ratio is improved : since high frequency spatio-temporal noise is reduced, enhanced details are not
|
||||
corrupted by any enhanced noise.
|
||||
|
||||

|
||||
|
||||
Below is the output of the Magnocellular output of the retina model. Its signals are strong where
|
||||
transient events occur. Here, a student is moving at the bottom of the image thus generating high
|
||||
energy. The remaining of the image is static however, it is corrupted by a strong noise. Here, the
|
||||
retina filters out most of the noise thus generating low false motion area 'alarms'. This channel
|
||||
can be used as a transient/moving areas detector : it would provide relevant information for a low
|
||||
cost segmentation tool that would highlight areas in which an event is occurring.
|
||||
|
||||

|
||||
|
||||
Retina use case
|
||||
---------------
|
||||
|
||||
This model can be used basically for spatio-temporal video effects but also in the aim of :
|
||||
|
||||
- performing texture analysis with enhanced signal to noise ratio and enhanced details robust
|
||||
against input images luminance ranges (check out the Parvocellular retina channel output)
|
||||
- performing motion analysis also taking benefit of the previously cited properties.
|
||||
|
||||
Literature
|
||||
----------
|
||||
|
||||
For more information, refer to the following papers : @cite Benoit2010
|
||||
|
||||
- Please have a look at the reference work of Jeanny Herault that you can read in his book @cite Herault2010
|
||||
|
||||
This retina filter code includes the research contributions of phd/research colleagues from which
|
||||
code has been redrawn by the author :
|
||||
|
||||
- take a look at the *retinacolor.hpp* module to discover Brice Chaix de Lavarene phD color
|
||||
mosaicing/demosaicing and his reference paper @cite Chaix2007
|
||||
|
||||
- take a look at *imagelogpolprojection.hpp* to discover retina spatial log sampling which
|
||||
originates from Barthelemy Durette phd with Jeanny Herault. A Retina / V1 cortex projection is
|
||||
also proposed and originates from Jeanny's discussions. More informations in the above cited
|
||||
Jeanny Heraults's book.
|
||||
|
||||
Code tutorial
|
||||
-------------
|
||||
|
||||
Please refer to the original tutorial source code in file
|
||||
*opencv_folder/samples/cpp/tutorial_code/bioinspired/retina_tutorial.cpp*.
|
||||
|
||||
@note do not forget that the retina model is included in the following namespace: cv::bioinspired
|
||||
|
||||
To compile it, assuming OpenCV is correctly installed, use the following command. It requires the
|
||||
opencv_core *(cv::Mat and friends objects management)*, opencv_highgui *(display and image/video
|
||||
read)* and opencv_bioinspired *(Retina description)* libraries to compile.
|
||||
|
||||
@code{.sh}
|
||||
// compile
|
||||
gcc retina_tutorial.cpp -o Retina_tuto -lopencv_core -lopencv_highgui -lopencv_bioinspired -lopencv_videoio -lopencv_imgcodecs
|
||||
|
||||
// Run commands : add 'log' as a last parameter to apply a spatial log sampling (simulates retina sampling)
|
||||
// run on webcam
|
||||
./Retina_tuto -video
|
||||
// run on video file
|
||||
./Retina_tuto -video myVideo.avi
|
||||
// run on an image
|
||||
./Retina_tuto -image myPicture.jpg
|
||||
// run on an image with log sampling
|
||||
./Retina_tuto -image myPicture.jpg log
|
||||
@endcode
|
||||
|
||||
Here is a code explanation :
|
||||
|
||||
Retina definition is present in the bioinspired package and a simple include allows to use it. You
|
||||
can rather use the specific header : *opencv2/bioinspired.hpp* if you prefer but then include the
|
||||
other required openv modules : *opencv2/core.hpp* and *opencv2/highgui.hpp*
|
||||
@code{.cpp}
|
||||
#include "opencv2/opencv.hpp"
|
||||
@endcode
|
||||
Provide user some hints to run the program with a help function
|
||||
@code{.cpp}
|
||||
// the help procedure
|
||||
static void help(std::string errorMessage)
|
||||
{
|
||||
std::cout<<"Program init error : "<<errorMessage<<std::endl;
|
||||
std::cout<<"\nProgram call procedure : retinaDemo [processing mode] [Optional : media target] [Optional LAST parameter: \"log\" to activate retina log sampling]"<<std::endl;
|
||||
std::cout<<"\t[processing mode] :"<<std::endl;
|
||||
std::cout<<"\t -image : for still image processing"<<std::endl;
|
||||
std::cout<<"\t -video : for video stream processing"<<std::endl;
|
||||
std::cout<<"\t[Optional : media target] :"<<std::endl;
|
||||
std::cout<<"\t if processing an image or video file, then, specify the path and filename of the target to process"<<std::endl;
|
||||
std::cout<<"\t leave empty if processing video stream coming from a connected video device"<<std::endl;
|
||||
std::cout<<"\t[Optional : activate retina log sampling] : an optional last parameter can be specified for retina spatial log sampling"<<std::endl;
|
||||
std::cout<<"\t set \"log\" without quotes to activate this sampling, output frame size will be divided by 4"<<std::endl;
|
||||
std::cout<<"\nExamples:"<<std::endl;
|
||||
std::cout<<"\t-Image processing : ./retinaDemo -image lena.jpg"<<std::endl;
|
||||
std::cout<<"\t-Image processing with log sampling : ./retinaDemo -image lena.jpg log"<<std::endl;
|
||||
std::cout<<"\t-Video processing : ./retinaDemo -video myMovie.mp4"<<std::endl;
|
||||
std::cout<<"\t-Live video processing : ./retinaDemo -video"<<std::endl;
|
||||
std::cout<<"\nPlease start again with new parameters"<<std::endl;
|
||||
std::cout<<"****************************************************"<<std::endl;
|
||||
std::cout<<" NOTE : this program generates the default retina parameters file 'RetinaDefaultParameters.xml'"<<std::endl;
|
||||
std::cout<<" => you can use this to fine tune parameters and load them if you save to file 'RetinaSpecificParameters.xml'"<<std::endl;
|
||||
}
|
||||
@endcode
|
||||
Then, start the main program and first declare a *cv::Mat* matrix in which input images will be
|
||||
loaded. Also allocate a *cv::VideoCapture* object ready to load video streams (if necessary)
|
||||
@code{.cpp}
|
||||
int main(int argc, char* argv[]) {
|
||||
// declare the retina input buffer... that will be fed differently in regard of the input media
|
||||
cv::Mat inputFrame;
|
||||
cv::VideoCapture videoCapture; // in case a video media is used, its manager is declared here
|
||||
@endcode
|
||||
In the main program, before processing, first check input command parameters. Here it loads a first
|
||||
input image coming from a single loaded image (if user chose command *-image*) or from a video
|
||||
stream (if user chose command *-video*). Also, if the user added *log* command at the end of its
|
||||
program call, the spatial logarithmic image sampling performed by the retina is taken into account
|
||||
by the Boolean flag *useLogSampling*.
|
||||
@code{.cpp}
|
||||
// welcome message
|
||||
std::cout<<"****************************************************"<<std::endl;
|
||||
std::cout<<"* Retina demonstration : demonstrates the use of is a wrapper class of the Gipsa/Listic Labs retina model."<<std::endl;
|
||||
std::cout<<"* This demo will try to load the file 'RetinaSpecificParameters.xml' (if exists).\nTo create it, copy the autogenerated template 'RetinaDefaultParameters.xml'.\nThen tweak it with your own retina parameters."<<std::endl;
|
||||
// basic input arguments checking
|
||||
if (argc<2)
|
||||
{
|
||||
help("bad number of parameter");
|
||||
return -1;
|
||||
}
|
||||
|
||||
bool useLogSampling = !strcmp(argv[argc-1], "log"); // check if user wants retina log sampling processing
|
||||
|
||||
std::string inputMediaType=argv[1];
|
||||
|
||||
//////////////////////////////////////////////////////////////////////////////
|
||||
// checking input media type (still image, video file, live video acquisition)
|
||||
if (!strcmp(inputMediaType.c_str(), "-image") && argc >= 3)
|
||||
{
|
||||
std::cout<<"RetinaDemo: processing image "<<argv[2]<<std::endl;
|
||||
// image processing case
|
||||
inputFrame = cv::imread(std::string(argv[2]), 1); // load image in RGB mode
|
||||
}else
|
||||
if (!strcmp(inputMediaType.c_str(), "-video"))
|
||||
{
|
||||
if (argc == 2 || (argc == 3 && useLogSampling)) // attempt to grab images from a video capture device
|
||||
{
|
||||
videoCapture.open(0);
|
||||
}else// attempt to grab images from a video filestream
|
||||
{
|
||||
std::cout<<"RetinaDemo: processing video stream "<<argv[2]<<std::endl;
|
||||
videoCapture.open(argv[2]);
|
||||
}
|
||||
|
||||
// grab a first frame to check if everything is ok
|
||||
videoCapture>>inputFrame;
|
||||
}else
|
||||
{
|
||||
// bad command parameter
|
||||
help("bad command parameter");
|
||||
return -1;
|
||||
}
|
||||
@endcode
|
||||
Once all input parameters are processed, a first image should have been loaded, if not, display
|
||||
error and stop program :
|
||||
@code{.cpp}
|
||||
if (inputFrame.empty())
|
||||
{
|
||||
help("Input media could not be loaded, aborting");
|
||||
return -1;
|
||||
}
|
||||
@endcode
|
||||
Now, everything is ready to run the retina model. I propose here to allocate a retina instance and
|
||||
to manage the eventual log sampling option. The Retina constructor expects at least a cv::Size
|
||||
object that shows the input data size that will have to be managed. One can activate other options
|
||||
such as color and its related color multiplexing strategy (here Bayer multiplexing is chosen using
|
||||
*enum cv::bioinspired::RETINA_COLOR_BAYER*). If using log sampling, the image reduction factor
|
||||
(smaller output images) and log sampling strength can be adjusted.
|
||||
@code{.cpp}
|
||||
// pointer to a retina object
|
||||
cv::Ptr<cv::bioinspired::Retina> myRetina;
|
||||
|
||||
// if the last parameter is 'log', then activate log sampling (favour foveal vision and subsamples peripheral vision)
|
||||
if (useLogSampling)
|
||||
{
|
||||
myRetina = cv::bioinspired::createRetina(inputFrame.size(), true, cv::bioinspired::RETINA_COLOR_BAYER, true, 2.0, 10.0);
|
||||
}
|
||||
else// -> else allocate "classical" retina :
|
||||
myRetina = cv::bioinspired::createRetina(inputFrame.size());
|
||||
@endcode
|
||||
Once done, the proposed code writes a default xml file that contains the default parameters of the
|
||||
retina. This is useful to make your own config using this template. Here generated template xml file
|
||||
is called *RetinaDefaultParameters.xml*.
|
||||
@code{.cpp}
|
||||
// save default retina parameters file in order to let you see this and maybe modify it and reload using method "setup"
|
||||
myRetina->write("RetinaDefaultParameters.xml");
|
||||
@endcode
|
||||
In the following line, the retina attempts to load another xml file called
|
||||
*RetinaSpecificParameters.xml*. If you created it and introduced your own setup, it will be loaded,
|
||||
in the other case, default retina parameters are used.
|
||||
@code{.cpp}
|
||||
// load parameters if file exists
|
||||
myRetina->setup("RetinaSpecificParameters.xml");
|
||||
@endcode
|
||||
It is not required here but just to show it is possible, you can reset the retina buffers to zero to
|
||||
force it to forget past events.
|
||||
@code{.cpp}
|
||||
// reset all retina buffers (imagine you close your eyes for a long time)
|
||||
myRetina->clearBuffers();
|
||||
@endcode
|
||||
Now, it is time to run the retina ! First create some output buffers ready to receive the two retina
|
||||
channels outputs
|
||||
@code{.cpp}
|
||||
// declare retina output buffers
|
||||
cv::Mat retinaOutput_parvo;
|
||||
cv::Mat retinaOutput_magno;
|
||||
@endcode
|
||||
Then, run retina in a loop, load new frames from video sequence if necessary and get retina outputs
|
||||
back to dedicated buffers.
|
||||
@code{.cpp}
|
||||
// processing loop with no stop condition
|
||||
while(true)
|
||||
{
|
||||
// if using video stream, then, grabbing a new frame, else, input remains the same
|
||||
if (videoCapture.isOpened())
|
||||
videoCapture>>inputFrame;
|
||||
|
||||
// run retina filter on the loaded input frame
|
||||
myRetina->run(inputFrame);
|
||||
// Retrieve and display retina output
|
||||
myRetina->getParvo(retinaOutput_parvo);
|
||||
myRetina->getMagno(retinaOutput_magno);
|
||||
cv::imshow("retina input", inputFrame);
|
||||
cv::imshow("Retina Parvo", retinaOutput_parvo);
|
||||
cv::imshow("Retina Magno", retinaOutput_magno);
|
||||
cv::waitKey(10);
|
||||
}
|
||||
@endcode
|
||||
That's done ! But if you want to secure the system, take care and manage Exceptions. The retina can
|
||||
throw some when it sees irrelevant data (no input frame, wrong setup, etc.). Then, i recommend to
|
||||
surround all the retina code by a try/catch system like this :
|
||||
@code{.cpp}
|
||||
try{
|
||||
// pointer to a retina object
|
||||
cv::Ptr<cv::Retina> myRetina;
|
||||
[---]
|
||||
// processing loop with no stop condition
|
||||
while(true)
|
||||
{
|
||||
[---]
|
||||
}
|
||||
|
||||
}catch(cv::Exception e)
|
||||
{
|
||||
std::cerr<<"Error using Retina : "<<e.what()<<std::endl;
|
||||
}
|
||||
@endcode
|
||||
|
||||
Retina parameters, what to do ?
|
||||
-------------------------------
|
||||
|
||||
First, it is recommended to read the reference paper @cite Benoit2010
|
||||
|
||||
Once done open the configuration file *RetinaDefaultParameters.xml* generated by the demo and let's
|
||||
have a look at it.
|
||||
@code{.cpp}
|
||||
<?xml version="1.0"?>
|
||||
<opencv_storage>
|
||||
<OPLandIPLparvo>
|
||||
<colorMode>1</colorMode>
|
||||
<normaliseOutput>1</normaliseOutput>
|
||||
<photoreceptorsLocalAdaptationSensitivity>7.5e-01</photoreceptorsLocalAdaptationSensitivity>
|
||||
<photoreceptorsTemporalConstant>9.0e-01</photoreceptorsTemporalConstant>
|
||||
<photoreceptorsSpatialConstant>5.7e-01</photoreceptorsSpatialConstant>
|
||||
<horizontalCellsGain>0.01</horizontalCellsGain>
|
||||
<hcellsTemporalConstant>0.5</hcellsTemporalConstant>
|
||||
<hcellsSpatialConstant>7.</hcellsSpatialConstant>
|
||||
<ganglionCellsSensitivity>7.5e-01</ganglionCellsSensitivity></OPLandIPLparvo>
|
||||
<IPLmagno>
|
||||
<normaliseOutput>1</normaliseOutput>
|
||||
<parasolCells_beta>0.</parasolCells_beta>
|
||||
<parasolCells_tau>0.</parasolCells_tau>
|
||||
<parasolCells_k>7.</parasolCells_k>
|
||||
<amacrinCellsTemporalCutFrequency>2.0e+00</amacrinCellsTemporalCutFrequency>
|
||||
<V0CompressionParameter>9.5e-01</V0CompressionParameter>
|
||||
<localAdaptintegration_tau>0.</localAdaptintegration_tau>
|
||||
<localAdaptintegration_k>7.</localAdaptintegration_k></IPLmagno>
|
||||
</opencv_storage>
|
||||
@endcode
|
||||
Here are some hints but actually, the best parameter setup depends more on what you want to do with
|
||||
the retina rather than the images input that you give to retina. Apart from the more specific case
|
||||
of High Dynamic Range images (HDR) that require more specific setup for specific luminance
|
||||
compression objective, the retina behaviors should be rather stable from content to content. Note
|
||||
that OpenCV is able to manage such HDR format thanks to the OpenEXR images compatibility.
|
||||
|
||||
Then, if the application target requires details enhancement prior to specific image processing, you
|
||||
need to know if mean luminance information is required or not. If not, the the retina can cancel or
|
||||
significantly reduce its energy thus giving more visibility to higher spatial frequency details.
|
||||
|
||||
|
||||
## Basic parameters
|
||||
|
||||
The simplest parameters are as follows :
|
||||
|
||||
- **colorMode** : let the retina process color information (if 1) or gray scale images (if 0). In
|
||||
that last case, only the first channels of the input will be processed.
|
||||
- **normaliseOutput** : each channel has such parameter: if the value is set to 1, then the considered
|
||||
channel's output is rescaled between 0 and 255. Be aware at this case of the Magnocellular output
|
||||
level (motion/transient channel detection). Residual noise will also be rescaled !
|
||||
|
||||
**Note :** using color requires color channels multiplexing/demultipexing which also demands more
|
||||
processing. You can expect much faster processing using gray levels : it would require around 30
|
||||
product per pixel for all of the retina processes and it has recently been parallelized for multicore
|
||||
architectures.
|
||||
|
||||
## Photo-receptors parameters
|
||||
|
||||
The following parameters act on the entry point of the retina - photo-receptors - and has impact on all
|
||||
of the following processes. These sensors are low pass spatio-temporal filters that smooth temporal and
|
||||
spatial data and also adjust their sensitivity to local luminance,thus, leads to improving details extraction
|
||||
and high frequency noise canceling.
|
||||
|
||||
- **photoreceptorsLocalAdaptationSensitivity** between 0 and 1. Values close to 1 allow high
|
||||
luminance log compression's effect at the photo-receptors level. Values closer to 0 provide a more
|
||||
linear sensitivity. Increased alone, it can burn the *Parvo (details channel)* output image. If
|
||||
adjusted in collaboration with **ganglionCellsSensitivity**,images can be very contrasted
|
||||
whatever the local luminance there is... at the cost of a naturalness decrease.
|
||||
- **photoreceptorsTemporalConstant** this setups the temporal constant of the low pass filter
|
||||
effect at the entry of the retina. High value leads to strong temporal smoothing effect : moving
|
||||
objects are blurred and can disappear while static object are favored. But when starting the
|
||||
retina processing, stable state is reached later.
|
||||
- **photoreceptorsSpatialConstant** specifies the spatial constant related to photo-receptors' low
|
||||
pass filter's effect. Those parameters specify the minimum value of the spatial signal period allowed
|
||||
in what follows. Typically, this filter should cut high frequency noise. On the other hand, a 0 value
|
||||
cuts none of the noise while higher values start to cut high spatial frequencies, and progressively
|
||||
lower frequencies... Be aware to not go to high levels if you want to see some details of the input images !
|
||||
A good compromise for color images is a 0.53 value since such choice won't affect too much the color spectrum.
|
||||
Higher values would lead to gray and blurred output images.
|
||||
|
||||
## Horizontal cells parameters
|
||||
|
||||
This parameter set tunes the neural network connected to the photo-receptors, the horizontal cells.
|
||||
It modulates photo-receptors sensitivity and completes the processing for final spectral whitening
|
||||
(part of the spatial band pass effect thus favoring visual details enhancement).
|
||||
|
||||
- **horizontalCellsGain** here is a critical parameter ! If you are not interested with the mean
|
||||
luminance and want just to focus on details enhancement, then, set this parameterto zero. However, if
|
||||
you want to keep some environment luminance's data, let some low spatial frequencies pass into the system and set a
|
||||
higher value (\<1).
|
||||
- **hcellsTemporalConstant** similar to photo-receptors, this parameter acts on the temporal constant of a
|
||||
low pass temporal filter that smoothes input data. Here, a high value generates a high retina
|
||||
after effect while a lower value makes the retina more reactive. This value should be lower than
|
||||
**photoreceptorsTemporalConstant** to limit strong retina after effects.
|
||||
- **hcellsSpatialConstant** is the spatial constant of these cells filter's low pass one.
|
||||
It specifies the lowest spatial frequency allowed in what follows. Visually, a high value leads
|
||||
to very low spatial frequencies processing and leads to salient halo effects. Lower values
|
||||
reduce this effect but has the limit of not go lower than the value of
|
||||
**photoreceptorsSpatialConstant**. Those 2 parameters actually specify the spatial band-pass of
|
||||
the retina.
|
||||
|
||||
**NOTE** Once the processing managed by the previous parameters is done, input data is cleaned from noise
|
||||
and luminance is already partly enhanced. The following parameters act on the last processing stages
|
||||
of the two outing retina signals.
|
||||
|
||||
## Parvo (details channel) dedicated parameter
|
||||
|
||||
- **ganglionCellsSensitivity** specifies the strength of the final local adaptation occurring at
|
||||
the output of this details' dedicated channel. Parameter values remain between 0 and 1. Low value
|
||||
tend to give a linear response while higher values enforce the remaining low contrasted areas.
|
||||
|
||||
**Note :** this parameter can correct eventual burned images by favoring low energetic details of
|
||||
the visual scene, even in bright areas.
|
||||
|
||||
## IPL Magno (motion/transient channel) parameters
|
||||
|
||||
Once image's information are cleaned, this channel acts as a high pass temporal filter that
|
||||
selects only the signals related to transient signals (events, motion, etc.). A low pass spatial filter
|
||||
smoothes extracted transient data while a final logarithmic compression enhances low transient events
|
||||
thus enhancing event sensitivity.
|
||||
|
||||
- **parasolCells_beta** generally set to zero, can be considered as an amplifier gain at the
|
||||
entry point of this processing stage. Generally set to 0.
|
||||
- **parasolCells_tau** the temporal smoothing effect that can be added
|
||||
- **parasolCells_k** the spatial constant of the spatial filtering effect, set it at a high value
|
||||
to favor low spatial frequency signals that are lower subject for residual noise.
|
||||
- **amacrinCellsTemporalCutFrequency** specifies the temporal constant of the high pass filter.
|
||||
High values let slow transient events to be selected.
|
||||
- **V0CompressionParameter** specifies the strength of the log compression. Similar behaviors to
|
||||
previous description but here enforces sensitivity of transient events.
|
||||
- **localAdaptintegration_tau** generally set to 0, has no real use actually in here.
|
||||
- **localAdaptintegration_k** specifies the size of the area on which local adaptation is
|
||||
performed. Low values lead to short range local adaptation (higher sensitivity to noise), high
|
||||
values secure log compression.
|
||||
@@ -0,0 +1,14 @@
|
||||
Discovering the human retina and its use for image processing {#tutorial_table_of_content_retina}
|
||||
============================
|
||||
|
||||
- @subpage tutorial_bioinspired_retina_model
|
||||
|
||||
*Author:* Alexandre Benoit
|
||||
|
||||
Processing regular images
|
||||
|
||||
- @subpage tutorial_bioinspired_retina_illusion
|
||||
|
||||
*Author:* Andrea Merello
|
||||
|
||||
See how to reproduce human eye optical illusions
|
||||