vendor: OpenCV 5.0.0 snapshot at 755e50675d97db9b7d449d8bd6b09888646f6c6e

This commit is contained in:
Gitea Mirror Bot
2026-08-22 00:11:13 +08:00
commit 12022378a3
3872 changed files with 2513409 additions and 0 deletions
@@ -0,0 +1,402 @@
//
// IMPORTANT: READ BEFORE DOWNLOADING, COPYING, INSTALLING OR USING.
//
// By downloading, copying, installing or using the software you agree to this license.
// If you do not agree to this license, do not download, install,
// copy or use the software.
//
//
// License Agreement
// For Open Source Computer Vision Library
//
// Copyright (C) 2014, OpenCV Foundation, all rights reserved.
// Third party copyrights are property of their respective owners.
//
// Redistribution and use in source and binary forms, with or without modification,
// are permitted provided that the following conditions are met:
//
// * Redistribution's of source code must retain the above copyright notice,
// this list of conditions and the following disclaimer.
//
// * Redistribution's in binary form must reproduce the above copyright notice,
// this list of conditions and the following disclaimer in the documentation
// and/or other materials provided with the distribution.
//
// * The name of the copyright holders may not be used to endorse or promote products
// derived from this software without specific prior written permission.
//
// This software is provided by the copyright holders and contributors "as is" and
// any express or implied warranties, including, but not limited to, the implied
// warranties of merchantability and fitness for a particular purpose are disclaimed.
// In no event shall the Intel Corporation or contributors be liable for any direct,
// indirect, incidental, special, exemplary, or consequential damages
// (including, but not limited to, procurement of substitute goods or services;
// loss of use, data, or profits; or business interruption) however caused
// and on any theory of liability, whether in contract, strict liability,
// or tort (including negligence or otherwise) arising in any way out of
// the use of this software, even if advised of the possibility of such damage.
#ifndef __OPENCV_SURFACE_MATCHING_HPP__
#define __OPENCV_SURFACE_MATCHING_HPP__
#include "surface_matching/ppf_match_3d.hpp"
#include "surface_matching/icp.hpp"
/** @defgroup surface_matching Surface Matching
Note about the License and Patents
-----------------------------------
The following patents have been issued for methods embodied in this
software: "Recognition and pose determination of 3D objects in 3D scenes
using geometric point pair descriptors and the generalized Hough
Transform", Bertram Heinrich Drost, Markus Ulrich, EP Patent 2385483
(Nov. 21, 2012), assignee: MVTec Software GmbH, 81675 Muenchen
(Germany); "Recognition and pose determination of 3D objects in 3D
scenes", Bertram Heinrich Drost, Markus Ulrich, US Patent 8830229 (Sept.
9, 2014), assignee: MVTec Software GmbH, 81675 Muenchen (Germany).
Further patents are pending. For further details, contact MVTec Software
GmbH (info@mvtec.com).
Note that restrictions imposed by these patents (and possibly others)
exist independently of and may be in conflict with the freedoms granted
in this license, which refers to copyright of the program, not patents
for any methods that it implements. Both copyright and patent law must
be obeyed to legally use and redistribute this program and it is not the
purpose of this license to induce you to infringe any patents or other
property right claims or to contest validity of any such claims. If you
redistribute or use the program, then this license merely protects you
from committing copyright infringement. It does not protect you from
committing patent infringement. So, before you do anything with this
program, make sure that you have permission to do so not merely in terms
of copyright, but also in terms of patent law.
Please note that this license is not to be understood as a guarantee
either. If you use the program according to this license, but in
conflict with patent law, it does not mean that the licensor will refund
you for any losses that you incur if you are sued for your patent
infringement.
Introduction to Surface Matching
--------------------------------
Cameras and similar devices with the capability of sensation of 3D structure are becoming more
common. Thus, using depth and intensity information for matching 3D objects (or parts) are of
crucial importance for computer vision. Applications range from industrial control to guiding
everyday actions for visually impaired people. The task in recognition and pose estimation in range
images aims to identify and localize a queried 3D free-form object by matching it to the acquired
database.
From an industrial perspective, enabling robots to automatically locate and pick up randomly placed
and oriented objects from a bin is an important challenge in factory automation, replacing tedious
and heavy manual labor. A system should be able to recognize and locate objects with a predefined
shape and estimate the position with the precision necessary for a gripping robot to pick it up.
This is where vision guided robotics takes the stage. Similar tools are also capable of guiding
robots (and even people) through unstructured environments, leading to automated navigation. These
properties make 3D matching from point clouds a ubiquitous necessity. Within this context, I will
now describe the OpenCV implementation of a 3D object recognition and pose estimation algorithm
using 3D features.
Surface Matching Algorithm Through 3D Features
----------------------------------------------
The state of the algorithms in order to achieve the task 3D matching is heavily based on
@cite drost2010, which is one of the first and main practical methods presented in this area. The
approach is composed of extracting 3D feature points randomly from depth images or generic point
clouds, indexing them and later in runtime querying them efficiently. Only the 3D structure is
considered, and a trivial hash table is used for feature queries.
While being fully aware that utilization of the nice CAD model structure in order to achieve a smart
point sampling, I will be leaving that aside now in order to respect the generalizability of the
methods (Typically for such algorithms training on a CAD model is not needed, and a point cloud
would be sufficient). Below is the outline of the entire algorithm:
![Outline of the Algorithm](img/outline.jpg)
As explained, the algorithm relies on the extraction and indexing of point pair features, which are
defined as follows:
\f[\bf{{F}}(\bf{{m1}}, \bf{{m2}}) = (||\bf{{d}}||_2, <(\bf{{n1}},\bf{{d}}), <(\bf{{n2}},\bf{{d}}), <(\bf{{n1}},\bf{{n2}}))\f]
where \f$\bf{{m1}}\f$ and \f$\bf{{m2}}\f$ are feature two selected points on the model (or scene),
\f$\bf{{d}}\f$ is the difference vector, \f$\bf{{n1}}\f$ and \f$\bf{{n2}}\f$ are the normals at \f$\bf{{m1}}\f$ and
\f$\bf{m2}\f$. During the training stage, this vector is quantized, indexed. In the test stage, same
features are extracted from the scene and compared to the database. With a few tricks like
separation of the rotational components, the pose estimation part can also be made efficient (check
the reference for more details). A Hough-like voting and clustering is employed to estimate the
object pose. To cluster the poses, the raw pose hypotheses are sorted in decreasing order of the
number of votes. From the highest vote, a new cluster is created. If the next pose hypothesis is
close to one of the existing clusters, the hypothesis is added to the cluster and the cluster center
is updated as the average of the pose hypotheses within the cluster. If the next hypothesis is not
close to any of the clusters, it creates a new cluster. The proximity testing is done with fixed
thresholds in translation and rotation. Distance computation and averaging for translation are
performed in the 3D Euclidean space, while those for rotation are performed using quaternion
representation. After clustering, the clusters are sorted in decreasing order of the total number of
votes which determines confidence of the estimated poses.
This pose is further refined using \f$ICP\f$ in order to obtain the final pose.
PPF presented above depends largely on robust computation of angles between 3D vectors. Even though
not reported in the paper, the naive way of doing this (\f$\theta = cos^{-1}({\bf{a}}\cdot{\bf{b}})\f$
remains numerically unstable. A better way to do this is then use inverse tangents, like:
\f[<(\bf{n1},\bf{n2})=tan^{-1}(||{\bf{n1} \wedge \bf{n2}}||_2, \bf{n1} \cdot \bf{n2})\f]
Rough Computation of Object Pose Given PPF
------------------------------------------
Let me summarize the following notation:
- \f$p^i_m\f$: \f$i^{th}\f$ point of the model (\f$p^j_m\f$ accordingly)
- \f$n^i_m\f$: Normal of the \f$i^{th}\f$ point of the model (\f$n^j_m\f$ accordingly)
- \f$p^i_s\f$: \f$i^{th}\f$ point of the scene (\f$p^j_s\f$ accordingly)
- \f$n^i_s\f$: Normal of the \f$i^{th}\f$ point of the scene (\f$n^j_s\f$ accordingly)
- \f$T_{m\rightarrow g}\f$: The transformation required to translate \f$p^i_m\f$ to the origin and rotate
its normal \f$n^i_m\f$ onto the \f$x\f$-axis.
- \f$R_{m\rightarrow g}\f$: Rotational component of \f$T_{m\rightarrow g}\f$.
- \f$t_{m\rightarrow g}\f$: Translational component of \f$T_{m\rightarrow g}\f$.
- \f$(p^i_m)^{'}\f$: \f$i^{th}\f$ point of the model transformed by \f$T_{m\rightarrow g}\f$. (\f$(p^j_m)^{'}\f$
accordingly).
- \f${\bf{R_{m\rightarrow g}}}\f$: Axis angle representation of rotation \f$R_{m\rightarrow g}\f$.
- \f$\theta_{m\rightarrow g}\f$: The angular component of the axis angle representation
\f${\bf{R_{m\rightarrow g}}}\f$.
The transformation in a point pair feature is computed by first finding the transformation
\f$T_{m\rightarrow g}\f$ from the first point, and applying the same transformation to the second one.
Transforming each point, together with the normal, to the ground plane leaves us with an angle to
find out, during a comparison with a new point pair.
We could now simply start writing
\f[(p^i_m)^{'} = T_{m\rightarrow g} p^i_m\f]
where
\f[T_{m\rightarrow g} = -t_{m\rightarrow g}R_{m\rightarrow g}\f]
Note that this is nothing but a stacked transformation. The translational component
\f$t_{m\rightarrow g}\f$ reads
\f[t_{m\rightarrow g} = -R_{m\rightarrow g}p^i_m\f]
and the rotational being
\f[\theta_{m\rightarrow g} = \cos^{-1}(n^i_m \cdot {\bf{x}})\\
{\bf{R_{m\rightarrow g}}} = n^i_m \wedge {\bf{x}}\f]
in axis angle format. Note that bold refers to the vector form. After this transformation, the
feature vectors of the model are registered onto the ground plane X and the angle with respect to
\f$x=0\f$ is called \f$\alpha_m\f$. Similarly, for the scene, it is called \f$\alpha_s\f$.
### Hough-like Voting Scheme
As shown in the outline, PPF (point pair features) are extracted from the model, quantized, stored
in the hashtable and indexed, during the training stage. During the runtime however, the similar
operation is perfomed on the input scene with the exception that this time a similarity lookup over
the hashtable is performed, instead of an insertion. This lookup also allows us to compute a
transformation to the ground plane for the scene pairs. After this point, computing the rotational
component of the pose reduces to computation of the difference \f$\alpha=\alpha_m-\alpha_s\f$. This
component carries the cue about the object pose. A Hough-like voting scheme is performed over the
local model coordinate vector and \f$\alpha\f$. The highest poses achieved for every scene point lets us
recover the object pose.
### Source Code for PPF Matching
~~~{cpp}
// pc is the loaded point cloud of the model
// (Nx6) and pcTest is a loaded point cloud of
// the scene (Mx6)
ppf_match_3d::PPF3DDetector detector(0.03, 0.05);
detector.trainModel(pc);
vector<Pose3DPtr> results;
detector.match(pcTest, results, 1.0/10.0, 0.05);
cout << "Poses: " << endl;
// print the poses
for (size_t i=0; i<results.size(); i++)
{
Pose3DPtr pose = results[i];
cout << "Pose Result " << i << endl;
pose->printPose();
}
~~~
Pose Registration via ICP
-------------------------
The matching process terminates with the attainment of the pose. However, due to the multiple
matching points, erroneous hypothesis, pose averaging and etc. such pose is very open to noise and
many times is far from being perfect. Although the visual results obtained in that stage are
pleasing, the quantitative evaluation shows \f$~10\f$ degrees variation (error), which is an acceptable
level of matching. Many times, the requirement might be set well beyond this margin and it is
desired to refine the computed pose.
Furthermore, in typical RGBD scenes and point clouds, 3D structure can capture only less than half
of the model due to the visibility in the scene. Therefore, a robust pose refinement algorithm,
which can register occluded and partially visible shapes quickly and correctly is not an unrealistic
wish.
At this point, a trivial option would be to use the well known iterative closest point algorithm .
However, utilization of the basic ICP leads to slow convergence, bad registration, outlier
sensitivity and failure to register partial shapes. Thus, it is definitely not suited to the
problem. For this reason, many variants have been proposed . Different variants contribute to
different stages of the pose estimation process.
ICP is composed of \f$6\f$ stages and the improvements I propose for each stage is summarized below.
### Sampling
To improve convergence speed and computation time, it is common to use less points than the model
actually has. However, sampling the correct points to register is an issue in itself. The naive way
would be to sample uniformly and hope to get a reasonable subset. More smarter ways try to identify
the critical points, which are found to highly contribute to the registration process. Gelfand et.
al. exploit the covariance matrix in order to constrain the eigenspace, so that a set of points
which affect both translation and rotation are used. This is a clever way of subsampling, which I
will optionally be using in the implementation.
### Correspondence Search
As the name implies, this step is actually the assignment of the points in the data and the model in
a closest point fashion. Correct assignments will lead to a correct pose, where wrong assignments
strongly degrade the result. In general, KD-trees are used in the search of nearest neighbors, to
increase the speed. However this is not an optimality guarantee and many times causes wrong points
to be matched. Luckily the assignments are corrected over iterations.
To overcome some of the limitations, Picky ICP @cite pickyicp and BC-ICP (ICP using bi-unique
correspondences) are two well-known methods. Picky ICP first finds the correspondences in the
old-fashioned way and then among the resulting corresponding pairs, if more than one scene point
\f$p_i\f$ is assigned to the same model point \f$m_j\f$, it selects \f$p_i\f$ that corresponds to the minimum
distance. BC-ICP on the other hand, allows multiple correspondences first and then resolves the
assignments by establishing bi-unique correspondences. It also defines a novel no-correspondence
outlier, which intrinsically eases the process of identifying outliers.
For reference, both methods are used. Because P-ICP is a bit faster, with not-so-significant
performance drawback, it will be the method of choice in refinment of correspondences.
### Weighting of Pairs
In my implementation, I currently do not use a weighting scheme. But the common approaches involve
*normal compatibility* (\f$w_i=n^1_i\cdot n^2_j\f$) or assigning lower weights to point pairs with
greater distances (\f$w=1-\frac{||dist(m_i,s_i)||_2}{dist_{max}}\f$).
### Rejection of Pairs
The rejections are done using a dynamic thresholding based on a robust estimate of the standard
deviation. In other words, in each iteration, I find the MAD estimate of the Std. Dev. I denote this
as \f$mad_i\f$. I reject the pairs with distances \f$d_i>\tau mad_i\f$. Here \f$\tau\f$ is the threshold of
rejection and by default set to \f$3\f$. The weighting is applied prior to Picky refinement, explained
in the previous stage.
### Error Metric
As described in , a linearization of point to plane as in @cite koklimlow error metric is used. This
both speeds up the registration process and improves convergence.
### Minimization
Even though many non-linear optimizers (such as Levenberg Mardquardt) are proposed, due to the
linearization in the previous step, pose estimation reduces to solving a linear system of equations.
This is what I do exactly using cv::solve with DECOMP_SVD option.
### ICP Algorithm
Having described the steps above, here I summarize the layout of the ICP algorithm.
#### Efficient ICP Through Point Cloud Pyramids
While the up-to-now-proposed variants deal well with some outliers and bad initializations, they
require significant number of iterations. Yet, multi-resolution scheme can help reducing the number
of iterations by allowing the registration to start from a coarse level and propagate to the lower
and finer levels. Such approach both improves the performances and enhances the runtime.
The search is done through multiple levels, in a hierarchical fashion. The registration starts with
a very coarse set of samples of the model. Iteratively, the points are densified and sought. After
each iteration the previously estimated pose is used as an initial pose and refined with the ICP.
#### Visual Results
##### Results on Synthetic Data
In all of the results, the pose is initiated by PPF and the rest is left as:
\f$[\theta_x, \theta_y, \theta_z, t_x, t_y, t_z]=[0]\f$
### Source Code for Pose Refinement Using ICP
~~~{cpp}
ICP icp(200, 0.001f, 2.5f, 8);
// Using the previously declared pc and pcTest
// This will perform registration for every pose
// contained in results
icp.registerModelToScene(pc, pcTest, results);
// results now contain the refined poses
~~~
Results
-------
This section is dedicated to the results of surface matching (point-pair-feature matching and a
following ICP refinement):
![Several matches of a single frog model using ppf + icp](img/gsoc_forg_matches.jpg)
Matches of different models for Mian dataset is presented below:
![Matches of different models for Mian dataset](img/snapshot27.jpg)
You might checkout the video on [youTube here](http://www.youtube.com/watch?v=uFnqLFznuZU).
A Complete Sample
-----------------
### Parameter Tuning
Surface matching module treats its parameters relative to the model diameter (diameter of the axis
parallel bounding box), whenever it can. This makes the parameters independent from the model size.
This is why, both model and scene cloud were subsampled such that all points have a minimum distance
of \f$RelativeSamplingStep*DimensionRange\f$, where \f$DimensionRange\f$ is the distance along a given
dimension. All three dimensions are sampled in similar manner. For example, if
\f$RelativeSamplingStep\f$ is set to 0.05 and the diameter of model is 1m (1000mm), the points sampled
from the object's surface will be approximately 50 mm apart. From another point of view, if the
sampling RelativeSamplingStep is set to 0.05, at most \f$20x20x20 = 8000\f$ model points are generated
(depending on how the model fills in the volume). Consequently this results in at most 8000x8000
pairs. In practice, because the models are not uniformly distributed over a rectangular prism, much
less points are to be expected. Decreasing this value, results in more model points and thus a more
accurate representation. However, note that number of point pair features to be computed is now
quadratically increased as the complexity is O(N\^2). This is especially a concern for 32 bit
systems, where large models can easily overshoot the available memory. Typically, values in the
range of 0.025 - 0.05 seem adequate for most of the applications, where the default value is 0.03.
(Note that there is a difference in this paremeter with the one presented in @cite drost2010 . In
@cite drost2010 a uniform cuboid is used for quantization and model diameter is used for reference of
sampling. In my implementation, the cuboid is a rectangular prism, and each dimension is quantized
independently. I do not take reference from the diameter but along the individual dimensions.
It would very wise to remove the outliers from the model and prepare an ideal model initially. This
is because, the outliers directly affect the relative computations and degrade the matching
accuracy.
During runtime stage, the scene is again sampled by \f$RelativeSamplingStep\f$, as described above.
However this time, only a portion of the scene points are used as reference. This portion is
controlled by the parameter \f$RelativeSceneSampleStep\f$, where
\f$SceneSampleStep = (int)(1.0/RelativeSceneSampleStep)\f$. In other words, if the
\f$RelativeSceneSampleStep = 1.0/5.0\f$, the subsampled scene will once again be uniformly sampled to
1/5 of the number of points. Maximum value of this parameter is 1 and increasing this parameter also
increases the stability, but decreases the speed. Again, because of the initial scene-independent
relative sampling, fine tuning this parameter is not a big concern. This would only be an issue when
the model shape occupies a volume uniformly, or when the model shape is condensed in a tiny place
within the quantization volume (e.g. The octree representation would have too much empty cells).
\f$RelativeDistanceStep\f$ acts as a step of discretization over the hash table. The point pair features
are quantized to be mapped to the buckets of the hashtable. This discretization involves a
multiplication and a casting to the integer. Adjusting RelativeDistanceStep in theory controls the
collision rate. Note that, more collisions on the hashtable results in less accurate estimations.
Reducing this parameter increases the affect of quantization but starts to assign non-similar point
pairs to the same bins. Increasing it however, wanes the ability to group the similar pairs.
Generally, because during the sampling stage, the training model points are selected uniformly with
a distance controlled by RelativeSamplingStep, RelativeDistanceStep is expected to equate to this
value. Yet again, values in the range of 0.025-0.05 are sensible. This time however, when the model
is dense, it is not advised to decrease this value. For noisy scenes, the value can be increased to
improve the robustness of the matching against noisy points.
*/
#endif
@@ -0,0 +1,170 @@
//
// IMPORTANT: READ BEFORE DOWNLOADING, COPYING, INSTALLING OR USING.
//
// By downloading, copying, installing or using the software you agree to this license.
// If you do not agree to this license, do not download, install,
// copy or use the software.
//
//
// License Agreement
// For Open Source Computer Vision Library
//
// Copyright (C) 2014, OpenCV Foundation, all rights reserved.
// Third party copyrights are property of their respective owners.
//
// Redistribution and use in source and binary forms, with or without modification,
// are permitted provided that the following conditions are met:
//
// * Redistribution's of source code must retain the above copyright notice,
// this list of conditions and the following disclaimer.
//
// * Redistribution's in binary form must reproduce the above copyright notice,
// this list of conditions and the following disclaimer in the documentation
// and/or other materials provided with the distribution.
//
// * The name of the copyright holders may not be used to endorse or promote products
// derived from this software without specific prior written permission.
//
// This software is provided by the copyright holders and contributors "as is" and
// any express or implied warranties, including, but not limited to, the implied
// warranties of merchantability and fitness for a particular purpose are disclaimed.
// In no event shall the Intel Corporation or contributors be liable for any direct,
// indirect, incidental, special, exemplary, or consequential damages
// (including, but not limited to, procurement of substitute goods or services;
// loss of use, data, or profits; or business interruption) however caused
// and on any theory of liability, whether in contract, strict liability,
// or tort (including negligence or otherwise) arising in any way out of
// the use of this software, even if advised of the possibility of such damage.
/**
* @file
*
* @brief Implementation of ICP (Iterative Closest Point) Algorithm
* @author Tolga Birdal <tbirdal AT gmail.com>
*/
#ifndef __OPENCV_SURFACE_MATCHING_ICP_HPP__
#define __OPENCV_SURFACE_MATCHING_ICP_HPP__
#include <opencv2/core.hpp>
#include "pose_3d.hpp"
#include <vector>
namespace cv
{
namespace ppf_match_3d
{
//! @addtogroup surface_matching
//! @{
/**
* @brief This class implements a very efficient and robust variant of the iterative closest point (ICP) algorithm.
* The task is to register a 3D model (or point cloud) against a set of noisy target data. The variants are put together
* by myself after certain tests. The task is to be able to match partial, noisy point clouds in cluttered scenes, quickly.
* You will find that my emphasis is on the performance, while retaining the accuracy.
* This implementation is based on Tolga Birdal's MATLAB implementation in here:
* http://www.mathworks.com/matlabcentral/fileexchange/47152-icp-registration-using-efficient-variants-and-multi-resolution-scheme
* The main contributions come from:
* 1. Picky ICP:
* http://www5.informatik.uni-erlangen.de/Forschung/Publikationen/2003/Zinsser03-ARI.pdf
* 2. Efficient variants of the ICP Algorithm:
* http://docs.happycoders.org/orgadoc/graphics/imaging/fasticp_paper.pdf
* 3. Geometrically Stable Sampling for the ICP Algorithm: https://graphics.stanford.edu/papers/stabicp/stabicp.pdf
* 4. Multi-resolution registration:
* http://www.cvl.iis.u-tokyo.ac.jp/~oishi/Papers/Alignment/Jost_MultiResolutionICP_3DIM03.pdf
* 5. Linearization of Point-to-Plane metric by Kok Lim Low:
* https://www.comp.nus.edu.sg/~lowkl/publications/lowk_point-to-plane_icp_techrep.pdf
*/
class CV_EXPORTS_W ICP
{
public:
CV_WRAP enum
{
ICP_SAMPLING_TYPE_UNIFORM = 0,
ICP_SAMPLING_TYPE_GELFAND = 1
};
CV_WRAP ICP()
{
m_tolerance = 0.005f;
m_rejectionScale = 2.5f;
m_maxIterations = 250;
m_numLevels = 6;
m_sampleType = ICP_SAMPLING_TYPE_UNIFORM;
m_numNeighborsCorr = 1;
}
virtual ~ICP() { }
/**
* \brief ICP constructor with default arguments.
* @param [in] iterations
* @param [in] tolerence Controls the accuracy of registration at each iteration of ICP.
* @param [in] rejectionScale Robust outlier rejection is applied for robustness. This value
actually corresponds to the standard deviation coefficient. Points with
rejectionScale * &sigma are ignored during registration.
* @param [in] numLevels Number of pyramid levels to proceed. Deep pyramids increase speed but
decrease accuracy. Too coarse pyramids might have computational overhead on top of the
inaccurate registrtaion. This parameter should be chosen to optimize a balance. Typical
values range from 4 to 10.
* @param [in] sampleType Currently this parameter is ignored and only uniform sampling is
applied. Leave it as 0.
* @param [in] numMaxCorr Currently this parameter is ignored and only PickyICP is applied. Leave it as 1.
*/
CV_WRAP ICP(const int iterations, const float tolerence = 0.05f, const float rejectionScale = 2.5f, const int numLevels = 6, const int sampleType = ICP::ICP_SAMPLING_TYPE_UNIFORM, const int numMaxCorr = 1)
{
m_tolerance = tolerence;
m_numNeighborsCorr = numMaxCorr;
m_rejectionScale = rejectionScale;
m_maxIterations = iterations;
m_numLevels = numLevels;
m_sampleType = sampleType;
}
/**
* \brief Perform registration
*
* @param [in] srcPC The input point cloud for the model. Expected to have the normals (Nx6). Currently,
* CV_32F is the only supported data type.
* @param [in] dstPC The input point cloud for the scene. It is assumed that the model is registered on the scene. Scene remains static. Expected to have the normals (Nx6). Currently, CV_32F is the only supported data type.
* @param [out] residual The output registration error.
* @param [out] pose Transformation between srcPC and dstPC.
* \return On successful termination, the function returns 0.
*
* \details It is assumed that the model is registered on the scene. Scene remains static, while the model transforms. The output poses transform the models onto the scene. Because of the point to plane minimization, the scene is expected to have the normals available. Expected to have the normals (Nx6).
*/
CV_WRAP int registerModelToScene(const Mat& srcPC, const Mat& dstPC, CV_OUT double& residual, CV_OUT Matx44d& pose);
/**
* \brief Perform registration with multiple initial poses
*
* @param [in] srcPC The input point cloud for the model. Expected to have the normals (Nx6). Currently,
* CV_32F is the only supported data type.
* @param [in] dstPC The input point cloud for the scene. Currently, CV_32F is the only supported data type.
* @param [in,out] poses Input poses to start with but also list output of poses.
* \return On successful termination, the function returns 0.
*
* \details It is assumed that the model is registered on the scene. Scene remains static, while the model transforms. The output poses transform the models onto the scene. Because of the point to plane minimization, the scene is expected to have the normals available. Expected to have the normals (Nx6).
*/
CV_WRAP int registerModelToScene(const Mat& srcPC, const Mat& dstPC, CV_IN_OUT std::vector<Pose3DPtr>& poses);
private:
float m_tolerance;
int m_maxIterations;
float m_rejectionScale;
int m_numNeighborsCorr;
int m_numLevels;
int m_sampleType;
};
//! @}
} // namespace ppf_match_3d
} // namespace cv
#endif
@@ -0,0 +1,188 @@
//
// IMPORTANT: READ BEFORE DOWNLOADING, COPYING, INSTALLING OR USING.
//
// By downloading, copying, installing or using the software you agree to this license.
// If you do not agree to this license, do not download, install,
// copy or use the software.
//
//
// License Agreement
// For Open Source Computer Vision Library
//
// Copyright (C) 2014, OpenCV Foundation, all rights reserved.
// Third party copyrights are property of their respective owners.
//
// Redistribution and use in source and binary forms, with or without modification,
// are permitted provided that the following conditions are met:
//
// * Redistribution's of source code must retain the above copyright notice,
// this list of conditions and the following disclaimer.
//
// * Redistribution's in binary form must reproduce the above copyright notice,
// this list of conditions and the following disclaimer in the documentation
// and/or other materials provided with the distribution.
//
// * The name of the copyright holders may not be used to endorse or promote products
// derived from this software without specific prior written permission.
//
// This software is provided by the copyright holders and contributors "as is" and
// any express or implied warranties, including, but not limited to, the implied
// warranties of merchantability and fitness for a particular purpose are disclaimed.
// In no event shall the Intel Corporation or contributors be liable for any direct,
// indirect, incidental, special, exemplary, or consequential damages
// (including, but not limited to, procurement of substitute goods or services;
// loss of use, data, or profits; or business interruption) however caused
// and on any theory of liability, whether in contract, strict liability,
// or tort (including negligence or otherwise) arising in any way out of
// the use of this software, even if advised of the possibility of such damage.
/** @file
@author Tolga Birdal <tbirdal AT gmail.com>
*/
#ifndef __OPENCV_SURFACE_MATCHING_POSE3D_HPP__
#define __OPENCV_SURFACE_MATCHING_POSE3D_HPP__
#include "opencv2/core/cvstd.hpp" // cv::Ptr
#include <vector>
#include <string>
namespace cv
{
namespace ppf_match_3d
{
//! @addtogroup surface_matching
//! @{
class Pose3D;
typedef Ptr<Pose3D> Pose3DPtr;
class PoseCluster3D;
typedef Ptr<PoseCluster3D> PoseCluster3DPtr;
/**
* @brief Class, allowing the storage of a pose. The data structure stores both
* the quaternions and the matrix forms. It supports IO functionality together with
* various helper methods to work with poses
*
*/
class CV_EXPORTS_W Pose3D
{
public:
CV_WRAP Pose3D()
{
alpha=0;
modelIndex=0;
numVotes=0;
residual = 0;
pose = Matx44d::all(0);
}
CV_WRAP Pose3D(double Alpha, size_t ModelIndex=0, size_t NumVotes=0)
{
alpha = Alpha;
modelIndex = ModelIndex;
numVotes = NumVotes;
residual=0;
pose = Matx44d::all(0);
}
/**
* \brief Updates the pose with the new one
* \param [in] NewPose New pose to overwrite
*/
CV_WRAP void updatePose(Matx44d& NewPose);
/**
* \brief Updates the pose with the new one
*/
CV_WRAP void updatePose(Matx33d& NewR, Vec3d& NewT);
/**
* \brief Updates the pose with the new one, but this time using quaternions to represent rotation
*/
CV_WRAP void updatePoseQuat(Vec4d& Q, Vec3d& NewT);
/**
* \brief Left multiplies the existing pose in order to update the transformation
* \param [in] IncrementalPose New pose to apply
*/
CV_WRAP void appendPose(Matx44d& IncrementalPose);
CV_WRAP void printPose();
Pose3DPtr clone();
int writePose(FILE* f);
int readPose(FILE* f);
int writePose(const std::string& FileName);
int readPose(const std::string& FileName);
virtual ~Pose3D() {}
CV_PROP double alpha, residual;
CV_PROP size_t modelIndex, numVotes;
CV_PROP Matx44d pose;
CV_PROP double angle;
CV_PROP Vec3d t;
CV_PROP Vec4d q;
};
/**
* @brief When multiple poses (see Pose3D) are grouped together (contribute to the same transformation)
* pose clusters occur. This class is a general container for such groups of poses. It is possible to store,
* load and perform IO on these poses.
*/
class CV_EXPORTS_W PoseCluster3D
{
public:
PoseCluster3D()
{
numVotes=0;
id=0;
}
PoseCluster3D(Pose3DPtr newPose)
{
poseList.clear();
poseList.push_back(newPose);
numVotes=newPose->numVotes;
id=0;
}
PoseCluster3D(Pose3DPtr newPose, int newId)
{
poseList.push_back(newPose);
this->numVotes = newPose->numVotes;
this->id = newId;
}
virtual ~PoseCluster3D()
{}
/**
* \brief Adds a new pose to the cluster. The pose should be "close" to the mean poses
* in order to preserve the consistency
* \param [in] newPose Pose to add to the cluster
*/
void addPose(Pose3DPtr newPose);
int writePoseCluster(FILE* f);
int readPoseCluster(FILE* f);
int writePoseCluster(const std::string& FileName);
int readPoseCluster(const std::string& FileName);
std::vector<Pose3DPtr> poseList;
size_t numVotes;
int id;
};
//! @}
} // namespace ppf_match_3d
} // namespace cv
#endif
@@ -0,0 +1,153 @@
//
// IMPORTANT: READ BEFORE DOWNLOADING, COPYING, INSTALLING OR USING.
//
// By downloading, copying, installing or using the software you agree to this license.
// If you do not agree to this license, do not download, install,
// copy or use the software.
//
//
// License Agreement
// For Open Source Computer Vision Library
//
// Copyright (C) 2014, OpenCV Foundation, all rights reserved.
// Third party copyrights are property of their respective owners.
//
// Redistribution and use in source and binary forms, with or without modification,
// are permitted provided that the following conditions are met:
//
// * Redistribution's of source code must retain the above copyright notice,
// this list of conditions and the following disclaimer.
//
// * Redistribution's in binary form must reproduce the above copyright notice,
// this list of conditions and the following disclaimer in the documentation
// and/or other materials provided with the distribution.
//
// * The name of the copyright holders may not be used to endorse or promote products
// derived from this software without specific prior written permission.
//
// This software is provided by the copyright holders and contributors "as is" and
// any express or implied warranties, including, but not limited to, the implied
// warranties of merchantability and fitness for a particular purpose are disclaimed.
// In no event shall the Intel Corporation or contributors be liable for any direct,
// indirect, incidental, special, exemplary, or consequential damages
// (including, but not limited to, procurement of substitute goods or services;
// loss of use, data, or profits; or business interruption) however caused
// and on any theory of liability, whether in contract, strict liability,
// or tort (including negligence or otherwise) arising in any way out of
// the use of this software, even if advised of the possibility of such damage.
//
/** @file
@author Tolga Birdal <tbirdal AT gmail.com>
*/
#ifndef __OPENCV_SURFACE_MATCHING_HELPERS_HPP__
#define __OPENCV_SURFACE_MATCHING_HELPERS_HPP__
#include <opencv2/core.hpp>
namespace cv
{
namespace ppf_match_3d
{
//! @addtogroup surface_matching
//! @{
/**
* @brief Load a PLY file
* @param [in] fileName The PLY model to read
* @param [in] withNormals Flag wheather the input PLY contains normal information,
* and whether it should be loaded or not
* @return Returns the matrix on successful load
*/
CV_EXPORTS_W Mat loadPLYSimple(const char* fileName, int withNormals = 0);
/**
* @brief Write a point cloud to PLY file
* @param [in] PC Input point cloud
* @param [in] fileName The PLY model file to write
*/
CV_EXPORTS_W void writePLY(Mat PC, const char* fileName);
/**
* @brief Used for debbuging pruposes, writes a point cloud to a PLY file with the tip
* of the normal vectors as visible red points
* @param [in] PC Input point cloud
* @param [in] fileName The PLY model file to write
*/
CV_EXPORTS_W void writePLYVisibleNormals(Mat PC, const char* fileName);
Mat samplePCUniform(Mat PC, int sampleStep);
Mat samplePCUniformInd(Mat PC, int sampleStep, std::vector<int>& indices);
/**
* Sample a point cloud using uniform steps
* @param [in] pc Input point cloud
* @param [in] xrange X components (min and max) of the bounding box of the model
* @param [in] yrange Y components (min and max) of the bounding box of the model
* @param [in] zrange Z components (min and max) of the bounding box of the model
* @param [in] sample_step_relative The point cloud is sampled such that all points
* have a certain minimum distance. This minimum distance is determined relatively using
* the parameter sample_step_relative.
* @param [in] weightByCenter The contribution of the quantized data points can be weighted
* by the distance to the origin. This parameter enables/disables the use of weighting.
* @return Sampled point cloud
*/
CV_EXPORTS_W Mat samplePCByQuantization(Mat pc, Vec2f& xrange, Vec2f& yrange, Vec2f& zrange, float sample_step_relative, int weightByCenter=0);
void computeBboxStd(Mat pc, Vec2f& xRange, Vec2f& yRange, Vec2f& zRange);
void* indexPCFlann(Mat pc);
void destroyFlann(void* flannIndex);
void queryPCFlann(void* flannIndex, Mat& pc, Mat& indices, Mat& distances);
void queryPCFlann(void* flannIndex, Mat& pc, Mat& indices, Mat& distances, const int numNeighbors);
Mat normalizePCCoeff(Mat pc, float scale, float* Cx, float* Cy, float* Cz, float* MinVal, float* MaxVal);
Mat transPCCoeff(Mat pc, float scale, float Cx, float Cy, float Cz, float MinVal, float MaxVal);
/**
* Transforms the point cloud with a given a homogeneous 4x4 pose matrix (in double precision)
* @param [in] pc Input point cloud (CV_32F family). Point clouds with 3 or 6 elements per
* row are expected. In the case where the normals are provided, they are also rotated to be
* compatible with the entire transformation
* @param [in] Pose 4x4 pose matrix, but linearized in row-major form.
* @return Transformed point cloud
*/
CV_EXPORTS_W Mat transformPCPose(Mat pc, const Matx44d& Pose);
/**
* Generate a random 4x4 pose matrix
* @param [out] Pose The random pose
*/
CV_EXPORTS_W void getRandomPose(Matx44d& Pose);
/**
* Adds a uniform noise in the given scale to the input point cloud
* @param [in] pc Input point cloud (CV_32F family).
* @param [in] scale Input scale of the noise. The larger the scale, the more noisy the output
*/
CV_EXPORTS_W Mat addNoisePC(Mat pc, double scale);
/**
* @brief Compute the normals of an arbitrary point cloud
* computeNormalsPC3d uses a plane fitting approach to smoothly compute
* local normals. Normals are obtained through the eigenvector of the covariance
* matrix, corresponding to the smallest eigen value.
* If PCNormals is provided to be an Nx6 matrix, then no new allocation
* is made, instead the existing memory is overwritten.
* @param [in] PC Input point cloud to compute the normals for.
* @param [out] PCNormals Output point cloud
* @param [in] NumNeighbors Number of neighbors to take into account in a local region
* @param [in] FlipViewpoint Should normals be flipped to a viewing direction?
* @param [in] viewpoint
* @return Returns 0 on success
*/
CV_EXPORTS_W int computeNormalsPC3d(const Mat& PC, CV_OUT Mat& PCNormals, const int NumNeighbors, const bool FlipViewpoint, const Vec3f& viewpoint);
//! @}
} // namespace ppf_match_3d
} // namespace cv
#endif
@@ -0,0 +1,179 @@
//
// IMPORTANT: READ BEFORE DOWNLOADING, COPYING, INSTALLING OR USING.
//
// By downloading, copying, installing or using the software you agree to this license.
// If you do not agree to this license, do not download, install,
// copy or use the software.
//
//
// License Agreement
// For Open Source Computer Vision Library
//
// Copyright (C) 2014, OpenCV Foundation, all rights reserved.
// Third party copyrights are property of their respective owners.
//
// Redistribution and use in source and binary forms, with or without modification,
// are permitted provided that the following conditions are met:
//
// * Redistribution's of source code must retain the above copyright notice,
// this list of conditions and the following disclaimer.
//
// * Redistribution's in binary form must reproduce the above copyright notice,
// this list of conditions and the following disclaimer in the documentation
// and/or other materials provided with the distribution.
//
// * The name of the copyright holders may not be used to endorse or promote products
// derived from this software without specific prior written permission.
//
// This software is provided by the copyright holders and contributors "as is" and
// any express or implied warranties, including, but not limited to, the implied
// warranties of merchantability and fitness for a particular purpose are disclaimed.
// In no event shall the Intel Corporation or contributors be liable for any direct,
// indirect, incidental, special, exemplary, or consequential damages
// (including, but not limited to, procurement of substitute goods or services;
// loss of use, data, or profits; or business interruption) however caused
// and on any theory of liability, whether in contract, strict liability,
// or tort (including negligence or otherwise) arising in any way out of
// the use of this software, even if advised of the possibility of such damage.
//
/**
** ppf_match_3d : Interfaces for matching 3d surfaces in 3d scenes. This module implements the algorithm from Bertram Drost and Slobodan Ilic.
** Use: Read a 3D model, load a 3D scene and match the model to the scene
**
**
** Creation - 2014
** Author: Tolga Birdal (tbirdal@gmail.com)
**
** Refer to the following research paper for more information:
** B. Drost, Markus Ulrich, N. Navab, S. Ilic
Model Globally, Match Locally: Efficient and Robust 3D Object Recognition
IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, California (USA), June 2010.
***/
/** @file
@author Tolga Birdal <tbirdal AT gmail.com>
*/
#ifndef __OPENCV_SURFACE_MATCHING_PPF_MATCH_3D_HPP__
#define __OPENCV_SURFACE_MATCHING_PPF_MATCH_3D_HPP__
#include <opencv2/core.hpp>
#include <vector>
#include "pose_3d.hpp"
#include "t_hash_int.hpp"
namespace cv
{
namespace ppf_match_3d
{
//! @addtogroup surface_matching
//! @{
/**
* @brief Struct, holding a node in the hashtable
*/
typedef struct THash
{
int id;
int i, ppfInd;
} THash;
/**
* @brief Class, allowing the load and matching 3D models.
* Typical Use:
* @code
* // Train a model
* ppf_match_3d::PPF3DDetector detector(0.05, 0.05);
* detector.trainModel(pc);
* // Search the model in a given scene
* vector<Pose3DPtr> results;
* detector.match(pcTest, results, 1.0/5.0,0.05);
* @endcode
*/
class CV_EXPORTS_W PPF3DDetector
{
public:
/**
* \brief Empty constructor. Sets default arguments
*/
CV_WRAP PPF3DDetector();
/**
* Constructor with arguments
* @param [in] relativeSamplingStep Sampling distance relative to the object's diameter. Models are first sampled uniformly in order to improve efficiency. Decreasing this value leads to a denser model, and a more accurate pose estimation but the larger the model, the slower the training. Increasing the value leads to a less accurate pose computation but a smaller model and faster model generation and matching. Beware of the memory consumption when using small values.
* @param [in] relativeDistanceStep The discretization distance of the point pair distance relative to the model's diameter. This value has a direct impact on the hashtable. Using small values would lead to too fine discretization, and thus ambiguity in the bins of hashtable. Too large values would lead to no discrimination over the feature vectors and different point pair features would be assigned to the same bin. This argument defaults to the value of RelativeSamplingStep. For noisy scenes, the value can be increased to improve the robustness of the matching against noisy points.
* @param [in] numAngles Set the discretization of the point pair orientation as the number of subdivisions of the angle. This value is the equivalent of RelativeDistanceStep for the orientations. Increasing the value increases the precision of the matching but decreases the robustness against incorrect normal directions. Decreasing the value decreases the precision of the matching but increases the robustness against incorrect normal directions. For very noisy scenes where the normal directions can not be computed accurately, the value can be set to 25 or 20.
*/
CV_WRAP PPF3DDetector(const double relativeSamplingStep, const double relativeDistanceStep=0.05, const double numAngles=30);
virtual ~PPF3DDetector();
/**
* Set the parameters for the search
* @param [in] positionThreshold Position threshold controlling the similarity of translations. Depends on the units of calibration/model.
* @param [in] rotationThreshold Position threshold controlling the similarity of rotations. This parameter can be perceived as a threshold over the difference of angles
* @param [in] useWeightedClustering The algorithm by default clusters the poses without weighting. A non-zero value would indicate that the pose clustering should take into account the number of votes as the weights and perform a weighted averaging instead of a simple one.
*/
void setSearchParams(const double positionThreshold=-1, const double rotationThreshold=-1, const bool useWeightedClustering=false);
/**
* \brief Trains a new model.
*
* @param [in] Model The input point cloud with normals (Nx6)
*
* \details Uses the parameters set in the constructor to downsample and learn a new model. When the model is learnt, the instance gets ready for calling "match".
*/
CV_WRAP void trainModel(const Mat& Model);
/**
* \brief Matches a trained model across a provided scene.
*
* @param [in] scene Point cloud for the scene
* @param [out] results List of output poses
* @param [in] relativeSceneSampleStep The ratio of scene points to be used for the matching after sampling with relativeSceneDistance. For example, if this value is set to 1.0/5.0, every 5th point from the scene is used for pose estimation. This parameter allows an easy trade-off between speed and accuracy of the matching. Increasing the value leads to less points being used and in turn to a faster but less accurate pose computation. Decreasing the value has the inverse effect.
* @param [in] relativeSceneDistance Set the distance threshold relative to the diameter of the model. This parameter is equivalent to relativeSamplingStep in the training stage. This parameter acts like a prior sampling with the relativeSceneSampleStep parameter.
*/
CV_WRAP void match(const Mat& scene, CV_OUT std::vector<Pose3DPtr> &results, const double relativeSceneSampleStep=1.0/5.0, const double relativeSceneDistance=0.03);
void read(const FileNode& fn);
void write(FileStorage& fs) const;
protected:
double angle_step, angle_step_radians, distance_step;
double sampling_step_relative, angle_step_relative, distance_step_relative;
Mat sampled_pc, ppf;
int num_ref_points;
hashtable_int* hash_table;
THash* hash_nodes;
double position_threshold, rotation_threshold;
bool use_weighted_avg;
int scene_sample_step;
void clearTrainingModels();
private:
void computePPFFeatures(const Vec3d& p1, const Vec3d& n1,
const Vec3d& p2, const Vec3d& n2,
Vec4d& f);
bool matchPose(const Pose3D& sourcePose, const Pose3D& targetPose);
void clusterPoses(std::vector<Pose3DPtr>& poseList, int numPoses, std::vector<Pose3DPtr> &finalPoses);
bool trained;
};
//! @}
} // namespace ppf_match_3d
} // namespace cv
#endif
@@ -0,0 +1,113 @@
//
// IMPORTANT: READ BEFORE DOWNLOADING, COPYING, INSTALLING OR USING.
//
// By downloading, copying, installing or using the software you agree to this license.
// If you do not agree to this license, do not download, install,
// copy or use the software.
//
//
// License Agreement
// For Open Source Computer Vision Library
//
// Copyright (C) 2014, OpenCV Foundation, all rights reserved.
// Third party copyrights are property of their respective owners.
//
// Redistribution and use in source and binary forms, with or without modification,
// are permitted provided that the following conditions are met:
//
// * Redistribution's of source code must retain the above copyright notice,
// this list of conditions and the following disclaimer.
//
// * Redistribution's in binary form must reproduce the above copyright notice,
// this list of conditions and the following disclaimer in the documentation
// and/or other materials provided with the distribution.
//
// * The name of the copyright holders may not be used to endorse or promote products
// derived from this software without specific prior written permission.
//
// This software is provided by the copyright holders and contributors "as is" and
// any express or implied warranties, including, but not limited to, the implied
// warranties of merchantability and fitness for a particular purpose are disclaimed.
// In no event shall the Intel Corporation or contributors be liable for any direct,
// indirect, incidental, special, exemplary, or consequential damages
// (including, but not limited to, procurement of substitute goods or services;
// loss of use, data, or profits; or business interruption) however caused
// and on any theory of liability, whether in contract, strict liability,
// or tort (including negligence or otherwise) arising in any way out of
// the use of this software, even if advised of the possibility of such damage.
//
/** @file
@author Tolga Birdal <tbirdal AT gmail.com>
*/
#ifndef __OPENCV_SURFACE_MATCHING_T_HASH_INT_HPP__
#define __OPENCV_SURFACE_MATCHING_T_HASH_INT_HPP__
#include <stdio.h>
#include <stdlib.h>
namespace cv
{
namespace ppf_match_3d
{
//! @addtogroup surface_matching
//! @{
typedef uint KeyType;
typedef struct hashnode_i
{
KeyType key;
void *data;
struct hashnode_i *next;
} hashnode_i ;
typedef struct HSHTBL_i
{
size_t size;
struct hashnode_i **nodes;
size_t (*hashfunc)(uint);
} hashtable_int;
/** @brief Round up to the next highest power of 2
from http://www-graphics.stanford.edu/~seander/bithacks.html
*/
inline static uint next_power_of_two(uint value)
{
--value;
value |= value >> 1;
value |= value >> 2;
value |= value >> 4;
value |= value >> 8;
value |= value >> 16;
++value;
return value;
}
hashtable_int *hashtableCreate(size_t size, size_t (*hashfunc)(uint));
void hashtableDestroy(hashtable_int *hashtbl);
int hashtableInsert(hashtable_int *hashtbl, KeyType key, void *data);
int hashtableInsertHashed(hashtable_int *hashtbl, KeyType key, void *data);
int hashtableRemove(hashtable_int *hashtbl, KeyType key);
void *hashtableGet(hashtable_int *hashtbl, KeyType key);
hashnode_i* hashtableGetBucketHashed(hashtable_int *hashtbl, KeyType key);
int hashtableResize(hashtable_int *hashtbl, size_t size);
hashtable_int *hashtable_int_clone(hashtable_int *hashtbl);
hashtable_int *hashtableRead(FILE* f);
int hashtableWrite(const hashtable_int * hashtbl, const size_t dataSize, FILE* f);
void hashtablePrint(hashtable_int *hashtbl);
//! @}
} // namespace ppf_match_3d
} // namespace cv
#endif