OpenCV  4.2.0
Open Source Computer Vision
Harris corner detector

Goal

In this tutorial you will learn:

  • What features are and why they are important
  • Use the function cv::cornerHarris to detect corners using the Harris-Stephens method.

Theory

What is a feature?

  • In computer vision, usually we need to find matching points between different frames of an environment. Why? If we know how two images relate to each other, we can use both images to extract information of them.
  • When we say matching points we are referring, in a general sense, to characteristics in the scene that we can recognize easily. We call these characteristics features.
  • So, what characteristics should a feature have?
    • It must be uniquely recognizable

Types of Image Features

To mention a few:

  • Edges
  • Corners (also known as interest points)
  • Blobs (also known as regions of interest )

In this tutorial we will study the corner features, specifically.

Why is a corner so special?

  • Because, since it is the intersection of two edges, it represents a point in which the directions of these two edges change. Hence, the gradient of the image (in both directions) have a high variation, which can be used to detect it.

How does it work?

  • Let's look for corners. Since corners represents a variation in the gradient in the image, we will look for this "variation".
  • Consider a grayscale image \(I\). We are going to sweep a window \(w(x,y)\) (with displacements \(u\) in the x direction and \(v\) in the y direction) \(I\) and will calculate the variation of intensity.

    \[E(u,v) = \sum _{x,y} w(x,y)[ I(x+u,y+v) - I(x,y)]^{2}\]

    where:

    • \(w(x,y)\) is the window at position \((x,y)\)
    • \(I(x,y)\) is the intensity at \((x,y)\)
    • \(I(x+u,y+v)\) is the intensity at the moved window \((x+u,y+v)\)
  • Since we are looking for windows with corners, we are looking for windows with a large variation in intensity. Hence, we have to maximize the equation above, specifically the term:

    \[\sum _{x,y}[ I(x+u,y+v) - I(x,y)]^{2}\]

  • Using Taylor expansion:

    \[E(u,v) \approx \sum _{x,y}[ I(x,y) + u I_{x} + vI_{y} - I(x,y)]^{2}\]

  • Expanding the equation and cancelling properly:

    \[E(u,v) \approx \sum _{x,y} u^{2}I_{x}^{2} + 2uvI_{x}I_{y} + v^{2}I_{y}^{2}\]

  • Which can be expressed in a matrix form as:

    \[E(u,v) \approx \begin{bmatrix} u & v \end{bmatrix} \left ( \displaystyle \sum_{x,y} w(x,y) \begin{bmatrix} I_x^{2} & I_{x}I_{y} \\ I_xI_{y} & I_{y}^{2} \end{bmatrix} \right ) \begin{bmatrix} u \\ v \end{bmatrix}\]

  • Let's denote:

    \[M = \displaystyle \sum_{x,y} w(x,y) \begin{bmatrix} I_x^{2} & I_{x}I_{y} \\ I_xI_{y} & I_{y}^{2} \end{bmatrix}\]

  • So, our equation now is:

    \[E(u,v) \approx \begin{bmatrix} u & v \end{bmatrix} M \begin{bmatrix} u \\ v \end{bmatrix}\]

  • A score is calculated for each window, to determine if it can possibly contain a corner:

    \[R = det(M) - k(trace(M))^{2}\]

    where:

    • det(M) = \(\lambda_{1}\lambda_{2}\)
    • trace(M) = \(\lambda_{1}+\lambda_{2}\)

    a window with a score \(R\) greater than a certain value is considered a "corner"

Code

Explanation

Result

The original image:

The detected corners are surrounded by a small black circle

cv::Mat::rows
int rows
the number of rows and columns or (-1, -1) when the matrix has more than 2 dimensions
Definition: mat.hpp:2086
cv::String
std::string String
Definition: cvstd.hpp:150
cv::NORM_MINMAX
@ NORM_MINMAX
flag
Definition: base.hpp:207
cv::cvtColor
void cvtColor(InputArray src, OutputArray dst, int code, int dstCn=0)
Converts an image from one color space to another.
cv::samples::findFile
cv::String findFile(const cv::String &relative_path, bool required=true, bool silentMode=false)
Try to find requested data file.
cv::Mat::zeros
static MatExpr zeros(int rows, int cols, int type)
Returns a zero array of the specified size and type.
cv::Mat::at
_Tp & at(int i0=0)
Returns a reference to the specified array element.
cv::normalize
void normalize(const SparseMat &src, SparseMat &dst, double alpha, int normType)
cv::threshold
double threshold(InputArray src, OutputArray dst, double thresh, double maxval, int type)
Applies a fixed-level threshold to each array element.
cv::waitKey
int waitKey(int delay=0)
Waits for a pressed key.
highgui.hpp
cv::namedWindow
void namedWindow(const String &winname, int flags=WINDOW_AUTOSIZE)
Creates a window.
cv::convertScaleAbs
void convertScaleAbs(InputArray src, OutputArray dst, double alpha=1, double beta=0)
Scales, calculates absolute values, and converts the result to 8-bit.
cv::imread
Mat imread(const String &filename, int flags=IMREAD_COLOR)
Loads an image from a file.
cv::Mat::empty
bool empty() const
Returns true if the array has no elements.
cv::Mat::cols
int cols
Definition: mat.hpp:2086
cv::Mat::size
MatSize size
Definition: mat.hpp:2108
cv::dnn::print
static void print(const MatShape &shape, const String &name="")
Definition: shape_utils.hpp:188
cv::imshow
void imshow(const String &winname, InputArray mat)
Displays an image in the specified window.
cv::Scalar
Scalar_< double > Scalar
Definition: types.hpp:669
cv::Point
Point2i Point
Definition: types.hpp:194
cv::Mat
n-dimensional dense array class
Definition: mat.hpp:791
cv::imshow
void imshow(const String &winname, const ogl::Texture2D &tex)
Displays OpenGL 2D texture in the specified window.
cv::CommandLineParser
Designed for command line parsing.
Definition: utility.hpp:796
cv::COLOR_BGR2GRAY
@ COLOR_BGR2GRAY
convert between RGB/BGR and grayscale, color conversions
Definition: imgproc.hpp:542
cv::createTrackbar
int createTrackbar(const String &trackbarname, const String &winname, int *value, int count, TrackbarCallback onChange=0, void *userdata=0)
Creates a trackbar and attaches it to the specified window.
cv
"black box" representation of the file storage associated with a file on disk.
Definition: affine.hpp:51
imgproc.hpp
cv::normalize
static Vec< _Tp, cn > normalize(const Vec< _Tp, cn > &v)
CV_32FC1
#define CV_32FC1
Definition: interface.h:118
cv::cornerHarris
void cornerHarris(InputArray src, OutputArray dst, int blockSize, int ksize, double k, int borderType=BORDER_DEFAULT)
Harris corner detector.
cv::circle
void circle(InputOutputArray img, Point center, int radius, const Scalar &color, int thickness=1, int lineType=LINE_8, int shift=0)
Draws a circle.