How can robots perceive the world and their own movements so that they accomplish navigation and manipulation tasks? In this module, we will study how images and videos acquired by cameras mounted on robots are transformed into representations like features and optical flow. Such 2D representations allow us then to extract 3D information about where the camera is and in which direction the robot moves. You will come to understand how grasping objects is facilitated by the computation of 3D posing of objects and navigation can be accomplished by visual odometry and landmark-based localization.
Perspective Projection II
Loading...
來自 University of Pennsylvania 的課程
Robotics: Perception
361 個評分
從本節課中
Geometry of Image Formation
Welcome to Robotics: Perception! We will begin this course with a tutorial on the standard camera models used in computer vision. These models allow us to understand, in a geometric fashion, how light from a scene enters a camera and projects onto a 2D image. By defining these models mathematically, we will be able understand exactly how a point in 3D corresponds to a point in the image and how an image will change as we move a camera in a 3D environment. In the later modules, we will be able to use this information to perform complex perception tasks such as reconstructing 3D scenes from video.
與講師見面
Kostas DaniilidisProfessor of Computer and Information Science
School of Engineering and Applied Science
Jianbo ShiProfessor of Computer and Information Science
School of Engineering and Applied Science
Next we're going to look at how do we form a line by defining two points on the line.
Now we know that in an image plane if
I gave you two points you can always draw a straight line between them.
Now the question is, how do I write down the expression of this line in term of a,
and b, and c, and form the location of x and y.
Let's think about how we can do this.
It's not the easy way to do if it were just following measured angle and
the distance to the origin.
It'll probably some work to get to it.
And we see, if we just follow the rules of homogeneous coordinates,
by thinking those lines no longer as lines in the image planes.
But take a step back of the plane and think of this line in fact as a plane,
it probably becomes easier to solve.
How?
Recall, we have two points on the image plane.
And these two points can be thought of as two rays coming from an origin
to those points into the space.
So now I have not just two points, but I have two rays.
And think about the plane that could be formed by two rays.
Two rays forms a plane.
That plane has a surface normal which illustrated here is l.
And what is l?
L is a vector perpendicular to these rays.
As recalled, in the previous slide, this l is exactly a parameter represent a, b,
and c, represent this line that we see in the space.
So this is the intuition that we're going to follow, that is,
given two points, x and x prime,
we define a line as the homogeneous coordinates of the lines,
simply as nothing but the x crossed with x prime.
Where both x is a three-dimensional ray, x prime is a three-dimensional ray,
x crossed with x given a vector which is perpendicular to both and
is a three-dimensional vector.
And this happened to correspond to the line equation that we see in the image.
And we can prove this to us, ourself, algebraically,
iIf I do the following form.
Let's say, I define the line as x cross, x prime, which is defined.
And we can verify that a point x
is on this line, which says x dot with this line l,
we can compute it as, x dot with x cross, x prime.
And that's equal to 0.
Similarly we can do that for x prime when we take a dot problem between x prime
with the line represented by x cross, x prime is also a 0.
That means, both the point x and
x prime sits on this construction of line as defined as x cross, x prime.
So, the beauty of this equation is that when we try to
find the vanishing points in the image, we no longer need to take this ruler out,
meticulously trace lines and measure where they intersect.
And if you're lucky, the intersection sits in the image, and if you're not lucky,
the intersection might be very far away from the images.
Now we have way of, a simple way, of computing where the intersection is,
by simply taking two points' measurements in image, x and x prime.
Think of those as not points anymore in 2D, but
in terms of rays in 3D space, adding one more coordinate to it.
And the cross part of between them.
And we now obtain a line.
And later when we see this lines can intersect
just the same way we see points intersect.
A point connect to a lines forming a point of intersection.
How do we compute a cross product?
Just a review.
We can follow a set of equations to compute a cross product between
three dimensional vectors, two three dimensional vectors.
Cross with each other, obtain another three dimensional vector.
In Matlab this is, in fact, simple to do.
There's a building function called a cross product and we can build a function
to take a formation of a line at two points.
As this follow, three lines of code allow you to do it.
Connection between two points into a line.
So we take points, again, in term of x,y locations.
We tack on one more dimension to it, 1,
to turn it into a ray in the three-dimensional space.
We take x, second x and y and we tack 1 to it, to second array.
We simply cross, it's a built in function in Matlab, the two arrays into a line.
The line's made of three parameters a, b and
c, which indicating a surface form over the plane formed by their line and origin.
And to visualize the line,
quite often what we do is normalize the line such that the two coefficients a and
b has norm one, therefore they are reading as cosine and sine in the image.
Let's go through some exercises.
Here we have a point, P2, and we have a line in question as illustrated here.
Just as a sanity check, we can plug in the parameters of x,
y of that point P2, and multiply by a and b.
We see, they should obtain as minus c.
This is a very close estimate.
Now we have two points on P1 and P2 that we clicked on the image.
And if we follow the previous routine, we can cross the two points
in the three-dimensional rays, forms a vector perpendicular to them.
And that vector is corresponding to the line which is shown here.
We can do that again for another two points.
Take two measurements in the image, click on two points, and
the two points will define the line.
And we can directly compute the line by taking a cross
product between the two points.
So, so far, we have defined how do we intersect.
Now, intersect,
we have to find a way of completing a line by looking at two points on a line.
Let's take a little moment to think about it.
We had said, cross product between two points give you a vector,
which is the definition of a line.
What if the line had the following forms, a, b, and 0?
What would that mean?
That means that a and b are vectors in a plane.
Sitting in a plane passing through the origin.
And the plane is exactly frontal parallel to the image plane that we're looking at.
So in fact, the vector is in this direction.
So what does it say about the lines?
Those are lines exactly in the image which are pivoting around the center of
the image, and has distance 0 wave on it, origin,
which means it goes exactly through the center of the origin and
the x, y space in the image.
Okay, so that's good to know.
How about the lines that have the coordinate system of 0, 0, 1?
So who are those lines?
Let's think about it.
Where is 0, 0, 1 vector pointing at?
It is pointing straight ahead.
So, a plane that's surface normal is pointing straight ahead,
what does it mean the plane itself?
Well that means the plane is perpendicular to the image plane.
Well, where does that lie?
To think about it, imagine the points not pointing at 0,0,1, but slightly off.
So, in fact, it has some coordinates which is not 0 in the first three elements.
I can see that's a plane which is tilting just slightly.
Such that it will intersect with the image plane at very, very far away.
So we can think of, when this tilting plane becomes eventually parallel
to the image plane, the intersection, in fact, happens at infinity.
And this is going to cross the line at infinity, and
corresponding to the horizon that we will see.
This is the horizon of the image plane, but
not the horizon that we're seeing of the horizon in the picture itself.
So now, we are understanding how two points form a line.
Now we want to see how two lines can form a point, intersect at a point.
So we can say, well, we can look at a high school geometry book,
given two lines in an image or two lines on a plane.
How do we find the intersection?
And that's a matter of solving an equation.
But yet, there's a simple way of thinking about it when we take a step
into the homogeneous coordinate system.
Again, looking at image not only as a plane but
as a three-dimensional object sitting at some distance away from the origin itself.
Think of again, lines are no longer just lines of an image.
A line is a set of a plane going through the origin and
that red line, it's going to form a plane with origin.
And that plane has a surface normal which is sticking out, indicating red.
And this point P,
is going to be intersecting between the red line and the green line.
So clearly, the P It's sitting in that plane, perpendicular to this L1.
So we know that L1 and P, whatever that P is, is going to be perpendicular.
So that's the first concept.
Again, looking at P sitting on the green line, the same point.
But if that's sitting on the green line, it means that the the plane formed by
the green line and origin, is perpendicular to the green surface normal.
So that means that P, if you think about sitting in a green line,
it's going to depict perpendicular to a green surface normal.
So what does it say?
It says, this ray, going from the origin to the intersection of two lines,
P is perpendicular to both L1 and L2,
the red surface normal and the green surface normal.
So if I were thinking there's two surface normal, one green,
one red, we're illustrating two planes.
And this ray P is sticking away from me, straight out.
That P is perpendicular to both of them, that means that if I form
a plane between L1, and L2, that P is perpendicular to that plane.
That's a little bit strange, but think about it.
If you work through the derivations a few times, it become clear to you.
With our intuition we know that the P which is a ray going from the origin
to the image, goes through the intersection between two lines.
Is in fact perpendicular to the two surface normals.
And that is written down in the algebra form is following,
that form P is intersection between two lines and
this can be thought of the as the cross product between L1 in red,
L2 in green and P is just the ray going from origin to the image.
And that ray is going to be perpendicular to both.
In fact, that's the point of intersection between two lines.
So with homogeneous coordinates,
we no longer had to go back to the old way of doing intersection between lines.
If I were to ask you how to intersect two lines, you simply write down to the line
equation a, b, and c, and a prime, b prime, c prime.
And I would just simply cross the two vectors.
You will see, you get an intersection back at the three dimensional elements,
which is a ray.
And that ray corresponding to the point of intersection on the image plane.
So when you need to do it in Matlabs, simply plugging in two lines, or
one or two.
Simply cross product between the two vectors, cross is building functions.
And if I want to visualize that point, in the image plane itself, recall,
I need to normalize by the third element, set the third element to be 1, so
I can measure that in image.
It's I were just doing computation, I can just carry out on
this three-dimensional homogeneous cord without a normalizing.
So here I have a set of illustrations, the same picture we saw before.
I click on three points, x1, x2, x3 where the intersection point is marked x2.
And now I can compute the intersection between two points, x1, x2,
to form a line, x2 cross x3, and a second line.
Two lines we can cross one more time to see if we indeed intersected x2.
And this derivation shows indeed we have those derivations.
So those are the concepts in which is used points as
a three-dimensional rays, lines as three dimensional planes.
And such, a point in the image has three-dimensional coordinates,
as well as lines at three dimensional coordinates.
