But now you have repeated this procedure several times for
nine hidden units in layer 1.
What if you do this for
some of the hidden units in the deeper layers of the neuron network.
And what does the neural network then learning at a deeper layers.
So in the deeper layers, a hidden unit will see a larger region of the image.
Where at the extreme end each pixel
could hypothetically affect the output of these later layers of the neural network.
So later units are actually seen larger image patches,
I'm still going to plot the image patches as the same size on these slides.
But if we repeat this procedure, this is what you had previously for layer 1,
and this is a visualization of what maximally activates nine different
hidden units in layer 2.
So I want to be clear about what this visualization is.
These are the nine patches that cause one hidden unit to be highly activated.
And then each grouping, this is a different set of nine image patches that
cause one hidden unit to be activated.
So this visualization shows nine hidden units in layer 2, and
for each of them shows nine image patches that causes that hidden unit
to have a very large output, a very large activation.
And you can repeat these for deeper layers as well.
Now, on this slide, I know it's kind of hard to see these tiny little
image patches, so let me zoom in for some of them.
For layer 1, this is what you saw.
So for example, this is that first unit we saw which was highly activated, if
in the region of the input image, you can see there's an edge maybe at that angle.
Now let's zoom in for layer 2 as well, to that visualization.
So this is interesting,
layer 2 looks it's detecting more complex shapes and patterns.
So for example, this hidden unit looks like it's looking for
a vertical texture with lots of vertical lines.
This hidden unit looks like its highly activated when
there's a rounder shape to the left part of the image.
Here's one that is looking for very thin vertical lines and so on.
And so the features the second layer is detecting are getting more complicated.
How about layer 3?
Let's zoom into that, in fact let me zoom in even bigger, so
you can see this better, these are the things that maximally activate layer 3.
But let's zoom in even bigger, and so this is pretty interesting again.
It looks like there is a hidden unit that seems to respond highly
to a rounder shape in the lower left hand portion of the image, maybe.
So that ends up detecting a lot of cars, dogs and
wonders is even starting to detect people.
And this one look like it is detecting certain textures like honeycomb shapes,
or square shapes, this irregular texture.
And some of these it's difficult to look at and manually figure out what is it
detecting, but it is clearly starting to detect more complex patterns.
How about the next layer?
Well, here is layer 4, and you'll see that the features or
the patterns is detecting or even more complex.
It looks like this has learned almost a dog detector, but
all these dogs likewise similar, right?
Is this, I don't know what dog species or dog breed this is.
But now all those are dogs, but they look relatively similar as dogs go.
Looks like this hidden unit and therefore it is detecting water.
