Previously, we looked at

information entropy when the probability is uniformly distributed.

So this was the Hartley's construction in 1928,

Hartley's formula in 1928 for uniform distribution case.

Again, that's where the uniform distribution comes from.

It comes from the modeling of the outcome distribution.

Now, what happens when this is not true when properly distribution is not uniform?

So, let's investigate that.

Let's look into that.

And we will look at a more generalized equation

to express information entropy.

Let's actually build on

what we have on

this sheet and we're going to actually build on this example of weather in Gotham City.

We will be using the same weather information as the random phenomenon,

but let's use a different distribution.

Let's actually use a different city altogether.

I'm just going to get rid of some of the parts that are less relevant.

And this was the uniform case.

And we'll be using the same variable definitions

with small n being the number of independent events.

Instead of Gotham City,

which has a very random weather,

let's use Colorado Springs.

And Colorado is known to be sunny.

So there will be a bias towards the sunny weather.

Again, let's suppose in Colorado Springs,

the weather is either sunny,

rainy, snowy or cloudy.

Let's make that probability distribution non-uniform.

And again, as I mentioned,

Colorado itself is famous for being a sunny state, a sunny Colorado.

So let's suppose that the probability of Colorado Springs being sunny is

half and the probability

of the weather being rainy is 0.125 or 1/8,

the probability of the weather being snowy is also 1/8 or 0.125,

the probability of being cloudy is 0.25 or a quarter.

And these four probabilities will sum up to one.

Similarly to what we did before,

we'll ask the question about how many bits are needed to

communicate the weather in Colorado Springs.

So it's no longer Gotham City,

but it will be Colorado Springs.

And we would want to make

this communication or the bit transfer as efficient as possible.

We can keep this way of encoding,

however, such encoding scheme is not going to be efficient.

Let's actually choose another encoding scheme which is more efficient.