diff --git a/_psets/3.md b/_psets/3.md index ff70eee3a209ee1e1661d2dce24c57d59ea21ff6..66e68bc102eca73a851f53a47958a7efd55d7614 100644 --- a/_psets/3.md +++ b/_psets/3.md @@ -335,6 +335,35 @@ rates. {:.question} Calculate the differential entropy of a Gaussian process. +Since we're integrating the Gaussian over the whole real line, translation is irrelevant. So without +loss of generality I'll calculate the differential entropy of a Gaussian with zero mean. + +$$ +\begin{align*} +H(N(\mu, \sigma^2)) +&= -\int_{-\infty}^\infty \frac{1}{\sqrt{2 \pi \sigma^2}} e^{\frac{-x^2}{2 \sigma^2}} \log \left( + \frac{1}{\sqrt{2 \pi \sigma^2}} e^{\frac{-x^2}{2 \sigma^2}} \right) \mathrm{d} x \\ +&= -\int_{-\infty}^\infty \frac{1}{\sqrt{2 \pi \sigma^2}} e^{\frac{-x^2}{2 \sigma^2}} \left( + -\frac{1}{2} \log(2 \pi \sigma^2) - \frac{x^2}{2 \sigma^2} \right) \mathrm{d} x \\ +&= \frac{1}{2} \log(2 \pi \sigma^2) \int_{-\infty}^\infty \frac{1}{\sqrt{2 \pi \sigma^2}} + e^{\frac{-x^2}{2 \sigma^2}} \mathrm{d} x + \int_{-\infty}^\infty \frac{1}{\sqrt{2 \pi \sigma^2}} + \frac{x^2}{2 \sigma^2} e^{\frac{-x^2}{2 \sigma^2}} \mathrm{d} x \\ +&= \frac{1}{2} \log(2 \pi \sigma^2) + \frac{1}{2} \int_{-\infty}^\infty + \left( \frac{x}{\sqrt{2 \pi \sigma^2}} \right) \left( \frac{x}{\sigma^2} + e^{\frac{-x^2}{2 \sigma^2}} \right) \mathrm{d} x \\ +&= \frac{1}{2} \log(2 \pi \sigma^2) + \frac{1}{2} \int_{-\infty}^\infty + \frac{1}{\sqrt{2 \pi \sigma^2}} e^{\frac{-x^2}{2 \sigma^2}} \mathrm{d} x \\ +&= \frac{1}{2} \log(2 \pi \sigma^2) + \frac{1}{2} \\ +&= \frac{1}{2} \log(2 \pi e \sigma^2) +\end{align*} +$$ + +The second term was integrated by parts. Note that + +$$ +\frac{d}{dx} e^{\frac{-x^2}{2 \sigma^2}} = -\frac{x}{\sigma^2} e^{\frac{-x^2}{2 \sigma^2}} +$$ + ## (4.5)