Archive for the ‘Math’ Category

Arranging a Beta Distribution into Exponential Family Form

Sunday, November 20th, 2016

A family of PMFs or PDFs is an exponential family if it can be arranged into the form

$f(x|\theta) = h(x)c(\theta)\exp\left(\sum_{i=1}^kw_i(\theta)t_i(x)\right)$

where $\theta$ is the vector of parameters.

The Beta distribution PDF takes the form

$f_X(x) = \frac{x^{\alpha – 1}(1 – x)^{\beta – 1}}{\textrm{B}(\alpha, \beta)}$

Given that the $\alpha$ and $\beta$ parameters are unknown, we now arrange $f_X(x)$ into an exponential family form:

$f(x|\alpha,\beta) = \frac{x^{\alpha – 1}(1 – x)^{\beta – 1}}{\textrm{B}(\alpha, \beta)}$
$= \frac{e^{(\alpha – 1)\ln(x)}e^{(\beta – 1)\ln(1 – x)}}{\textrm{B}(\alpha, \beta)}$
$= \frac{e^{(\alpha – 1)\ln(x) + (\beta – 1)\ln(1 – x)}}{\textrm{B}(\alpha, \beta)}$
$= \frac{1}{\textrm{B}(\alpha, \beta)}e^{(\alpha – 1)\ln(x) + (\beta – 1)\ln(1 – x)}$

The log identity $x^b = e^{b\ln(x)}$ is a very useful logarithmic identity to remember when trying to arrange PDFs into exponential family form.

We observe:
$h(x) = I_{x \in (0,1)}(x)$ (If you see that h(x) = 1, that is a cue to use an indicator function that ranges through the support of $x$.)
$c(k,\beta) = \frac{1}{\textrm{B}(\alpha, \beta)$
$w_1(k,\beta) = \alpha – 1$
$w_2(k,\beta) = \beta – 1$
$t_1(x) = \ln(x)$
$t_2(x) = \ln(1-x)$

Hence, the Beta distribution given unknown parameters $\alpha$ and $\beta$ is an exponential family with a two-dimensional parameter vector $\theta$.

A similar process will apply for showing that a Beta PDF with one unknown parameter, $\beta$ or $\alpha$ is an exponential family.

Arranging a Gamma Distribution Family into Exponential Family Form

Sunday, November 20th, 2016

A family of PMFs or PDFs is an exponential family if it can be arranged into the form

$f(x|\theta) = h(x)c(\theta)\exp\left(\sum_{i=1}^kw_i(\theta)t_i(x)\right)$

where $\theta$ is the vector of parameters.

Now, the shape-scale parameterization of the Gamma distribution PDF takes the form

$f_X(x) = \frac{1}{\Gamma(k)\beta(k)}x^{k-1}e^{-\frac{x}{\beta}}$

$\beta$ is typically used as the symbol for the rate parameter in the shape-rate parameterization of the Gamma PDF, but the default symbol for the scale parameter, $\theta$, would conflict with the symbol for our parameter vector.

Can we arrange that PDF into an exponential family form? Spoiler: yes.

Here, we demonstrate that a Gamma PDF given two unknown parameters, $\beta$ and $k$, is an exponential family.

$f(x|k,\beta) = \frac{1}{\Gamma(k)\beta(k)}x^{k-1}e^{-\frac{x}{\beta}}$
$ = \frac{1}{\Gamma(k)\beta(k)}e^{(k-1)\ln(x)}e^{-\frac{x}{\beta}}$
$ = \frac{1}{\Gamma(k)\beta(k)}e^{(k-1)\ln(x) – \frac{x}{\beta}}$

The log identity $x^b = e^{b\ln(x)}$ is a very useful logarithmic identity to remember when trying to arrange PDFs into exponential family form.

We observe:
$h(x) = I_{x>0}(x)$ (If you see that h(x) = 1, that is a cue to use an indicator function that ranges through the support of $x$.)
$c(k,\beta) = \frac{1}{\Gamma(k)\beta(k)}$
$w_1(k,\beta) = k – 1$
$w_2(k,\beta) = -\frac{1}{\beta}$
$t_1(x) = \ln(x)$
$t_2(x) = x$

Hence, the Gamma distribution given unknown parameters $\beta$ and $k$ is an exponential family with a two-dimensional parameter vector $\theta$.

A similar process will apply for showing that a Gamma PDF with one unknown parameter, $\beta$ or $k$ is an exponential family.

Matrix Diagonalization to Solve a Differential Equation

Saturday, January 2nd, 2016

Download (PDF, 82KB)