Tag Archives: Transformation

Transforming random variables

In certain cases, it is useful to transform a random variable \Theta to another random variable \Psi via a transformation \Psi=f(\Theta). For instance, this situation arises typically if one wants to sample from the probability density function p_{\scriptscriptstyle{\Theta}}(\theta) of a positively-valued random variable \Theta. Markov chain Monte Carlo (MCMC) algorithms are conventionally designed to draw correlated samples from a desired target (density) p_{\scriptscriptstyle{\Psi}}(\psi) of a random variable \Psi taking values over the real line. So, if the target of interest p_{\scriptscriptstyle{\Theta}} has support over the positive real line and the MCMC algorithm samples from a density p_{\scriptscriptstyle{\Psi}} with support over the real line, then a random variable transformation, such as \Psi=\log{(\Theta)}, can help resolve the matter. In particular, the transformation allows to sample from p_{\scriptscriptstyle{\Psi}} via the MCMC method of choice and then the inverse transformation \Theta=\exp{(\Psi)} converts the simulated Markov chain to a set of sample points from p_{\scriptscriptstyle{\Theta}}. Obviously, it is needed to find the form of the target density p_{\scriptscriptstyle{\Psi}} on which MCMC will be applied.

Although such random variable transformations are common practice, one may need to look up the formula for the transformation to pass from the original density p_{\scriptscriptstyle{\Theta}} to the transformed density p_{\scriptscriptstyle{\Psi}}. The main source of confusion is whether one needs the Jacobian associated with the transformation f or with the inverse transformation f^{-1}.

There is a way to retrieve the formula intuitively via a geometric argument, rather than trying to uncover it mnemonically. The main argument is that of area preservation in the case of univariate random variables. It suffices to realize that for a small displacement, the area below the curves of the two densities is the same, which means that


This realization suffices to recover the remaining steps. It follows that


Notice that

\Theta \overset{f}{\underset{f^{-1}}\rightleftarrows} \Psi ,

which gives the transformed density


The Jacobian in the univariate case is the derivative \frac{df^{-1}(\psi)}{d\psi}, associated with the inverse transformation f^{-1}. The absolute value of the derivative ensures that the density p_{\scriptscriptstyle{\Psi}} is non-negative. Having understood the univariate case, the multivariate scenario follows straightforwardly as

p_{\scriptscriptstyle{\boldsymbol\Psi}}(\boldsymbol\psi)=p_{\scriptscriptstyle{\boldsymbol\Theta}}(f^{-1}(\boldsymbol\psi))\left|\frac{\partial f^{-1}_{i}(\boldsymbol\psi)}{\partial_{j}\boldsymbol\psi}\right|,

where \left|\frac{\partial f^{-1}_{i}(\boldsymbol\psi)}{\partial_{j}\boldsymbol\psi}\right| denotes the determinant of the Jacobian of f^{-1}.

To follow through with the example

\Theta \overset{\log}{\underset{\exp}\rightleftarrows} \Psi ,

notice that f=\log, f^{-1}=\exp. So, the derivative \frac{df^{-1}(\psi)}{d\psi} becomes

\displaystyle\frac{df^{-1}(\psi)}{d\psi}=\frac{d \exp (\psi)}{d\psi}=\exp{(\psi)},


p_{\scriptscriptstyle{\Psi}}(\psi)=p_{\scriptscriptstyle{\Theta}}(\exp{(\psi)}) \exp{(\psi)}.

The target log-density for MCMC is thus

\log{(p_{\scriptscriptstyle{\Psi}}(\psi))}=\log{(p_{\scriptscriptstyle{\Theta}}(\exp{(\psi)}))} + \psi.