Introduction

In this eighth edition of our Bayesian series, we delve into a refined class of noninformative priors, specifically for location and scale parameters. This builds directly upon our previous edition, where we introduced the Jeffreys prior and principles of noninformativeness in Bayesian statistics.

Here, we demonstrate how certain parameter types naturally lead to standard forms of noninformative priors, with a focus on interpretation, invariance, and practical implications.

1. Location Parameters

A location parameter \(\theta\) appears in models of the form:

\(p(y \mid \theta) = f(y – \theta)\)

This structure implies that the distribution is shift-invariant, only the difference between the observation and the parameter matters.

Noninformative Prior for Location

To achieve a posterior distribution that is independent of both \(\theta\) and \(y\), we require:

\(p(\theta) \displaystyle \propto 1\)

This is a uniform prior over the real line. Though technically improper (since its integral diverges), it can still result in a proper posterior under suitable likelihood conditions.

2. Scale Parameters

A scale parameter \(\theta\) enters a model via:

\(p(y \mid \theta) = \frac{1}{\theta} f\left(\frac{y}{\theta}\right)\)

The density is invariant under rescaling of the observations.

Noninformative Prior for Scale

We want the posterior to reflect only the shape \(g(y/\theta)\), implying:

\(p(\theta) \displaystyle \propto \displaystyle \frac{1}{\theta}\)

This is the Jeffreys prior for scale parameters, reflecting the invariance under reparameterizations like \(\log(\theta)\). Alternative representations include:

  • \(p(\log(\displaystyle \theta)) \displaystyle \propto 1\)

  • \(p(\theta^2) \displaystyle \propto \displaystyle \frac{1}{\theta^2}\)

⚠️ As with location priors, these are often improper priors, so care must be taken to ensure the resulting posterior is proper.

3. Example: Normal Distribution with Unknown Mean and Variance

Let \(Y \sim \mathcal{N}(\mu, \sigma^2)\), where both parameters are unknown.

Choosing a Noninformative Prior

  • \(\mu\) is a location parameter → \(p(\mu) \displaystyle \propto 1\)

  • \(\sigma^2\) is a scale parameter → \(p(\sigma^2) \displaystyle \propto 1/\sigma^2\)

Thus, a standard joint prior is:

\(p(\mu, \sigma^2) \displaystyle \propto \displaystyle \frac{1}{\sigma^2}\)

This is widely used in Bayesian inference for Gaussian models with unknown parameters and leads to analytically tractable posteriors.

Conclusion

In this edition, we established that:

  • Uniform priors are suitable for location parameters

  • Inverse priors like \(1/\theta\) are appropriate for scale parameters

  • These forms emerge naturally from invariance considerations and can be interpreted as special cases of Jeffreys priors

In our next edition, we’ll dive into Quiz on Bayesian Fundamentals (Part I).

Stay tuned with 3 D Statistical Learning as we make Bayesian statistics both intuitive and applicable.