Unit 2 - Foundation of Inference

The trinity of statistical inference: estimation, confidence intervals and testing.

• Estimator: one value whose performance can be measured by consistency, asymptotic normality, bias, variance and quadratic risk.
• Confidence intervals provide “error bars” around estimators. Their size depends on the confidence level.
• Hypothesis testing: we want to ask a yes/no answer about an unknown parameter. They are characterized by hypotheses, level, power, test statistic and rejection region. Under the null hypothesis, the value of the unknown parameter becomes known (no need for plug-in).

Statistical model

Formal definition:

Let the observed outcome of a statistical experiment be a sample of n i.i.d. random variables in some measurable space (usually ) and denote by their common distribution. A statistical model associated to that statistical experiment is a pair:

where:

• is called sample space
• is a family of probability measured on E
• is any set , called parameter set.

For example: the statistical model of Bernoulli distribution:

Parametric, nonparametric and semiparametric models

Usually, we will assume that the statistical model is well specified, i.e., defined such that , for some . This particular is called the true parameter, and is unknown: The aim of the statistical experiment is to estimate , or check it’s properties when they have a special meaning.

• if for some , the model is called parametric
• if is infinite dimensional, the model is called nonparametric
• if , where is finite dimensional, and is infinite dimensional, then the model is called semiparametric.

Identifiability

The parameter is called identifiable iff the map is injective, i.e.:

or equivalently:

Estimation

A statistic is any measurable function of the sample.

An estimator of is a statistic whose expression does not depend on .

• An estimator of is weakly (resp. strongly) consistent if
• An estimator of is asymptotically normal if . The quantity is then called asymptotic variance of .

Bias of an estimator of :

If , we say that is unbiased.

We want estimators to have low bias and low variance at the same time.

The Risk (or quadratic risk) of an estimator is

which means:

For example: for Bernoulli distribution , using as an estimator for , this estimator is unbiased, consistent, and its quadratic risk tends to 0 as the sample size .

Confidence Intervals

Let be a statistical model based on observations and assume . Let .

• Confidence interval (C.I.) of level for : any random (depending on ) interval whose boudnaries do not depend on and such that:

• C.I. of asymptotic level for : any random interval whose boundaries do not depend on and such that:

Be aware that it is , not .

For example: for Bernoulli distribution , using as an estimator for , and from CLT:

For a fixed , if is the )-quantile of , then with probability (if is large enough),

It yields:

But it is not a confidence interval, because it depends on p !! Three solutions are presented below.

Conservative bound

Since , roughly with probability at least ,

Indeed:

Solving the (quadratic) equation for p

From

we can get

We need to find the roots of

This leads to , such that: .

Plug-in

This method uses the estimated to calculate the variance.

By LLN: , and by Slutsky:

This leads to:

such that:

Meaning of confidence interval

There is a frequentist interpretation:

95% C.I. means if we were to repeat the experiment then the true parameter would be in the resulting confidence interval about 95% of the time.

It is wrong to say that

By 95% of chance that the true parameter is in the resulting confidence interval

Because from the frequentists’ point of view, the true parameter is deterministic (fixed, even though unknown). Once the confidence interval is calculated, we can only say that the true parameter is in the C.I. or not, like a Bernoulli distribution, only 1 or 0 is taken. But I suppose we can say that:

The expectation of that Bernoulli distribution is 95%.

Steps to find a confidence interval

1. Find an estimator for for
2. Determine the (asymptotic) distribution of
3. Compute a confidence interval for based on with level

Delta method

Exponential distribution example (1/2)

Take Exponential distribution as an example, PDF: .

Let , and its sample mean: . By LLN: , because .

So a natural estimator of is:

Hence: .

Be careful that, .

By CLT:

How does the CLT transfer to ? How to find an asymptotic confidence interval for ? Here we need to use the Delta method.

The Delta method

Let sequence of r.v. that satisfies

for some and (the sequence is said to be asymptotically normal around ).

Let be continuously differentiable at the point . Then, is also asymptotically normal around ; More precisely:

Exponential distribution example (2/2)

By using the delta method, ,

To calculate the asymptotic confidence interval for :

Then we can use “Solve” or “Plug-in” method to get confidence interval for .

Hypothesis testing

Statistical formulation

• Consider a sample of i.i.d. random variables and a statistical model
• Let and be disjoint subsets of
• Consider the two hypotheses:
• is the null hypothesis, is the alternative hypothesis
• If we believe that the true is either in or in , we may want to test against
• We want to decide whether to reject (look for evidence against in the data)

Asymmetry in the hypotheses

• and in do not play a symmetric role: the data is is only used to try to disprove
• In particular lack of evidence, does not mean that is true (“innocent until proven guilty”)
• A test is a statistic such that:
  • If , is not rejected
  • If , is rejected

Errors

• Rejection region of a test :

• Type I error of a test (rejecting when it is actually true):

• Type II error of a test (not rejecting although is actually true):

• Power of a test :

Level, test statistic and rejection region

• A test has level if:

• A test has asymptotic level if:

• In general, a test has the form:

  for some statistic and threshold

• is called the test statistic. The rejection region is

One-sided vs two-sided tests

We can refine the terminology when and is of the form:

• If : two-sided test
• if or : one-sided test

p-value

The (asymptotic) p-value of a test is the smallest (asymptotic) level at which rejects . It is random, it depends on the sample.

is rejected by , at the (asymptotic) level

The smaller the p-value, the more confidently one can reject .

Steps of hypothesis testing

1. Find estimators
2. Find pivot and determine the distribution of pivot. Write some statistic , and let

It is pivot if we can manage to write it down in such a way that it’s distribution under the null hypothesis is known and does not depend on any additional parameters.

3. Adjust to match level .