Hypothesis Testing

Hypothesis testing the mathematical way of saying "We have enough statistical evidence to say this statement is true".

At its core, hypothesis testing is a statistical method used to determine whether there is enough evidence in a sample of data to infer that a certain condition holds true for an entire population.

Step by step process

1. Setting up the hypothesis

Every test require two hypothesis to be defined.

1. Null Hypothesis ($H_0$) This hypothesis says nothing changed, no effect happened, everything is as it was before.
2. Alternative Hypothesis ($H_1$ or $H_A$) This is the actual hypothesis you trying to prove. This says the thing has changed, there is a effect, it is different from what it was before.

Example: You did a marketing campaign for a restaurant.

• $H_0$: The campaign did not increase sales ($\mu_{\text{campaign sample}} = \mu_{restaurant}$).
• $H_A$: The campaign increased sales ($\mu_{\text{campaign sample}} > \mu_{restaurant}$).

2. Declaring Rejection threshold

This threshold is known as the Significance level ($\alpha$).

Before looking at the data, we decide how strong the evidence must be to reject $H_0$. This threshold is called the **significance level ($\alpha$)

• Standard practice sets $\alpha = 0.05$ ($5\%$).
• This means you are willing to accept a $5\%$ risk of wrongly rejecting $H_0$ when it was actually true.

Why set it before hand? This is done for authenticity. Imagine you get a $\alpha \approx 0.049$ and you say its under $0.05$ so the hypothesis is valid. But someone can argue that $\alpha$ is reached by P-Hacking since its so close to $0.05$ Also difference fields require different threshold levels due to the importance of the hypothesis if it is valid. Ex: A medical research hypothesis related to "A cure for cancer" may require the $\alpha$ to be $< 0.01$

3: Collect Data & Calculate a Test Statistic

You collect sample data and calculate a test statistic (like a $z$-score, $t$-score, or $F$-statistic). This measures how far your sample results fall from what $H_0$ predicted.

4: Find the $p$-value

The $p$-value is the probability of getting your sample results (or more extreme) if the Null Hypothesis ($H_0$) were true.

• Small $p$-value: The observed data is extremely unlikely under $H_0$.
• Large $p$-value: The observed data easily happens by random chance under $H_0$.

5: Make a Decision

Compare your $p$-value to your significance level ($\alpha$):

[ Hypothesis ] ──> [ Controlled Experiment ] ──> [ Replication ] ──> [ Consensus ]

6. Errors in hypothesis Testing

Since we are not $100\%$ certain the Null Hypothesis is wrong/correct, (i.e. Probability of Null hypothesis is not $0\%$ or $100\%$). There is a $\alpha$ probability the null hypothesis will be true. In that case, the test fails with error.

Type I Error ($\alpha$)

Rejected Null Hypothesis, but Null hypothesis is true. (False Positive) This is the probability that our hypothesis is actually wrong.

The probability of Type I Error is $\alpha$, which is usually $0.05\%$ which is also known as the "Significance level".

Type II Error ($\beta$)

Failed to reject Null hypothesis, but Null hypothesis is actually wrong. This is the probability that our hypothesis will be correct even though we deduced its not. There was a difference made but we think it didn't.

The probability of Type II Error is $\beta$

The Statistical Power is known as ($1- \beta$). This is the probability of correctly rejecting the null hypothesis when its actually wrong.

Relationship of $\alpha$ and $\beta$

To reduce both, the best option is to increase the sample size ($n$).

Why Alpha and Beta Inverse in Hypothesis errors