Significance level and P-value are related to the Type I error of Hypothesis testing, hence lets go back to basics.
Hypothesis testing is used to statistically validate a theory defined for a sample or among multiple samples. Unlike common belief, hypothesis testing does not help the researcher to choose between two options but researcher will always stick to null hypothesis unless there is significant evidence that alternate hypothesis is true.
Null Hypothesis (H0) is general ‘no effect’ hypothesis
Alternate Hypothesis (H1) is the condition that researcher is trying to prove right.
In hypothesis testing, there are two possible errors
1. Type I error: Rejecting null hypothesis when it is true.
For example, we are trying to validate if Average handling time (AHT) has improved post conducting refresher training.
Null Hypothesis: Refresh training has no effect on (AHT)
Alternate Hypothesis: AHT has improved post conducting refresher training.
Here, If a researcher is making type I error, s/he is acting assuming there is an effect. In our AHT example, company will start investing more on conducting refresher trainings assuming it will improve AHT.
Hence this error also referred as Producer’s risk.
The chances of making Type I error is denoted using α (alpha) symbol, which is the level of significance set for the hypothesis testing.
If α is 0.05, this means researcher is willing to accept 5% chances that s/he is wrong in rejecting the null hypothesis.
Statistical analysis generates a probability denoted as ‘p-value’ which refers the likelihood of rejecting the null hypothesis when it is correct.
Hence if the p-value is greater than α, we fail to reject null hypothesis considering that is the safest option for producer and we do not have enough evidence to prove that alternate theory is correct.
Now since we are taking about errors, let me complete article by adding bit explanation of Type II error as well.
2. Type II error: Failing to reject the null hypothesis when it is false.
Type II error is Consumer’s risk because even though there was an effect, the producer did not act on it hence the consumer did not get the better results.
The probability of making type II error is referred as β(Beta). The power of the test is in giving the correct result, i.e., accepting alternate hypothesis when null is false because that was the whole objective of conducting a hypothesis test. Hence the power of the test is referred using (1-β).