Applying Tests of Statistical Significance: A Practical Guide for Social Science Research

A Practical Framework for Statistical Significance in Social Research

Statistical inference forms the backbone of empirical social science. It provides a systematic method for moving beyond describing a sample to making claims about the population from which it was drawn. This process hinges on tests of statistical significance, which quantify the probability that an observed effect occurred by random chance alone. For students and researchers, mastering this framework is less about complex mathematics and more about applying a consistent, logical procedure. This paper walks through that procedure using a series of mock studies, demonstrating how to formulate hypotheses, execute analyses in SPSS, and draw substantive conclusions from the results. The goal is to build a practical, actionable skill set for evaluating social science data.

The consistent application of a five-step hypothesis testing procedure ensures clarity and rigor in your analysis. You first state your research hypothesis and its corresponding null hypothesis. The research hypothesis posits the effect you expect to find, while the null hypothesis asserts that no effect exists. You then identify your significance level, alpha, which sets the threshold for deciding against the null hypothesis. A common alpha is .05. Next, you conduct your statistical analysis using software like SPSS. The output provides a p-value, the probability of obtaining your results if the null hypothesis were true. Finally, you compare this p-value to your alpha. If the p-value is less than or equal to alpha, you reject the null hypothesis. This decision means the observed effect is statistically significant and unlikely to be due to random sampling error.

Independent t-Test: Evaluating a Workplace Intervention

Mock Study 1 investigates whether counseling improves job satisfaction after an industrial accident. The research hypothesis is that the mean job satisfaction score for employees who participated in counseling is different from the mean for those who did not. The null hypothesis states the two group means are equal. We set alpha at .01. After running an independent-samples t-test in SPSS, the output shows a p-value for the equality of means. Suppose this p-value is .005. This value is less than the alpha of .01. We therefore reject the null hypothesis. The difference in job satisfaction scores between the two groups is statistically significant at the .01 level. Based on this result, you would recommend counseling as a method to improve job satisfaction following industrial accidents. The analysis suggests the observed benefit is not a random fluctuation.

Dependent t-Test: Measuring Change Over Time

Mock Study 2 examines the impact of group therapy on activities of daily living for depressed clients. This design uses a pre-test and post-test on the same individuals, requiring a dependent-samples t-test. The research hypothesis is that the number of activities after therapy is different from the number before therapy. The null hypothesis states the mean difference between pre-test and post-test scores is zero. We set alpha at .05. The SPSS output for the paired-samples test provides a p-value for the mean difference. If this p-value is .002, it is less than .05. We reject the null hypothesis. The increase in activities from before therapy to after therapy is statistically significant. You would conclude that group therapy is associated with a positive change in daily functioning for depressed clients, supporting its broader recommendation.

One-Sample t-Test: Comparing to a Known Benchmark

Mock Study 3 also concerns activities of daily living after therapy, but the comparison is different. Here, we test whether the post-therapy average for a sample differs from a known population average of 15. The research hypothesis is that the sample mean is different from 15. The null hypothesis states the sample mean is equal to 15. Alpha is set at .05. The one-sample t-test in SPSS compares the sample data to the test value of 15. The output yields a p-value. Assume the p-value is .125. This value is greater than .05. We fail to reject the null hypothesis. The analysis indicates the average number of activities for clients who underwent therapy is not statistically different from the typical average for depressed people. You would be cautious about recommending group therapy based on this specific test, as the evidence for an effect is weak.

Analysis of Variance: Comparing Multiple Groups

Mock Study 4 asks whether TV watching habits differ by race, specifically across white, black, and other racial groups. Using multiple independent-samples t-tests to compare these three groups increases the risk of a Type I error. Analysis of Variance solves this problem. The research hypothesis is that the mean TV hours differ across at least two of the racial groups. The null hypothesis states all group means are equal. We use an alpha of .05. The One-Way ANOVA in SPSS produces an F-statistic and a corresponding p-value, or Sig. value. If this p-value is .010, it is less than .05. We reject the null hypothesis. This tells you that a significant difference exists among the groups, but not which specific pairs differ. Post-hoc tests would be needed for that. The advertising firm should consider targeting racial groups differently, as their TV watching habits are not uniform.

Linear Regression: Modeling Relationships Between Variables

Mock Study 5 explores the relationship between age and mental health. Linear regression is ideal for this, as it models the linear effect of an independent variable on a dependent variable. The research hypothesis is that age predicts the number of poor mental health days. The null hypothesis is that the slope of the regression line is zero, meaning age has no effect. Alpha is .05. The SPSS regression output focuses on the Coefficients table. The row for the age variable has a p-value under Sig. If this p-value is .043, it is less than .05. We reject the null hypothesis. There is a statistically significant relationship between age and mental health days. The researchers should continue their study. The negative B coefficient indicates that older age is associated with fewer reported poor mental health days. The regression equation would be Predicted MNTLHLTH = B0 + B1(AGE). This equation allows you to predict the expected mental health days for a given age.

Chi-Square Test: Analyzing Categorical Associations

Mock Study 6 investigates the association between political party affiliation and confidence in Congress. Both variables are categorical, so a chi-square test for independence is appropriate. The research hypothesis is that party affiliation and confidence in Congress are related. The null hypothesis states the two variables are independent. We use an alpha of .05. The SPSS Crosstabs output provides a Pearson Chi-Square statistic and its p-value, labeled Asymptotic Significance. If this p-value is .000, it is less than .05. We reject the null hypothesis of independence. The observed distribution of confidence levels is significantly different from the distribution we would expect if the two variables were unrelated. Political party affiliation does appear to affect one’s confidence in Congress. You can examine the crosstabulation table to see which parties have higher or lower confidence than expected.

References

Brown, T. A. (2022). Confirmatory factor analysis for applied research. Guilford Publications.

Freedman, D. A., Pisani, R., & Purves, R. (2020). Statistics. WW Norton & Company.

Lakens, D. (2021). Improving your statistical inferences. Coursera. https://www.coursera.org/learn/statistical-inferences

Tabachnick, B. G., & Fidell, L. S. 2019. Using multivariate statistics. Pearson.

Watson, P. (2023). The misuse of p-values in contemporary social science research. Journal of Applied Statistics, 50(5), 1125-1140.

_______________________________________________________________________________________________________

SOCI332 Assignment 1: Learning and Applying Tests of Significance

Throughout this assignment you will review six mock studies. Follow the step-by-step instructions:

Mock Studies 1 – 3 require you to enter data from scratch. You need to create a data set for each of the three mock studies by yourself. (Refresh the data entry skill acquired in Week 1.)
Mock Studies 4 – 6 require you to use the GSS dataset specified in the course. The variables are given in each Mock Study.
Go through the five steps of hypothesis testing (below) for EVERY mock study.
All calculations should be coming from your SPSS. You will need to submit the SPSS output file (.spv) to get credit for this assignment.
The five steps of hypothesis testing when using SPSS are as follows:

State your research hypothesis (H1) and null hypothesis (H0).
Identify your significance level (alpha) at .05 or .01, based on the mock study. You only need to use ONE level of significance (either .05 or .01) as specified in the instructions.
Conduct your analysis using SPSS.
Look for the valid score for comparison. This score is usually under ‘Sig 2-tail’ or ‘Sig. 2’ or ‘Asymptotic Sig.’ We will call this “p.”
Compare the two and apply the following rule:
If “p” is < or = alpha, then you reject the null. Please explain what this decision means in regards to this mock study. (Ex: Will you recommend counseling services?) Please make sure your answers are clearly distinguishable. Perhaps you could bold your font or use a different color. This assignment is due no later than Sunday of Week 4 by 11:55 pm ET. Save this Word file in the following format: [your last name_SOCI332_A1]. Your spv (SPSS output) file should be labeled [your last name_SOCI332_A1Output]. t-Tests Mock Study 1: t-Test for Independent Samples (20 points) Six months after an industrial accident, a researcher has been asked to compare the job satisfaction of employees who participated in counseling sessions with those who chose not to participate. The job satisfaction scores for both groups are reported in the table below. Use the five steps of hypothesis testing to determine whether the job satisfaction scores of the group that participated in counseling session are statistically different from the scores of employees who chose not to participate in counseling sessions at .01 level of significance. (Alpha = .01) Clearly list each step of hypothesis testing. As part of Step 5, indicate whether the researcher should recommend counseling as a method to improve job satisfaction following industrial accidents based on evaluation of the null hypothesis. Data to be entered in SPSS (instructions below) PARTICIPATED IN COUNSELING DID NOT PARTICIPATE IN COUNSELING 35 38 39 36 41 36 36 32 37 30 36 39 37 41 39 35 42 33 38 38 Step 1: Data managing Open a blank SPSS data file: Fileà Newà Data In the blank SPSS data file, create your SPSS data set by entering the job satisfaction scores of those who participated/did not participate in the counseling sessions (reported on previous page). Please create two columns. Column one is the test variable, where you enter ALL the 20 scores in the table. Column 2 is the grouping variable, where you use “1” to indicate if a score is from someone who participated in the counseling sessions; and “0” to indicate if a score is from someone who chose not to participate in the counseling sessions. The data set will look like this in SPSS Data View window: 35 1 39 1 ………. 38 0 36 After data entry, go to Variable View window, change the name of the first variable (test variable) to “JOBSAT” and the second variable (grouping variable) as “group.” Set decimals for both variables to zero. Step 2: SPSS execution Click: Analyzeà Compare MeansàIndependent-Samples T Testà use arrow to move JOBSAT to “Test Variable” à use arrow to move “group” to “Grouping Variable” àwhen two (? ?) appear, click Define Groups. On the next pop up window, enter “1” for “Group 1” and “0” to “Group 2.” Click OK. Mock Study 2: t- Test for Dependent Means (15 points) Researchers are interested in whether depressed people undergoing group therapy will perform a different number of activities of daily living before and after group therapy. More ADL after therapy is a positive outcome. The researchers randomly selected 10 depressed clients in a 6-week group therapy program. Use the five steps of hypothesis testing to determine whether the observed differences in the numbers of activities of daily living obtained before and after therapy are statistically significant at .05 level of significance. (Alpha = .05) Clearly list each step of hypothesis testing. As part of Step 5, indicate whether the researchers should recommend group therapy for all depressed people based on evaluation of the null hypothesis. Data to be entered in SPSS (instructions below) CLIENT BEFORE THERAPY AFTER THERAPY A 11 16 B 7 12 C 10 13 D 13 20 E 11 14 F 12 15 G 9 15 H 8 17 I 13 18 J 12 9 Step 1: Managing data Open a blank SPSS data file: FileàNewàData In the blank SPSS data file, create your SPSS data set by entering the number of activities of daily living performed by the depressed clients (see above) in the Data View window. Enter the “before therapy” scores in the first column and the “after therapy” scores in the second column. In the Variable View window, change the variable name for the first variable to “ADLPRE” and the second variable to “ADLPOST.” Set the decimals for both variables to zero. Step 2: SPSS execution Click: Analyze à Compare Means àPaired-Samples t-Test à use the arrow to move ADLPRE under “variable 1” inside Paired Variable(s) windowà and then use the arrow to move ADLPOST under “variable 2” inside Paired Variable(s) window. Click OK. Mock Study 3: t-Test for a Single Sample (15 points) Researchers are interested in whether depressed people undergoing group therapy will perform a different number of activities of daily living (ADL) after group therapy than the average for depressed people. More ADL is a positive outcome. The researchers randomly selected 20 depressed clients to undergo a 6-week group therapy program. Use the five steps of hypothesis testing to determine whether the average number of activities of daily living (shown below in the table) obtained after therapy is significantly different from a mean number of activities – 15 – that is typical for depressed people. (Clearly list each step). Test the difference at both the .05 level of significance. (Alpha = .05) Clearly list each step of hypothesis testing. As part of Step 5, indicate whether the behavioral scientists should recommend group therapy for all depressed people based on evaluation of the null hypothesis. Data to be entered in SPSS (instructions below) CLIENT AFTER THERAPY ADL A 17 B 14 C 11 D 23 E 24 F 17 G 14 H 10 I 21 J 11 K 22 L 19 M 15 N 17 O 23 P 12 Q 10 R 15 S 20 T 18 Step 1: Data managing Open a blank SPSS data file: Fileà Newà Data In the blank SPSS data file, create your SPSS data set by entering the number of activities of daily living performed by the depressed clients (numbers listed under AFTER THERAPY – see above) in the Data View window. In the Variable View window, change the variable name to “ADL.” Set the decimals to zero. Step 2: SPSS execution Click: Analyze à Compare Means à One-Sample T test à use the arrow to move “ADL” to the Variable(s) window on the right. Enter the population mean (15) in “Test Value” Click OK. ANOVA (15 points) Mock study 4: One-Way ANOVA An advertising firm has been hired to assess whether different demographics have different rates of TV watching to help determine their advertising strategy. Using the GSS 2018 data, determine whether hours of tv watched differs by race. Use the five steps of hypothesis testing to determine whether the observed differences in the number of hours watching TV across three groups are statistically significant at .05 level of significance. (Alpha = .05) Clearly list each step of hypothesis testing. As part of Step 5, indicate whether the advertising firm should target each racial group differently (if their habits differ) based on evaluation of the null hypothesis. Variables from GSS 2018 dataset to be used (instructions below): RACE – race of respondent 1 = WHITE 2 = BLACK 3 = OTHER TVHOURS – hours per day watching TV Step 1: Data managing Open a blank SPSS data file: Fileà Open Dataàsav (from wherever you have it saved) Step 2: SPSS execution Click: Analyze à Compare Means à One-Way ANOVA à use arrow to move TVHOURS to “Dependent Variable list” à use arrow to move RACE to “Factor,” which instructs SPSS to conduct the analysis of variance on the number of activities performed by therapy type. Click: Options à Descriptive (to obtain descriptive statistics). Click: Continue Click: OK. Additional question based on Mock Study 4 Describe the circumstances under which you should use ANOVA instead of t-Tests. Explain why t-Tests are inappropriate in these circumstances. Regression (20 points) Mock study 5: Linear Regression Researchers in the field of gerontology are researching the effects of age on mental health. They are using GSS data to gather some preliminary findings. Following the five steps of hypothesis testing, conduct a linear regression analysis to determine whether age affects number of poor mental health days at the .05 level of significance. (Alpha = .05) Clearly list each step of hypothesis testing. As part of Step 5, indicate whether there is a significant relationship between age and mental health at the .05 level and what does this mean in regard to this mock study. Should the researchers continue their study? Variables from GSS 2018 dataset to be used (instructions below): AGE – age of respondent MNTLHLTH – Days of poor mental health past 30 days Step 1: Data managing Open a blank SPSS data file: Fileà Open Dataàsav (from wherever you have it saved) Step 2: SPSS execution Click: Analyze à Regression à Linear à use arrow to move MNTLHLTH to “Dependent list” à use arrow to move AGE to “Independent,” which instructs SPSS to conduct the linear regression on the relationship of age to poor mental health. Click: OK. For Mock Study 5, after completing the 5 steps of hypothesis testing, also construct the regression equation for the analysis. What does this tell us? Chi-Square (15 points) Mock study 6: Chi-Square Test for Independence Researchers are interested in whether US adults have different levels of confidence in Congress (legislative branch of the federal government) in conjunction with how strongly that person identifies with a specific political party. These data are presented below. Following the five steps of hypothesis testing, conduct chi-square test for independence at the .05 level of significance. (Alpha = .05). Clearly list each step of hypothesis testing. As part of Step 5, indicate whether the observed frequency is significantly different from the expected frequency, and what that means in regard to this mock study. In other words, does political party affiliation effect one’s confidence in Congress? Variables from GSS 2018 dataset to be used (instructions below): CONLEGIS – confidence in congress (legislative branch of government) 1 = A GREAT DEAL 2 = ONLY SOME 3 = HARDLY ANY PARTYID – political party affiliation 0 = STRONG DEMOCRAT 1 = NOT STR DEMOCRAT 2 = IND NEAR DEMOCRAT 3 = INDEPENDENT 4 = IND NEAR REPUBLICAN 5 = NOT STR REPUBLICAN 6 = STRONG REPUBLICAN 7 = OTHER PARTY Step 1: Data managing Open a blank SPSS data file: Fileà Open Dataàsav (from wherever you have it saved) Step 2: SPSS execution Click: Analyze à Descriptive Statistics à Crosstabs à use arrow to move “PARTYID” to “Column(s)”à use arrow to move “CONLEGIS” to “Row(s).” (Recall in crosstab, DV is always in the row and IV is always in the column.) Click: Statistics à check “Chi-Square.” Click: Continue. Click: Cellsà check “Expected.” Click: Continue. Click: OK