Data Analysis
Propensity Scores
- Propensity scores are a newer approach to estimate causal effects in nonrandomized studies. They help mimic the random selection of participants of a randomized control trial in an observational survey (Rosenbaum & Rubin, 1983).
- A propensity score is the predicted probably of treatment after accounting for important matching variables (Reutzel, Spichtig, & Petscher, 2012).
- The goal or objective for a researcher using propensity scores is to select a sequence of variables that are considered important in matching participants (Reutzel, Spichtig, & Petscher, 2012).
- Reading research theory suggests that race/ethnicity, socioeconomic status, English language learner status, gender, and a baseline measure of achievement are all important variables to include in my model (Reutzel, Spichtig, & Petscher, 2012). The baseline measure for this study was the letter ID measure.
- Once propensity scores were estimated for participants from the control and treatment groups using logistic regression, the probabilities were used to match students who received the treatment with those who did not receive treatment (Austin, 2011; Reutzel, Spichtig, & Petscher, 2012).
- In theory, the matched samples are identical (or very similar) on many meaningful characteristics and only differ in their treatment status (Thoemmes & Kim, 2011).
Statistical Analysis
- To help answer my first research question, I conducted a descriptive discriminant analysis [DDA] (Huberty, 1994) on the matched sample. Canonical discriminant functions are used to determine if the variance in the synthetic dependent variable can be explained by the independent variable in the model. DDA can also help determine the relevance of the dependent variables and evaluate which of the variables contributed to group differences.
- To help answer research questions #2 and #3 and evaluate the effect of the level of teacher literacy support, I conducted a 2 X 3 (Istation®: Yes/No X Teacher Support: Low/Medium/High) multivariate between-subjects analysis of variance to test for the main effect of teacher support and interaction between use of Istation and the level of teacher support.
Data Transformation
- Histograms and significance tests of the data indicated a violation of the assumption of multivariate normality (z=-8.143, p=.001).
- For the Letter Sound Knowledge and Hearing and Recording Sounds subtests, in particular, a negatively skewed distribution was evident and univariate tests of normality showed substantial deviations from a normal distribution.
- To help the data meet normality and heteroscedasticity assumptions, the six dependent variables were transformed using Box-Cox procedures (Osborne, 2010).
- Tests of the transformed data indicated that all of the variables met the assumption of multivariate normality.
References
Austin, P.C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46, 399-424.
Huberty , C.J. (1994). Applied discriminant analysis. New York, NY: Wiley and Sons.
Osborne, J.W. (2010). Improving your data transformations: Applying the Box-Cox transformation. Practical Assessment, Research & Evaluation, 15(12), 1-9.
Reutzel, D.R., Petscher, Y., & Spichtig, A.N. (2012). Exploring the value added of a guided, silent reading intervention: Effects on struggling third-grade readers’ achievement. The Journal of Educational Research, 105, 404-415.
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
Thoemmes, F.J., & Kim, E.S. (2011). A systematic review of propensity score methods in the social sciences. Multivariate Behavioral Research, 46, 90-118.
Huberty , C.J. (1994). Applied discriminant analysis. New York, NY: Wiley and Sons.
Osborne, J.W. (2010). Improving your data transformations: Applying the Box-Cox transformation. Practical Assessment, Research & Evaluation, 15(12), 1-9.
Reutzel, D.R., Petscher, Y., & Spichtig, A.N. (2012). Exploring the value added of a guided, silent reading intervention: Effects on struggling third-grade readers’ achievement. The Journal of Educational Research, 105, 404-415.
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.
Thoemmes, F.J., & Kim, E.S. (2011). A systematic review of propensity score methods in the social sciences. Multivariate Behavioral Research, 46, 90-118.