Background. International large-scale assessments (ILSAs), such as the Trends in Mathematics and Science Study (TIMSS), provide large-scale data of students, classrooms, and schools that can be used to test hypotheses about relations among constructs, differences between groups, and ultimately whether these generalize across countries (Martin et al., 2016). However, the sheer size and hierarchical structure of these assessments raise challenges for their analysis.

Multilevel structural equational modeling (MSEM) is commonly used to analyze ILSA data. MSEM can account for the hierarchical structure of the data and, at the same time, control for measurement error at different levels of analysis (Muthén & Muthén, 2015). However, the use of MSEM for the synthesis of ILSA results is problematic. For instance, a researcher may be interested in the extent to which the relation between a contextual variable (e.g., school climate) and a student-level variable (e.g., individual achievement) varies between countries. To quantify the variation of this contextual effect, a three-level model (L1: students, L2: schools, L3: countries) would be needed to estimate the variance of the random slope at the country level. However, besides convergence and power issues, high computational demands, and software limitations, random slopes of derived model parameters (i.e., the contextual effect as the difference between the L2 and the L1 effects) are not explicitly accessible in MSEM (e.g., Nagengast & Marsh, 2012).

The split, analyze, and meta-analyze (SAM) approach offers an alternative to overcome the difficulties in the analysis and synthesis of findings using ILSA data (Cheung & Jak, 2016). In this approach, the data are first divided by countries. Next, each data set is analysed independently and, finally, the parameter estimates are combined using meta-analysis. As a result, the SAM approach allows estimating a weighted average effect size and its variation from a ILSAs by combining MSEM and meta-analysis.

Objectives. In this study, we illustrate how the SAM approach can be used to synthesize specific results of ILSAs, replicating the study of the Big-Fish-Little-Pond-Effect (BFLPE) by Nagengast and Marsh (2012). The BFLPE refers to the negative contextual effect of classroom mathematics achievement on students’ mathematics self-concept. Specifically, we examine whether country-level variation in the BFLPE exist and which factors may explain this variation.

Research Questions:

• RQ1: To what extent does evidence for the generalizability of the BFLPE and its variation across countries exist?
• RQ2: To what extent do country-level variables (Human development index, individualism, power distance, uncertainty avoidance, masculinity, indulgence, and long-term orientation) explain the variation in the BFLPE across countries?

Method. We analyzed the data from 293,747 fourth-grade students from 49 countries that participated in TIMSS 2015. Student and classroom indicators of mathematics achievement and mathematics self-concept were extracted directly from the TIMSS data. The country-level variables were obtained from the Human Development Index database of the United Nations, and the cultural dimensions data from Hofstede et al. (2010).

The analyses were divided into three different parts: First, we split the dataset by countries. Second, we performed multigroup multilevel confirmatory factor analysis and two-level MSEM to evaluate the invariance of the measurement models and obtain the BFLPE per country. Third, we submitted the extracted effect sizes of the BFLPE to random-effects meta-analysis and mixed-effects meta-regression.

Results. Multi-group MCFA indicated the invariance of the self-concept measurement model across countries and between levels of analysis (RMSEA = 0.034, CFI = 0.98, TLI = 0.96). Extending this model to a contextual model by adding achievement at the respective levels resulted in an acceptable-fitting model, yielding the BFLPE effect sizes (RMSEA = 0.06, CFI = 0.91, TLI = 0.87).

The analysis of the BFLPE effect sizes with random-effects meta-analysis suggested that there was a negative relation between classroom-achievement and students’ self-concept in mathematics. The weighted average BFLPE was ES = -0.623 (95 % CI [-0.681, -0.564]) and varied substantially between countries (Q_E[48] = 15,101, p < .001; τ²= 0.04, I² = 100 %, H²= 31.46). The mixed-effects meta-regression model explained 84.5 % of the between-country variance by five of the cultural dimensions, yet not the Human Development Index (Table 1). However, considerable residual heterogeneity remained even with the inclusion of the seven moderators (Q_E[33] = 1,878.75, p < .001, I² = 98.24 %, H² = 56.93, R²= 84.54 %).

Conclusions. Researchers using ILSAs can use the SAM approach to synthesize the results from those datasets, given that the raw data are openly available. Integrating the SAM approach and MSEM offers an alternative methodology for the analysis and synthesis of research evidence using ILSA data. Our study illustrates how educational researchers can use the SAM approach to investigate country-level differences in relations among constructs. We hope that our illustration will inspire researchers to utilize ILSA data to provide evidence about educational and psychological effects.

References

Cheung, M. W. L., & Jak, S. (2016). Analyzing big data in psychology: A split/analyze/meta-analyze approach. Frontiers in Psychology, 7(738), 1–13. https://doi.org/10.3389/fpsyg.2016.00738

Hofstede, G., Hofstede, G. J., & Minkov, M. (2010). Cultures and organizations: Software of the mind ; intercultural cooperation and its importance for survival (Rev. and expanded 3. ed). McGraw-Hill.

Martin, M. O., Mullis, I. V. S., & Hooper, M. (2016). Methods and procedures in Timss 2015. International Study Center, Lynch School of Education, Boston College.

Nagengast, B., & Marsh, H. W. (2012). Big fish in little ponds aspire more: Mediation and cross-cultural generalizability of school-average ability effects on self-concept and career aspirations in science. Journal of Educational Psychology, 104(4), 1033–1053. https://doi.org/10.1037/a0027697