4 Next ideas for the research agenda
As a concluding chapter, it is essential to project methodologies and knowledge gaps that directly influence the DSE agenda of NUDOS. In the first subsection, the limits on measurement and application of self-efficacy in ILSA are critically examined. Next, the main methodologies used in the reviewed studies are analyzed, with a particular focus on multilevel studies incorporating country-level data and International Large-Scale Assessments (ILSA). Finally, the chapter addresses the knowledge gaps surrounding the determinants of DSE.
4.1 Limits on Self-efficacy conceptualization: aplications and measurements on ILSA’s
As reviewed, ILSA studies set out to measure self-efficacy in a variety of ways, and with very different approaches. In some, the digital problem is central, in others it is marginal, and the scale construction strategies are very different.
Despite these contrasts, no work was found that attempted to make a comparative exercise between the approaches to digital self-efficacy of the three ILSAs that were brought into the discussion. A good opportunity to advance the research agenda would be a comparative review of ICILS, PISA and TIMSS around the measurement of the concept of digital self-efficacy.
For this, the different studies present opportunities to go beyond. ICILS presents two types of Digital Self-efficacy, which is not present on other studies, so it’s to answer why. TIMSS have another batteries of self-efficacies in non-digital activities, as maths, finance or language. This study make possible to compare the levels of generazibility of DSE on other subjects. And PISA, as an official OECD project, facilitates comparing between developed and under-developed countries.
4.2 Methods studying Digital Self-Efficacy differences
Throughout the literature review, a multitude of methodological tools have been identified as useful for the DSE agenda of NUDOS. The vast majority of the studies reviewed involve multilevel models and, at times, structural equation modeling, due to the availability of school-level data in ILSAs and the effort to explore mediators of relationships already demonstrated in the literature. Additionally, periodic meta-analyses that synthesize findings from previous research are evident.
The study by Hatlevik et al. (2018), the most relevant one with ICT self-efficacy as the dependent variable, conducts a path analysis, a multivariate model that allows the inclusion of multiple independent and dependent variables simultaneously. It seeks to identify the main determinants of ICT self-efficacy and, additionally, the relationship between digital literacy and digital self-efficacy, using ICILS 2013 data. The analysis included 14 countries. For the reporting of results, the range of effects across all countries was mentioned, with occasional emphasis on outlier cases.
The article by Chen & Hu (2020) is particularly relevant to the agenda because it uses PISA data, encompassing a large number of countries (30). The study conducts a multilevel mediation model (utilizing the school level) that does not account for relationships between mediators, assuming instead that they do not covary. The independent variable is interest in ICT, the dependent variable is ICT self-efficacy, and the mediating variables considered were ICT use at home for leisure, ICT use at home for school tasks, ICT use for socializing, and ICT use in school. Additionally, gender and the index of economic, social, and cultural status (ESCS)—the latter being readily available in the PISA database—were included as control variables. The chosen technique for mediation analysis was the parallel multiple mediator model, which allows for the inclusion of multiple mediating variables in the analysis simultaneously. Similar to Hatlevik’s article, the range of effects across the different countries was reported.
The paper by Campos & Scherer (2024) may serve as an excellent roadmap for research on PISA 2022. It conducts a meta-analysis combining data from ICILS 2013, ICILS 2018, and findings from 165 different articles sourced from databases (ERIC and PsycINFO). The study’s objective is to determine the extent to which gender differences in digital skills are mediated by attitudes, analyzing heterogeneity across countries and studies. Missing data were imputed through a two-level mean-matching approach (considering background data, technology use, and attitudes toward technology), resulting in 100 datasets for each ICILS cycle. Beyond this, and of particular relevance to the Digital Self-Efficacy agenda, a variety of indices were used to explore relationships between country-level outcomes and socioeconomic development, gender inequality, and other factors. Specifically, the following indices were employed: Human Development Index, Gender Inequality Index, Global Innovation Index, and ICT Use Index. Additionally, UNESCO’s country classification was used. In total, four distinct models were executed for each attitude addressed in the study (self-efficacy, beliefs, and affect). The first was a multivariate model with simple random effects; the second was a multilevel model adding the country variable; the third was a multilevel, multivariate model capturing the average effects of direct and indirect variables and their variance at different levels; and the fourth added the aforementioned indices to the first model.
4.3 Knowledge gaps regarding Digital Self-efficacy
The relationship between various socioeconomic, sociodemographic, and subjective variables and technology-related self-efficacy has been studied. However, there is still a need to examine diverse relationships and profiles of technology users who, to this day, are being left behind in the face of the burgeoning technological revolution. One of the most notable gaps is the lack of studies considering migration background or ethnicity as an independent variable. In other fields of study, this has been addressed; for example, a study conducted in Chile found a positive relationship between students’ migrant status and their general self-efficacy (Céspedes et al., 2021). It is crucial to examine this relationship in the context of digital self-efficacy, as digital competencies have become vital for adapting to a new country and achieving economic advancement.
Similarly, there is a noticeable lack of studies that delve into explaining differences in ICT self-efficacy across the various countries that have participated in ILSAs. Although articles such as Campos & Scherer (2024) incorporate indices like the Human Development Index and the ICT Use Index, they do not use digital self-efficacy as the dependent variable, and analyzing the role of the country in the development of a particular technological self-efficacy is far from being their primary objective. This issue is particularly relevant when addressing the digital divide in its broadest dimension: the global one.
4.4 Closure
It is hoped that the work carried out in this document will serve, in the first place, as a systematization of the information related to digital self-efficacy, both in its background and theoretical foundations and in the way it is measured in the different ILSAs, in order to facilitate access to this knowledge for NUDO members. In addition, this working paper can serve as a basis for future research projects on digital issues, such as the creation of academic articles, methodological proposals and even new guidelines.