A meta-analysis study of the effect of the blended learning model on students’ mathematics learning achievement

Many studies have researched the effect of blended learning on mathematics learning achievement, but the results of previous research reported different results. Therefore, this research aims to determine the effect of the blended learning model on mathematics learning achievement. The design of this study used a meta-analysis approach by analyzing 20 effect sizes from 18 primary studies that were Scopus-indexed and met the inclusion criteria. The results of the analysis show that the use of the blended learning model affects mathematics achievement compared to traditional learning (d = 0.725; p < 0.05). The results of the analysis according to the moderator variable are known that the effect of the blended learning model on mathematics achievement is different based on the level of education (Qb = 13.923; p < 0.05) and the year of research (Qb = 16.140; p < 0.05). However, no differences were found according to the sample size group (Qb = 0.039; p > 0.05) and media platform (Qb = 2.861; p > 0.05. The findings show the consistency of the publication of research results on the effect of using the blended learning model on students' mathematics learning achievement.


Introduction
Advances in information and communication technology in the 21st century have significantly influenced the world of education, especially in the learning process (Akgundus & Akinoglu, 2016). With the rapid advancement of educational technology today, the teaching and learning environment has begun to change and develop (Karagöl & Emrullah, 2019). Technology can improve student collaboration (Keser et al., 2012), higher-order thinking skills (Kurt, 2010), as well as learning engagement, and motivation (Baytak et al., 2011). Technology integration benefits students by providing extra practice, the opportunity to evaluate their problems, and the amplest opportunity to choose different alternative answers (Gonzalez & Birchm, 2000;Juandi & Priatna, 2018;Nurjanah et al., 2020;Sung et al., 2016). Technology can create practical and meaningful mathematics learning and visualize mathematical concepts or objects (Herron, 2010;Setyaningrum, 2018). Using technology in creative learning and according to student needs can assist in developing mathematical knowledge and skills to meet the quality of education and the needs of 21stcentury society (Adelabu et al., 2019;Chen et al., 2020). Students who can maximize technology as a learning resource are proven to have good math skills (Bulut & Delen, 2011). The availability of technology in the school environment also has a positive influence on academic performance (Hu et al., 2018). Therefore, educators in the learning process are expected to be able to integrate technology in designing learning.
Using appropriate learning models can improve the quality of the learning process so that it affects the achievement of competence or student learning achievement (Prasetya et al., 2018). The learning process's general flow is the educator's delivery of material, then making assignments or practicing during class time. However, this kind of teacher-centered learning provides time constraints, so students must continue their learning activities at home (Cobena & Surjono, 2022). The study's results by Sanuaka et al. (2017) also reported that using inappropriate strategies and limited time made it difficult for students to develop their skills. Setiawan et al. (2022) revealed that teachers need to apply a learning model that can provide ample time and give students wider opportunities to explore their learning styles. Based on this problem, the blended learning model can be applied because it allows students to explore their styles and adapt to their learning speed. The blended learning model integrates different online and face-to-face learning methods, for example, lectures, independent learning, and online discussions. After students have established an overview of the course, they can move on to learning and interacting online (Lin, 2017). A similar definition by Lalima and Dangwal (2017) states that blended learning integrates direct, indirect, collaborative, and computer-assisted learning. Blended learning requires internet access, but the process is not only displaying the learning web in the classroom but also using learning strategies that suit student needs. Some commonly used blended learning models are rotation, flex, self-blend, and Enriched-Virtual (Staker & Horn, 2012).
Previous research has proven that blended learning can increase engagement and overcome traditional learning approaches' weaknesses (Alammary et al., 2015;Dziuban et al., 2018). Able to increase flexibility and convenience in learning, learning achievement, and student learning engagement (Owston et al., 2013). Marco et al. (2013) revealed that the advantages of blended learning include: Increasing access and flexibility, good student response, increasing pedagogical abilities, cost-effectiveness, feedback speed, and facilitating access to everyone who needs training.
Although many studies have proven that the use of blended learning models has a positive effect on learning achievement in mathematics, but they found different results (Bhagat et al., 2016;Clark, 2015;Ramadhani et al., 2019). Their research revealed that the use of blended learning had no significant effect on learning achievement in mathematics. Based on these problems, efforts are needed to combine previous findings related to the effect of blended learning on mathematics learning achievement to be evaluated quantitatively so that it can provide broader and more accurate results. In this case, a meta-analysis approach can be used to evaluate the results of previous studies to reach in-depth and accurate conclusions (Retnawati et al., 2018;Schmidt & Hunter, 2004;. The main aim of the meta-analysis was to find the effect size. Effect size is a measure of the magnitude of the influence, the magnitude of the difference, and the relationship of a variable with other variables. A meta-analysis study on the effect of the blended-learning model on mathematical achievement conducted by Setiawan et al. (2022) revealed that blended learning affects students' mathematical abilities. However, the meta-analysis studies only focused on studies conducted in Indonesian addition; they also emphasized that the limitations of the research were that there was no analysis of moderator variables. The moderator variable analysis was conducted to determine other factors that influence heterogeneity (Borenstein et al., 2009). The moderator variables analyzed in this study were education level, sample size, media platform, and year of study. The results of research conducted by Strelan et al. (2020), Juandi et al. (2021), (Purnomo et al., 2022), and (Paloloang et al., 2020) prove that the moderator variables of education level, sample size, and year of study affect the effect size. In addition, research by Khoirunnisa and Adistana (2021) revealed that using a blended learning model with the help of a learning management system (LMS) improved student learning outcomes.
Therefore, this study aims to measure the effect of the blended learning model on mathematics learning achievement using a meta-analysis approach. In addition, this study also investigates whether the effect of using the blended learning model on mathematics achievement is different based on the moderator variables of education level, sample size, media platform, and year of research.

Method
The design of this study used a group contrast meta-analysis approach. This approach was used to examine the results of research that examines the effectiveness of the blended learning model on mathematics achievement. In general, the procedure in this meta-analysis study refers to (Borenstein et al., 2009) and (Retnawati et al., 2018), among others; 1) determine inclusion criteria, 2) Data collection and coding, 3) Data analysis.

Determine Inclusion criteria
The determination of inclusion criteria to facilitate the search for studies was at a later stage. The studies collected in the initial search were then examined and assessed using the inclusion criteria defined for inclusion in meta-analysis and further evaluation. The inclusion criteria established in this meta-analysis included: 1. The year of publication ranges from 2014 to 2022; 2. Articles must be Scopus indexed at the time of publication; 3. Studies using experimental or quasi-experimental research methods; 4. There is at least 1 experimental group with blended learning and the comparison group as a control group with traditional learning; 5. The study must report the mean, standard deviation and sample size of each experimental group and control group; or sample size and t-value; or sample size and p-value; or sample size with F-value

Data collection and coding
Collecting relevant studies uses online databases such as Google Scholar, ERIC, Elsevier, SAGE, and SpringerLink. The keywords used in searching the relevant literature are "Effect or Impact or Effectiveness of Blended Learning on Mathematics." From the search results based on the specified criteria, 18 primary studies were collected from 119 initial search studies. Table 1 describes information on primary studies published by various Scopusindexed journals. Several journals have been discontinued from Scopus (like numbers 1, 3, and 6). However, we took the studies included in these journals because the journals published had been continued at the time of the studies. After getting an article that is eligible (meets the inclusion criteria), identify the literature's characteristics by coding. The coding in this study was carried out by two people (raters) so that subjective errors could be avoided. The coding content includes information; 1) Education Level; 2) Sample size of the experimental group; 3) Platform media; 4) Year of Research; 5) Frequency; and 6) Percentage. Table 2 presents a summary of the coding results. From the 18 primary studies that met the inclusion criteria, the researcher obtained 20 independent samples. Table 2 presents a description of the moderator variables. The distribution of education level groups shows that most studies are conducted at universities and high schools, as much as 35%.
In comparison, for studies conducted at junior high schools, as much as 20% and at least 10% are carried out in elementary schools. In terms of sample size, it was found that most studies were conducted in the small sample size group (≤ 50), as much as 70%, and the large sample, as much as 30%. Based on the media platform, it was found that most studies were carried out in the Learning Management system (LMS) group as much as 65% and social media as much as 35%. While in terms of the year of the study, it was found that most studies were carried out in the 2018-2022 group, as much as 60%, and in 2014-2017, as much as 40%.

Data analysis
This meta-analysis study analyzed data using the OpenMEE for Windows 10 application. The data analysis procedure followed the following steps: 1) calculating the effect size of each study; 2) performing heterogeneity test; 3) Calculating summary or combined effects; 4) Test and analyzing moderator variables; 5) Publication bias test. The classification of each effect size or combined effect of this meta-analysis study follows the classification of Cohen et al. (2018), which is shown in Table 3 below: Before calculating effect sizes from meta-analytical studies, heterogeneity was first tested. The heterogeneity test aims to select the appropriate effect size measurement model. The heterogeneity test in this study uses the Q parameter. The decision-making criteria is if the p-value <0.05, then the measurement model used to calculate the effect size is a random effect, and if the p-value> 0.05, then the fixed effect is used (Borenstein et al., 2009;Retnawati et al., 2018). Furthermore, a publication bias test is carried out to ensure that the research included in the meta-analysis has shown results that follow field conditions (objective) Retnawati et al., 2018;Setiawan et al., 2022;. The approach used to evaluate publication bias is File-Safe N (FSN). If the FSN value is greater than 5k + 10, where k is the number of studies, it can be concluded that there is no publication bias problem.

Characteristics and effect sizes of each study
The first objective of this study was to determine the overall effect of the blended learning model on mathematical achievement. The first step in this meta-analysis was to calculate the effect size of each study. Study effect sizes were calculated with the help of OpenMEE software. Table 4 provides a summary of the effect size values for each study.  Based on Table 4 above, out of a total of 20 effect sizes from the studies conducted, the effect size values ranged from 0.033 to 2.058, with a 95% confidence level. Referring to the classification of (Cohen et al., 2018), there are four effect sizes (n = 4) classified as a negligible effect, five effect sizes (n = 5) classified as small effect, three effect sizes (n = 3) classified as moderate effect, four effect sizes (n = 4) were classified as large effects, and the four effect sizes (n = 5) were classified as very large effects. For clarity, figure 1 visualizes a comparison of the effect size classifications between studies.

Heterogeneity test and overall effect size
The heterogeneity test aims to select a suitable model to calculate the combined effect size. Many approaches are used to test for heterogeneity, but in this study, the Q parameter approach was used by looking at the p-value. If the p-value < 0.05, the effect size variance is heterogeneous, so a random effects model is used. If the p value > 0.05, the effect size variance is homogeneous, so the model used is a fixed effect. The heterogeneity test results (see Table 5) show that the Q value is 176.276. Since this value is greater than the chi-square value (df = 19) and the p-value < 0.05, it can be concluded that the studies conducted to calculate the effect size were heterogeneous. The I² value found to reach 88.97% reflects high heterogeneity (Higgins et al., 2003). Since the studies used were heterogeneous, the overall effect size value was based on the random effects model. Based on the random effects model, the effect size value is 0.725. This effect size belongs to the medium effect category (Cohen et al., 2018). Thus, these results reveal that the use of blended learning has a moderate effect on students' mathematics achievement.

Moderator variable analysis
Because the analyzed studies are heterogeneous in distribution, it is the potential to analyze moderator variables. The moderator variables identified in this study were education level, sample size, media platform, and year of study. Table 6 presents the results of the analysis of moderator variables.

Educational stage
The moderator variable for education level consists of four groups: elementary, junior high, high school, and university. The results of the analysis (see Table 6) found that the mean effect sizes of the four levels were significantly different (Qb = 13.923; p < 0.05). These results indicate that the effectiveness of blended learning compared to traditional learning in mathematics achievement differs according to education level. The use of blended learning was most effective in the university group (d = 0.867; p < 0.05), followed by the senior high school group (d = 0.722; p < 0.05) and junior high school (d = 0.707; p < 0.05). Meanwhile, the primary school group did not prove significant (d = 0.305; p > 0.05). These results reveal that blended learning is not proven effective compared to traditional learning at the elementary level.

Sample Size
The moderator variable of sample size consists of two groups, namely small and large sample groups. The results of the analysis (see Table 6) found that the effect size in the small sample group was (d = 0.631; p < 0.05), and the effect size in the large sample group was (d = 0.920; p < 0.05). Based on the different tests, it was found that the average effect size of the two sample size groups was not significantly different (Qb = 0.039; p > 0.05). These results indicate that the effect of blended learning compared to traditional learning in mathematics achievement does not differ according to to sample size.

Platform Media
The moderator variables used consisted of two groups: LMS and social media. The results of the analysis (see Table 6) found that the effect size of using LMS was (d = 0.805; p < 0.05), and the effect size of social media use was (d = 0.565; p < 0.05). Based on the different tests, it was found that the average effect size of the two groups using the media platform was not significantly different (Qb = 2.861; p > 0.05). These results indicate that the effect of blended learning compared to traditional learning in mathematics achievement is similar according to the media platform's use.

Year of Study
The moderator variables of the research year used consisted of two groups, namely 2014-2017 and 2018-2022. The results of the analysis (see Table 6) found that the effect size of the 2014-2017 group was (d = 0.406; p < 0.05), and the group effect size of 2018-2022 was (d = 0.907; p < 0.05). Based on the difference test, it is known that the average effect size of the two groups in the study year is significantly different (Qb = 16.140; p < 0.05). These results indicate that the effect of blended learning compared to traditional learning on mathematics achievement differs according to the study year group.

Evaluation of publication bias
Meta-analytical studies that are scientifically justified and reflect objectivity can be assessed by evaluating publication bias. This study examines publication bias with the File-Safe N (FSN) approach. The results of the analysis (see Table 7) were obtained (FSN = 1624 > 5k+10= 110). These results suggest that this meta-analysis study does not have publication bias issues. The following table provides a summary of the evaluation of publication bias.

Discussion
The analysis results show that the overall effect size using the random effect model is (d = 0.725; and k = 20). These results indicate that overall mathematics learning achievement using the blended learning model is more effective than traditional learning. These results align with the findings of (Setiawan et al., 2022), who conducted a meta-analysis of studies in Indonesia. Their findings show that students' mathematical abilities using mixed learning are more effective than traditional learning. Another finding in line with this study is the metaanalysis conducted by (Lusa et al., 2021). However, the variables were not focused on mathematics achievement; their findings revealed that blended learning positively affected thinking skills, motivation, and learning independence. The blended learning model is more effective than traditional learning because, in traditional learning, students cannot develop at their own pace, and if they are stuck, it is not easy to catch up on what they have missed. However, when using technology-assisted blended learning, each student can control their learning progress and learn without being distracted. Students can browse the learning materials as much as they need and repeat the exercises to understand the content and can more broadly explore their learning styles (Hung, 2007;Lin, 2017;Sung et al., 2016;Wang & Yu, 2012;Wiginton, 2013).
Based on the moderator variable of education level, the analysis results show that the effect of applying the blended learning model on mathematics learning achievement differs according to education level. The application of the blended learning model has a positive impact at the university, high school, and junior high school levels. Meanwhile, it has yet to impact the elementary school level positively. These results are in line with the findings of Belanger (2018) in the United States. The study's results revealed that the mathematics learning outcomes of elementary school students who used traditional learning were better than mixed learning.
Another result supporting this research is the meta-analysis findings of Strelan et al. (2020), which found that blended learning using the flipped classroom method had little effect at the elementary school level. It indicates that the higher the level of education where the blended learning model is applied, the higher the effectiveness on mathematics achievement. It is in line with the findings of Ahlfeldt et al. (2005) and  that higher grades have higher engagement rates. Further research also needs a meta-analysis to get broader and more accurate conclusions, especially at the elementary school level.
Based on the sample size moderator variable, the analysis results show that the effectiveness of using the blended learning model compared to traditional teaching on math skills is not significantly different. These results indicate that the use of blended learning on mathematics achievement is equally effective when applied to a small sample group (≤ 50) and a group with a large sample size (≥ 50). Thus the difference in research sample size does not change the size of the effect of studies comparing the application of blended learning and conventional models to students' mathematics achievement. This result differs from the findings of Karagöl and Emrullah (2019). Their findings show that small sample sizes produce larger effect sizes. To achieve consistent results, necessary to involve more primary studies in the analysis.
Based on the moderating variable of the use of technology media, the analysis results show that the effectiveness of the use of blended learning models when compared to traditional teaching on mathematical abilities is not significantly different. These results indicate that blended learning on mathematical abilities is equally effective when applied to groups using LMS and social media technology. However, another fact was also found that using LMS was not proven effective at the elementary school level as in research (Belanger, 2018).
The use of LMS also has little effect at the elementary school level (Lin, 2017). Ramadhani et al. (2019) found a different result, which showed that the use of LMS did not have a significant effect at the high school level. While the use of social media was also found to be effective at the high school level (Ojaleye & Awofala, 2018), at the junior high school level, the use of social media tends to have a negligible effect. Differences in results related to the use of technology media in blended learning provide suggestions for future research to conduct a meta-analysis related to the use of technology media in supporting blended learning by involving more studies.
Based on the moderator variable in the year of research, the analysis results show that the effectiveness of blended learning models, when compared with traditional teaching, on mathematics achievement is significantly different. The use of the blended learning model in the 2018-2022 group gives a more significant effect size than in 2014-2017. The use of the blended learning model in the current research year has undergone a development process from previous years, so the results obtained will be better than the previous year. Although these results show significant differences between groups in the year of study, overall, the use of the blended learning model on mathematics achievement has proven effective in 2014-2017 and 2018-2022. A previous meta-analysis supports this finding by Vo et al. (2017), which investigated variations in primary study outcomes by year of study.
This study gives precise results that the confusing effect size differences between the variables of the blended learning model and mathematical achievement based on various kinds of literature (some fall into the category of no effect to large) become evident after a meta-analysis is carried out in the medium category. Based on the analysis results, it is very urgent to implement Blended Learning to support student achievement. Schools must be able to apply Blended learning properly according to the needs of students. Thus the quality of the learning process will increase. In addition, the results of this study will provide an overview of the effect of Blended learning on student achievement so that it can be used as a basis for policy-making to improve the quality of learning. In addition, the results of this study can also be used as a reference to compare the application of Blended learning to achievement in terms of various places/locations or countries in answering the importance of implementing Blended learning in the learning process in the classroom.

Conclusion
The application of the blended learning model affects students' learning achievement in mathematics compared to the application of the traditional learning model. Based on the analysis of the moderator variables, it is known that the effect of the blended learning model on mathematics learning achievement differs according to the education level group and year of study but not according to the sample size group and media platform. The findings of this meta-analysis show the consistency of the publication of research results on the effect of using the blended learning model on students' mathematics learning achievement.
Apart from the validation results reported, this study also has limitations. This study only analyzed 20 effect sizes. This study also only analyzes mathematics achievement in general. Further research needs to expand the research sample and analyze mathematics achievement more specifically, for example, critical thinking skills, mathematical communication, and others. In addition, it is also recommended to be more specific in reviewing the analysis of moderator variables by involving more research so that research findings become more accurate.

Acknowledgment
Thank you to the Faculty of Education, State University of Yogyakarta for providing material and facility support so that this research can be carried out properly.

Conflicts of Interest
The authors declare that there is no conflict of interest in connection with the publication of this manuscript. In addition, ethical issues, including plagiarism, infringement, falsification and/or falsification of data, publication and/or double submission, and redundancy are fully borne by the authors.

Funding Statement
This work received no specific grant from any public, commercial, or not-for-profit funding agency.