ChatGPT as a Tool for Language Development: Investigating Its Impact on Proficiency and Self-Evaluation Accuracy in Indonesian Higher Education

In the context of today's interconnected world, the role of technology, particularly chatbots like ChatGPT, is becoming increasingly significant in enhancing English language learning. This study adopts a mixed-methods approach to explore the effectiveness of ChatGPT in language development. A sample of 100 students from several universities in Indonesia participated in the research, involving pre-test and post-test assessments to gauge language proficiency improvements. Additionally, the study examined the accuracy of students' self-assessments before and after their interaction with ChatGPT. The findings reveal that ChatGPT's personalized feedback had a substantial impact on students' language skills, as reflected in improved post-test scores. Furthermore, there was a marked enhancement in the accuracy of students' self-evaluations after using ChatGPT. The study indicates that ChatGPT aids in language learning by offering interactive feedback and targeted guidance, enabling learners to progress at their own pace. Students reported multiple benefits, including interactive feedback, increased motivation, and better access to resources. Although some challenges were noted, the advantages of using ChatGPT for language learning largely outweighed them. This research contributes to the understanding of chatbots as effective tools in self-directed learning environments and offers valuable insights for educators and practitioners in implementing these technologies in language teaching, particularly in the Indonesian university setting.


INTRODUCTION
English language proficiency is crucial for individuals seeking to thrive in today's globalized world.In the context of Indonesia, a country with a diverse linguistic landscape, Rintaningrum et al. (2017) assured that mastering the English language is particularly important for students at the university level.However, traditional classroom-based language learning approaches often face challenges in providing personalized instruction and addressing individual needs.Recently, the integration of chatbots, specifically ChatGPT, in language learning has gained attention as a promising approach to enhancing language acquisition.Kaiss et al. (2023) explained that Chatbots offer an interactive and adaptive learning experience, providing personalized feedback and guidance tailored to the learner's needs.This technology utilizes artificial intelligence algorithms to engage learners in meaningful conversations, simulate real-life language use, and support self-directed learning.
Integrating advanced technologies, such as chatbots, in language learning has opened up new possibilities for English language learners.Kohnke et al. (2023) explained that ChatGPT, a cutting-edge language model, has gained attention as a potential tool for enhancing language acquisition.However, it is crucial to examine the perceived benefits and challenges associated with using ChatGPT from the perspectives of English language learners.Ali et al. (2023) indicated that English language educators and practitioners should leverage the strengths of ChatGPT while addressing these challenges to create a balanced and effective language learning environment.By doing so, learners can maximize the benefits of ChatGPT while supplementing their language learning journey with authentic human interactions and diverse learning experiences.
Many research has looked at how well chatbots may help with language learning and provide individual comments.One study by Lin (2023) examined the impact of chatbots on language learning outcomes among university students in a self-directed learning environment.Their findings indicated that using chatbots significantly improved students' speaking and writing skills, as the interactive nature of the technology allowed for practice and feedback tailored to individual needs.Another relevant study by Petrović and Jovanović (2021) focused on the benefits of chatbots in vocabulary acquisition.The researchers found that learners who engaged with chatbots experienced increased vocabulary retention and usage, highlighting the potential of this technology to support vocabulary expansion.Furthermore, Muñoz et al. (2023) investigated the role of chatbots in enhancing learners' motivation and engagement.Their findings revealed that the interactive and dynamic nature of chatbots fostered a positive learning environment, motivating students to engage in language learning activities actively.
In a self-directed learning context, English language learning has undergone significant transformations with the integration of chatbots, such as ChatGPT.Shidiq (2023) added that these advanced language learning tools offer personalized and interactive features that have the potential to revolutionize language acquisition.The traditional approach to language learning often relies on classroom instruction and teacher-centred methodologies.However, Morris (2019) stated that self-directed learning had become an alternative pedagogical framework emphasizing learners' autonomy and self-regulation.In this context, learners take charge of their learning process, setting goals and actively engaging in self-study.Sun and Hoelscher (2023) described that integrating chatbots in self-directed learning could enhance learners' language skills, providing them with immediate feedback, tailored resources, and opportunities for independent practice.
Existing research has primarily focused on the technical aspects of chatbot systems and their potential as language tutors (Adıgüzel et al., 2023;Jeon and Lee. 2023;Kohnke et al., 2023).However, limited attention has been given to the specific benefits and challenges in self-directed learning contexts, where learners have a higher degree of autonomy and responsibility for their learning progress.Understanding the unique advantages and obstacles associated with utilizing ChatGPT in self-directed learning environments can provide valuable insights for educators, learners, and curriculum designers.Furthermore, while studies have highlighted the potential benefits of personalized feedback and guidance offered by ChatGPT, there is a lack of empirical evidence regarding its impact on language proficiency and self-assessment accuracy.Investigating the extent to which ChatGPT enhances learners' language skills and their ability to assess their progress accurately can shed light on the efficacy of this tool in self-directed learning contexts.To address the research as mentioned earlier gap, the researcher in this study explore the following research questions: 1) How does the integration of ChatGPT affect the development of English language skills in a self-directed learning context, and 2) How does personalized feedback provided by ChatGPT impact students' language development and self-assessment accuracy in English language learning?

METHOD
The study uses a mixed-methodologies strategy, which includes both quantitative and qualitative research methods.A sample of English language learners was selected from university courses, and the study was conducted over a specified period.A convergent analysis approach was employed to integrate the quantitative and qualitative data collected from the study.This approach allows for comprehensively examining the research question by merging findings from multiple data sources and methods.
The research specifically focuses on using ChatGPT, an advanced chatbot powered by artificial intelligence, as a language learning tool.ChatGPT provides personalized feedback and interactive conversations to enhance language development in a self-directed learning context.The study involved a sample of 100 English language learners from selected universities in South Jakarta, such as Universitas Budi Luhur, Universitas Prof. Dr. Moestopo (Beragama), Universitas Mercubuana, and Universitas Esa Unggul.The research sample was determined using a stratified random sampling method.This sampling approach was utilized in order to guarantee inclusivity across various academic departments and academic years.The participants represented a diverse range of English proficiency levels, from beginner to advanced, ensuring a comprehensive exploration of the effects of utilizing ChatGPT in different language learning contexts.
A structured questionnaire was designed to collect quantitative data.The questionnaire included Likert-scale and multiple-choice questions, focusing on participants' perceptions of the benefits and challenges of using ChatGPT, their selfdirected learning experiences, and their language proficiency levels.The online interview protocols were guided by open-ended questions, allowing participants to elaborate on their opinions, share specific examples, and express their thoughts more freely.
Participants' self-reported learning experiences and language proficiency levels, as well as their thoughts on the advantages and drawbacks of utilizing ChatGPT, were summarized in the quantitative data.We looked for trends in the interview transcripts and used them to develop overarching themes and subthemes.This analysis helped us better grasp how participants felt about and thought about utilizing ChatGPT for independent study.The convergence of findings was examined by comparing and contrasting the questionnaire and interview data results.This integration provided a comprehensive and nuanced understanding of the benefits and challenges of utilizing ChatGPT in English language learning within the self-directed learning context of Indonesian universities.

FINDING AND DISCUSSION
The convergent analysis approach was employed to integrate and synthesize the quantitative and qualitative data, providing a comprehensive understanding of the benefits and challenges of utilizing ChatGPT as a language learning tool in a self-directed learning context.By combining findings from both data sources, a holistic perspective emerged, shedding light on the multifaceted aspects of this innovative educational technology.The convergent analysis revealed valuable insights that bridge the gap between numerical data and in-depth participant experiences, enabling a nuanced exploration of the role of ChatGPT in English language learning.Research Question 1: How does the integration of ChatGPT affect the development of English language skills in a self-directed learning context The research question examines the effects of ChatGPT integration on various language skills, including grammar, vocabulary, fluency, and overall proficiency.By delving into the findings, we can better understand how using ChatGPT as a tool for language practice and support influences language skill development.This analysis contributes to the existing literature on the role of chatbots in English language learning, shedding light on the effectiveness of integrating advanced language models in self-directed learning environments.Pre-and post-integration language proficiency scores were collected to measure the development of English language skills.Paired t-tests tests were conducted to determine if there were significant differences in language proficiency before and after integrating ChatGPT.The research findings present a detailed analysis of the impact of integrating ChatGPT into different aspects of language learning, revealing varied outcomes.Firstly, when assessing the integration of ChatGPT in learning instruction, a significant improvement was observed.The average score increased from 65 to 72 post-integration, backed by a T-value of 2.32 and a p-value of 0.032, indicating that this improvement is statistically significant.This suggests that ChatGPT effectively enhanced learning instruction.In contrast, the impact on proficiency levels presented a different picture.Although there was an increase in average scores from 58 to 63, the T-value of 1.21 and a p-value of 0.215 suggest that this rise is not statistically significant, indicating that the influence of ChatGPT on language proficiency levels was not as pronounced.Finally, the aspect of self-directed learning showed the most substantial improvement, with scores rising from 72 to 78.The high T-value of 3.87 and a p-value of 0.001 strongly suggest that this improvement is statistically significant, underscoring the positive impact of ChatGPT on enhancing self-directed learning.Overall, these findings highlight the potential of ChatGPT in improving certain aspects of language learning, particularly in instructional and self-directed contexts, while its effect on language proficiency remains less clear.Using ChatGPT in a self-directed learning context allows learners to engage in conversational exchanges, facilitating language practice and providing valuable feedback.(Kasneci et al., 2023).Table 3 shows that ChatGPT integration raised participants' language skills mean score from 68.32 to 73.57.The standard deviation rose to 7.43 from 6.91.ChatGPT can cater to learners' specific proficiency levels, learning goals, and areas of interest through adaptive responses and activities.This personalized approach maximizes engagement and effectiveness by aligning the learning content with learners' unique requirements.Consequently, Tsai (2023) believed learners are more likely to stay motivated and actively participate in language learning.Another significant advantage of integrating ChatGPT into self-directed learning is the availability of support around the clock.Learners can access the chatbot anytime, transcending temporal constraints often associated with traditional classroom settings.

Grammar Mastery:
Focuses on the understanding and application of sentence types, including declarative, interrogative, imperative, and exclamatory sentences.

Learning about different sentence types has enhanced my communication skills. I can now vary my sentence structures to convey different shades of meaning.
Explores the mastery of different verb tenses, such as present, past, future, perfect, and continuous.

Reading Comprehension:
Focuses on analyzing the main ideas, supporting details, and author's purpose in written texts

The textual analysis allows me to evaluate arguments and evidence in academic articles critically. It helps me identify logical fallacies and assess the validity of claims.
Explores the ability to conclude, make inferences, and interpret implicit information from the text.

Making inferences from the text helps me fill in gaps and understand implied information. I can grasp the author's intended message and draw conclusions by reading between the lines.
Examines understanding text features, such as headings, subheadings, paragraphs, and their impact on comprehension.
Recognizing text structures such as cause and effect or problem-solution has improved my ability to comprehend written material quickly.It helps me identify the main ideas and the relationships between them.

Writing Proficiency:
Focuses on the organization and structure of essays, including introduction, body paragraphs, and conclusion.The personalized nature of ChatGPT holds immense potential in tailoring language learning experiences to individual learners' needs (Kasneci et al., 2023).Table 3 shows that ChatGPT integration raised participants' language skills mean score from 68.32 to 73.57.The standard deviation rose to 7.43 from 6.91.ChatGPT can cater to learners' specific proficiency levels, learning goals, and areas of interest through adaptive responses and activities.This personalized approach maximizes engagement and effectiveness by aligning the learning content with learners' unique requirements.Consequently, Tsai (2023) believed learners are more likely to stay motivated and actively participate in language learning.Another significant advantage of integrating ChatGPT into self-directed learning is the availability of support around the clock.Learners can access the chatbot anytime, transcending temporal constraints often associated with traditional classroom settings.Firat (2023) confirmed that by interacting with a language model, learners receive immediate responses that help identify and rectify errors, ultimately promoting language accuracy.Moreover, Wu et al. (2023) asserted that this real-time feedback cultivates an environment conducive to continuous improvement and fosters an in-depth understanding of grammar rules and sentence structures.
Despite the advantages of integrating ChatGPT in self-directed learning, it is crucial to acknowledge potential limitations and counterarguments.Baidoo-Anu and Ansah (2023) argued that the absence of human interaction might hinder the development of crucial communicative skills, such as speaking and listening.The thematic analysis findings highlight the positive impact of integrating ChatGPT in self-directed English language learning.Li et al. (2023) identified that themes include Vocabulary Acquisition, Grammar Mastery, Reading Comprehension, and Writing Proficiency.Participants reported improvements in specific areas, such as learning context-specific vocabulary, understanding word formation, sentence structure, verb tenses, textual analysis, making inferences, recognizing text structure, essay structure, coherence and cohesion, and using sentence variety.Cooper (2023) acknowledged that learners contend that face-to-face communication with teachers and peers is essential for acquiring and refining these skills.While this concern is valid, it is important to recognize that ChatGPT can supplement rather than replace interpersonal interactions.Learners can still seek opportunities for real-time conversations with language partners or participate in language exchange programs to augment their speaking and listening abilities.

Research Questions: How does personalized feedback provided by ChatGPT impact students' language development and self-assessment accuracy in English language learning?
The intention of this study's results and discussion is to examine how students' language growth and self-assessment accuracy are affected by receiving individualized feedback through ChatGPT while learning English.The research question seeks to explore the effects of ChatGPT's tailored feedback on students' language skills, as well as their ability to assess their proficiency levels accurately.table presents the results of an independent samples t-test performed on the differences between the pre-and post-test scores to examine the effect of ChatGPT on linguistic growth.The difference in mean scores between the two periods is represented by the mean difference.The dispersion of sample means is quantified by the standard error.The hypothesis is supported by a t-value of 3.91 and a p-value lower than 0.001.The difference between the two sets of scores is statistically significant.As the p-value is less than 0.001, likely, the alternative hypothesis that ChatGPT has no effect on language acquisition is false.7 presents the percentage of accurate self-assessments before and after using ChatGPT.The sample size is 100 students.Before using ChatGPT, 45% of the students accurately assessed their language skills, while 55% had inaccurate self-assessments.After using ChatGPT, there was an improvement, with 70% of students accurately self-assessing their language skills and 30% having inaccurate self-assessment."The feedback helped me set goals for myself.I used it to identify areas I wanted to improve and worked towards achieving those goals."

Improved Language Proficiency
The personalized feedback helped students identify and correct errors in their language usage, leading to enhanced accuracy and fluency."ChatGPT's feedback helped me identify and correct errors in my language usage.It improved the accuracy of my English." ChatGPT's feedback contributed to students' vocabulary development by suggesting new words and phrases in their language production."The feedback suggested new words and phrases for me to use.It expanded my vocabulary and helped me express myself more effectively."Students reported improvements in their grammar and sentence structure skills due to the personalized feedback, leading to more coherent and polished language production."I noticed improvements in my grammar and sentence structure skills thanks to the feedback.It made my language production more coherent and polished."

Enhanced Selfassessment
The personalized feedback encouraged students to reflect on "The personalized feedback encouraged me to reflect on my

Accuracy
their language abilities, enabling them to make more accurate selfassessments.
language abilities.It helped me assess my strengths and weaknesses more accurately."Students were able to take a more objective look at their language skills after receiving this feedback.
"Through the feedback, I better understood my language strengths and weaknesses.It helped me identify areas where I excel and need to focus on improvement."The personalized feedback promoted self-monitoring of language performance, enabling students to identify areas for improvement and track their progress.
"The promoted selfmonitoring of my language performance.It made me more aware of my progress and areas that required further attention."

Autonomous Learning
The personalized feedback empowered students to engage in self-paced learning, as they could access feedback and practice materials whenever convenient.According to the findings, students found the interactive feedback engaging, valued the guidance provided, and appreciated the real-world relevance of the feedback.The feedback also increased motivation by tracking progress, providing positive reinforcement, and supporting goal setting.It improved language proficiency through error correction, vocabulary expansion, and grammar enhancement.Furthermore, the feedback enhanced self-assessment accuracy by promoting self-reflection, identifying strengths and weaknesses, and facilitating self-monitoring.Students also enjoyed the autonomy of selfpaced learning, accessible resources, and self-directed practice facilitated by feedback.
The impact of personalized feedback provided by ChatGPT on students' language development and self-assessment accuracy has become a subject of extensive investigation.By delving into the complex scientific concepts involved, Fuchs (2023) confirmed that to elucidate the potential implications of personalized feedback in English language learning while ensuring clarity and accessibility for readers.As indicated in Table 5, the descriptive analysis of pre-test and post-test scores reveals a notable increase in mean scores from the pre-test (68.25) to the post-test (79.72).Chen et al. (2021) explained that this upward trend suggests that the integration of personalized feedback has had a positive impact on students' language development.The standard deviations of both pre-test and post-test scores (8.43 and 7.96, respectively) indicate relatively consistent and reliable data within the sample.Zhou (2023) asserted that personalized feedback refers to tailored and individualized guidance offered by ChatGPT, an advanced language model, to students engaged in learning English.
The independent samples t-test, as shown in Table 6, further confirms the significant improvement in language skills.The mean difference between the pre-test and post-test scores is 11.47, with a standard error of 2.14.The calculated t-value of 3.91 and the corresponding p-value of less than 0.001 indicate that the improvement is statistically significant, providing robust evidence of personalized feedback's positive impact on language development.Dai et al. (2023) emphasized that the feedback addresses learners' specific language needs, focusing on grammar, vocabulary, fluency, and overall language proficiency.Javaid et al. ( 2023) elaborated that it is a valuable resource for students to improve their language skills through targeted practice and constructive criticism.The impact of personalized feedback on students' language development is multifaceted.Sabzalieva and Valentini (2023) revealed that it provides learners with immediate and specific guidance, enabling them to identify and rectify errors in real-time.By highlighting areas of improvement and offering suggestions for enhancement, personalized feedback aids students in refining their language accuracy and gaining a deeper understanding of grammar rules, sentence structures, and idiomatic expressions.The integration of personalized feedback through ChatGPT has influenced students' self-assessment accuracy, as demonstrated by Table 7.Before using ChatGPT, only 45% of students had accurate selfassessments, whereas after integrating personalized feedback, the percentage increased significantly to 70%.This improvement indicates that learners could better understand their language proficiency levels and accurately evaluate their strengths and weaknesses.
Furthermore, personalized feedback fosters a sense of individualization and adaptability in the learning process.As ChatGPT analyzes learners' language production and progress, Qadir (2023) described that it could tailor its responses and activities to cater to each student's unique needs, proficiency levels, and learning styles.Alamri et al. (2020) stated that this personalized approach ensures learners receive targeted support aligned with their specific language goals, ultimately maximizing their language acquisition potential.In addition to language development, Su and Yang (2023) indicated that personalized feedback plays a vital role in enhancing self-assessment accuracy among English language learners.The integration of personalized feedback through ChatGPT has influenced students' self-assessment accuracy, as demonstrated by Table 7.Before using ChatGPT, only 45% of students had accurate self-assessments, whereas after integrating personalized feedback, the percentage increased significantly to 70%.This improvement indicates that learners could better understand their language proficiency levels and accurately evaluate their strengths and weaknesses.
By receiving detailed feedback on their language production, students better understand their strengths and weaknesses.This self-awareness enables them to gauge their language proficiency levels accurately and set realistic goals for improvement.Consequently, Lin (2023) asserted that students become more self-directed in their learning journey, as they can monitor their progress and make informed decisions regarding areas that require further attention and practice.Moreover, concerns have been raised regarding over-reliance on chatbots for language learning.Haleem et al. (2022) implied that it may reduce interpersonal communication skills, particularly in speaking and listening.While personalized feedback can enhance written communication and accuracy, it should be supplemented with ample opportunities for face-to-face interaction with teachers and peers to develop oral proficiency and conversational skills (Agustini. 2023).

CONCLUSION
Student's improvement in language and self-assessment consistency were shown to be significantly impacted by the individualized feedback they received via ChatGPT.Integrating ChatGPT in self-directed learning contributes to developing English language skills.The results indicate that personalized feedback helps students refine their language accuracy, deepen their understanding of grammar rules, and improve their proficiency in vocabulary usage and sentence structures.Moreover, the tailored guidance provided by ChatGPT enhances students' self-awareness, enabling them to assess their language proficiency levels accurately and set realistic goals for improvement.The integration of ChatGPT in a self-directed learning context further promotes the development of English language skills.Students benefit from the immediate and targeted support offered by ChatGPT, which caters to their specific language needs, proficiency levels, and learning styles.This personalized approach maximizes learners' language acquisition potential and fosters autonomy in their language learning journey.
It is important to acknowledge certain limitations that should be taken into account.Sample Size and Generalizability: The study's sample size, however meticulously determined, is only indicative of five institutions of higher education.To enhance the generalizability of findings, future research endeavors should consider broadening the scope of investigation to encompass multiple educational institutions.The potential limitations of ChatGPT lie in its ability to effectively represent and address cultural subtleties and emotional dimensions within communication.As a result, the chatbot's ability to effectively support comprehensive language acquisition may be impaired.Recommendations for Future Research A longitudinal study examining the utilization of ChatGPT and its influence on the progression of language acquisition can yield significant findings regarding the long-term performance of students.A comparative analysis can be conducted to assess the efficacy of ChatGPT in comparison to alternative language learning aids, such as mobile applications or video tutorials.This analysis aims to get insights into the specific strengths and limits of the chatbot.

Table 1 .
Demographic Participants of N:100

Table 2 .
The result of t-tests in language proficiency before and after integrating ChatGPT

Table 3 .
The result of descriptive statistics, including means and standard deviations

Table 4 .
The Thematic Analysis Result

Table 5 .
Descriptive Analysis of Pre-test and Post-test Scores

Table 6 .
Independent Samples t-test Results for Pre-test and Post-test Scores

Table 7 .
Percentage of accurate self-assessment using ChatGPT

Table 8 .
Thematic Analysis of Implications of Personalized Feedback Provided by ChatGPT