General information
Title: Assessing reading literacy and comprehension of early graders in Bulgaria and Italy
Duration: 2023 – 2025
 |
 |
Project leader of the Bulgarian team: prof. Svetla Koeva
Bulgarian team members: Ivelina Stoyanova, Valentina Stefanova, Tsvetana Dimitrova, Maria Todorova, Hristina Kukova |
 |
Project leader of the Italian team: prof. Vito Pirrelli
Italian team members: Claudia Marzi, Marcello Ferro, Andrea Nadalini, Alessandro Lento |
Main goal: To increase the level of literacy and reading abilities of primary school children through education. To contribute to the achievement of this goal, a thorough investigation focused on assessing reading literacy and comprehension in children of early school age in Bulgaria and Italy will be carried out.
Funding: the Bulgarian Academy of Sciences and the National Research Council of Italy
The project is performed in collaboration with 21st Hristo Botev Secondary School in Sofia.
Objectives
The project has the following objectives:
– To implement, and test assessment strategies for monitoring and evaluating the reading skills and the level of word comprehension of Bulgarian and Italian early graders.
– To collect reading and comprehension evidence from the two populations of children according to the same battery of tools and comparable, highly-controlled test language materials.
– To compare collected data across children from different age and social groups and with different languages (Bulgarian and Italian) and model the results.
– To document and make available evidence-based procedures, protocols, and tools for reading and word comprehension assessment.
Research method
The ICT platform with a tablet front-ent developed by the Comphys Lab at CNR ILC in Pisa will be used to provide accurate, evidence-based assessment of reading skills in early grade children. The platform, named Readlet, can automatically collect, preprocess and analyse time-aligned multimodal reading data that include: voice recording, finger sliding time, time spent to answer comprehension questions, and number of correct answers.
We will design a set of connected reading texts, containing short stories in Bulgarian and Italian, rigorously balanced and comparable in terms of their levels of readability and linguistic complexity.
Using state-of-art NLP tools, the reading texts will be annotated at different levels of linguistic analysis: from articulatory complexity (i.e. length and variety of consonant chunks) and phonological transparency, to part-of-speech tagging, lexical typicality (in terms of density and entropy of a word’s lexical neighbourhood), orthotactic probability (as a function of a word’s bigram and trigram probabilities), morphological complexity, token and type frequency, token’s position and syntactic role in a sentence.
By aligning finger tracking data with the annotated reading text and audion records, we will be in a position to relate children’s reading performance to specific linguistic factors across different grade levels, thereby gaining a better understanding of i) the basic mechanisms behind children’s reading strategy, ii) what text factors make reading more difficult, iii) and ways to enhance a reader’s strengths and overcome her weaknesses.
Meetings
Work meeting, online, 29.06.2023
On 29th June 2023 the first working meeting of the project team was held. The meeting was attended by all project members from the Institute for Bulgarian Language – prof. Svetla Koeva, assoc. prof. Tsvetana Dimitrova, assist. prof. Valentina Stefanova, assist. prof. Maria Todorova, assist. prof. Hristina Kukova (postdoctoral student), Ivelina Stoyanova, as well as by the Italian partners from the Institute for Computational Linguistics at the National Research Council of Italy – prof. Vito Pirrelli, senior researcher Claudia Marzi, researcher Marcello Ferro, researcher Andrea Nadalini, Alessandro Lento (PhD student).
All participants introduced themselves briefly and overviewed the general objectives during the first stage of the project. Prof. V. Pirrelli reviewed the work of the Italian team, presenting options for tracking the dynamics of the reading process, as well as checking reading comprehension in different age groups. The parameters and features of the science experiment implementation system (ReadLet) were discussed.
 |
 |
 |
Work meeting, Pisa, 6 – 8.11.2023
In the period 6th – 8th November 2023, prof. Svetla Koeva, assoc. prof. Tsvetana Dimitrova and Ivelina Stoyanova made a working visit to the Institute for Computational Linguistics A. Zampoli in Pisa, Italy. A series of working meetings were held with the Italian team – prof. Vito Pirrelli, senior researcher Claudia Marzi, researchers Marcello Ferro and Andrea Nadalini, and Alessandro Lento (PhD student). During the meetings, a general presentation of the project was made by the Italian team and a discussion was held on the main objectives, the ReadLet reading skills assessment application developed and used by the Italian team, as well as the main features and problems in data collection. Dr. Tsvetana Dimitrova presented a set of tasks for online experiments in the form of language games developed by the Bulgarian team, as well as a summary and analysis of the obtained results. The results of the Italian team’s research were also presented (comparative studies on children with typical language development and children with autism spectrum disorders, children in monolingual and bilingual environments, adults). Attention was paid to the advantages of the system and the method of finger tracking compared to other methods of evaluating reading skills (e.g., eye tracking), as well as the parameters of the campaign that is planned to be carried out in Bulgaria. The technical approaches to ensure the comparability of the Italian and the Bulgarian texts were discussed, as well as the possible analytical tools to be applied to process the results.
 |
 |
 |
Work meeting, Sofia, 5 – 6.12.2023 г.
On 5th and 6th December 2023, prof. Vito Pirrelli and researcher Andrea Nadalini visited the Institute for Bulgarian Language Prof. Lyubomir Andreychin, where they met the Bulgarian team: prof. Svetla Koeva, assist. prof. Maria Todorova, assist. prof. Hristina Kukova and assist. prof. Valentina Stefanova.
During the series of meetings prof. Pirrelli and researcher Nadalini explained the work with the ReadLet application, as well as the principles for conducting the experiments, according to their practice in Italy. Andrea Nadalini talked about the methods of data collection for the experiments, as well as the subsequent analysis and statistical processing. A meeting was held with Mrs. Maya Angelova, head-teacher of the 21. Hristo Botev Secondary School in Sofia, in order to discuss cooperation between the Bulgarian team and the school. The working schedule for the coming months, the means of communication and the expected results were outlined. On 6th December prof. Vito Pirrelli gave a lecture to a wider audience at the Bulgarian Academy of Sciences in order to promote the objectives and the expected results of the project.
 |
 |
 |
Results
In 2023 the following activities were carried out:
- Adaptation and customisation of existing protocols and software products to create language tasks and collect and store data on reading assessments of primary school children for both languages.
The ReadLet platform developed by the Italian team is used to automatically collect, pre-process and analyse synchronised multimodal reading data, which includes: voice recording, finger sliding time recording, response time to comprehension questions on the text, and the number of correct answers. Bulgarian texts were integrated into the platform to perform data collection on Bulgarian, and initial tests were carried out. The platform was localised for Bulgarian.
- Assessment of data protection and privacy requirements.
The European and Bulgarian legislation on personal data protection was studied and a package of documents necessary for conducting the experiment at school was prepared – an invitation to cooperate (with professionals and schools), an invitation to participate, a sample letter of consent from a parent or guardian.
- Design and development of comparable language materials and language tasks suitable for assessing reading skills and levels of information decoding and content comprehension during reading for primary school children for both languages.
The texts were created specifically for the intended research purposes in Italian. The texts have a specific structure: each of them consists of five episodes, and the level of difficulty of the text increases with each subsequent episode. After each episode, two questions are asked about the text with four possible answers, from which the correct one can be chosen.
The texts have been translated into Bulgarian. The translation was further adapted to match the intended complexity (sentence length calculated as the average number of words in a sentence and word length calculated as the average number of letters in a word), lexical characteristics (percentage of unique words in the text vs. selected for age reference list; distribution between ‘core vocabulary’, words with high frequency of use and words with relatively lower frequency referring to everyday objects or actions) and morpho-syntactic features (depth of the syntactic tree or longest path from the root of the tree to the leaves / words; the average depth of the syntactic tree of subordinate clauses; the word order of subordinate clauses; the length of dependent links, etc.).
- Design and development of controlled comparable language materials and language tasks suitable for assessing the level of knowledge of core vocabulary and its appropriate use by primary school children for both languages.
The experiments to assess the understanding of the meaning of a selected set of words were conducted in a partially controlled manner during the Sofia Science Festival (11-14.05.2023) and the European Day of Languages (30.09.2023) in Sofia with 73 respondents. The work on the creation of the tasks and the conduct of the experiments, as well as the analysis of the obtained results, were presented to the Italian team during a meetingp in Pisa (6.11.2023).
- Defining target groups and agreeing data collection with school authorities and parents.
The experiment in Italy was conducted with 120 students from 2nd to 5th grade. In Bulgaria, cooperation was established with the 21st Hristo Botev Secondary School in Sofia to conduct the experiment with 80 students from 2nd to 5th grade. Consent data from parents or guardians is being processed.
Presentations
During the Eighth Forum on Research Approaches in Bulgarian Language Education, which took place on 1st November 2023 in the Acad. Alexander Teodorov-Balan Hall at the Institute for Bulgarian Language in Sofia, the project Assessing reading literacy and comprehension of early graders in Bulgaria and Italy was presented to scientists, university lecturers, teachers and Bulgarian language experts. The paper was submitted for anonymous review to the Bulgarian Language Journal.
 |
 |
 |
During the International Conference Bulgarian Linguistic Studies, held on 23rd and 24th November 2023 at Sofia University, the Bulgarian team introduced the objectives and the tasks of the project, as well as the experiments planned to assess reading skills of children from the project’s target group. The paper is expected to be published next year in the Conference proceedings.
 |
 |
 |
On 6th December 2023, in the Ivan Evstratiev Geshov Hall (207) at the Bulgarian Academy of Sciences, prof. Vito Pirrelli gave a lecture on the topic Assessing reading literacy with the finger-voice span. The lecture took place amid increased interest from the academic community – representatives of the Bulgarian Academy of Sciences, university lecturers, students and teachers, as well as experts from the Ministry of Education and Science were present.
 |
 |
 |
Publications
Светла Коева, Валентина Стефанова, Ивелина Стоянова, Мария Тодорова. Оценка на уменията за четене в начална училищна възраст. – Български език. Приета за печат. (Svetla Koeva, Valentina Stefanova, Ivelina Stoyanova, Maria Todorova. Assessment of reading skills in primary school age. – Bulgarian Language. Accepted for publication.)
Vito Pirelli, Svetla Koeva. Developing Materials for Assessing Reading Literacy and Comprehension of early Graders in Bulgaria and Italy. – Foreign Language Education. Accepted for publication.
