The Method for Comprehensive Quality Evaluation of Tests. Part 1

  • V. M. Kukharenko National Technical University “Kharkiv Polytechnic Institute”
  • L. P. Perkhun National Academy of Statistics, Accounting and Audit
  • N. M. Tovmachenko National Academy of Statistics, Accounting and Audit
Keywords: distance training, test task, test, validity, correlation, factor analysis

Abstract

Informatization of the modern society has led to the wide-scale and rapid introduction of distance training technologies in virtually all the categories of Ukrainian HEEs. Studies of the aspects related to applications of digital technologies in education processes are subject to close attention in Ukraine and beyond. An important component of the training process is test control of knowledge.   

Education activities at the National Academy of Statistics, Accounting and Audit rely on criteria oriented tests. They are realized on the basis of distance training system Moodle that allows for creating test questions of various types and their repeated inclusion in various packages of test tasks. The Moodle environment allows for computation of selected statistical indicators on a fulfilled test and its individual tasks: average estimate and median, standard deviation, asymmetry, excess, internal agreement rate, standard error etc. However, these characteristics are not enough for justified acceptance of test results.  

The article presents the first phase in elaborating a comprehensive method for quality evaluation of selected test tasks and the test as a whole. This method combining the classical theory, Data Mining and Item Response theory methods involves six steps. The first step, based on indicators of descriptive statistics, allows for evaluating the obtained distribution of test results. The second step involves evaluation of the validity of test tasks. The point-bead ratio is computed to derive the correlation between individual test task and individual test score of a student, with values higher than 0.5 considered satisfactory. Pearson correlation coefficient for binary variables shows the correlation between pairs of test tasks. The test tasks with negative correlations with the other test tasks are not considered as valid, and they have to be corrected or replaced. At the third step, the factor validity of the test is evaluated. The test tasks combined in groups using factor analysis methods are subject to further analysis to determine their impact on the final result, the individual test score of a student. All the above mentioned steps are illustrated by example. The computation is made by SPSS software package. The difference in interpretation of the computation results in each step for norm oriented and criteria oriented tests is demonstrated.              

The description of further steps involves in the method for comprehensive quality evaluation of tests, which use Data Mining and Item Response Theory methods, will be continued in next publications.

Downloads

Download data is not yet available.

References

1. Kulaha, I. V., Ilnytskyi, D. O., Strelnyk, S. O., Matviichuk, A. V., Vasylkova, N. V., & Turchaninova, V. M., et al. (2013). Svitovyi dosvid organizatsii ta rozvytku universytetskoi systemy dystantsiinoho navchannia [World experience of organization and development of university system of distance learning]. Kyiv: Instytut vyshchoi osvity KNEU imeni Vadyna Hetmana. Retrieved from https://kneu.edu.ua/userfiles/education2_0/13-4713_verstka%281%29.pdf [in Ukrainian].
2. Sinytskyi, M. Ye. (2015). Statystychni insrumenty vymiriuvannia yakosti osvity. Ch. 2. Klassychnyi pidkhid [Statistical tools for measuring the education quality. Part 2. Classical approuch]. Naukovyi visnyk Natsionalnoi akademii statystyky, obliku ta audytu. – Scientific bulletin of the National Academy of Statistics, Accounting and Audit, 1, 75–86.
3. Deryhlazov L. V., Kukharenko, V. M., Perkhun, L. P., Tovmachenko, N. M. (2017). The Models of Distance Forms of Learning in National Academy of Statistics, Accounting and Audit. Naukovyi visnyk Natsionalnoi akademii statystyky, obliku ta audytu – Scientific bulletin of the National Academy of Statistics, Accounting and Audit, 3, 79–90 [in English].
4. Deryhlazov, L. V., Kukharenko, V. M., Perkhun, L. P., Tovmachenko, N. M. (2018). Statystychni metody analizu rezultativ testovoho kontroliu znan [Statistical methods for analyzing the results of knowledge test control]. Proceedings from MoodleMoot Ukraine 2018. Theory and Practice of Using the Moodle Learning Management System: Shosta mizhnarodna naukovo-praktychna konferentsiia (25 travnia 2018 hoda) – Sixth International Scientific and Practical Conference. ( p. 28). Kyiv, KNUBA [in Ukrainian].
5. Mokriiev M.V. (2017). Analiz testovykh zavdan zasobamy Moodle [Analysis of test tasks by means of Moodle]. Proceedings from MoodleMoot Ukraine 2017. Theory and Practice of Using the Moodle Learning Management System: Shosta mizhnarodna naukovo-praktychna konferentsiia (26–27 travnia 2017 hoda) – Sixth International Scientific and Practical Conference. (p. 18). Kyiv, KNUBA [in Ukrainian].
6. Quiz statistics report. MoodleDocs. docs.moodle.org. Retrieved from https://docs.moodle.org/32/en/Quiz_statistics_report
7. Kukhar, L. O., & Serhiienko, V. P. (2010). Konstruiuvannia testiv [Create a quality test]. Lutsk [in Ukrainian].
8. Tovmachenko, N. M. (2007). Osnovni kharakterystyky paketa statystychnykh prohram Statistical Package for Social Science ta yoho zastosuvannia v prykladnii statystytsi [The basic properties of Statistical Package for the Social Science and its use in the applied statistics] // Naukovyi visnyk Derzhavnoi akademii statystyky, obliku ta audytu – Scientific bulletin of the State Academy of Statistics, Accounting and Audit, 2, 11–16 [in Ukrainian].
9. Anastazi, A. & Urbina, S. (2001). Psikholohicheskoe testirovanie [Psychological Testing]. Sankt-Peterburg: Piter [in English].

Abstract views: 564
PDF Downloads: 390
Published
2018-09-04
How to Cite
Kukharenko, V. M., Perkhun, L. P., & Tovmachenko, N. M. (2018). The Method for Comprehensive Quality Evaluation of Tests. Part 1. Statistics of Ukraine, 82(3), 40-48. https://doi.org/10.31767/su.3(82)2018.03.04

Most read articles by the same author(s)