Investigation of processing test results based on knowledge similarity

Abstract

The traditional scoring is based on the difficulty of the task. However, the same total score can be earned with different knowledge, hence it is difficult to create homogenous groups by only relying on the total score. In our work, we aim to present a new scoring method where such knowledge-based groups can be created, as opposed to the previous point-based method. For comparison of the test result done by the students, we utilized different distance measures. The main challenge with finding the similarities between the results is the high dimensionality of the data compared to the total number of observations. First, we used the traditional Minkowski distance with different p values, then we used local similarity hashes and high dimensional embedding techniques designed originally for natural language processing. With these never before used techniques, we were able to identify students with similar skillsets and knowledge. Furthermore, we utilized dimension reduction methods (t-SNE and UMAP) to make a lower dimension representation where we can cluster the data easier. These clusters and pairwise similarities were assessed by oral exam subjective scores. The teacher's subjective scores correlated more with this new metric-based method of scoring than the total test score itself. The presented procedures can not only be used effectively in research but can also help to get a more complete picture of the students' knowledge we teach in everyday practice.