The present study conducted the translation quality of Korean neural machine translation in Chinese statutory provisions with automatic assessment and human judgement and compares and analyzes the results from the perspective of Chesterman’s expectancy norms among the descriptive translations studies.
The purpose of this is to examine whether Chinese legal texts of neural machine translation meets the reader’s expected standard to explore the translation quality of neural machine translation and how human evaluators can efficiently use automatic assessment.
This study targets a total of 648 sentences of 219 articles composed of the Constitution of People’s Republic of China, Chinese Nationality Act, Chinese State Reimbursement Act and Anti-foreign Sanctions Act provided by the World Laws Information Center under the Office of Legislation. The expected standards for legal text translation were set as accuracy, readability, and consistency in accordance with the legal translation guidelines of the Office of Legislation and the Korea Legislation Research Institute, which are public institutions in charge of legal translation in Korea. Machine translation was calculated through Naver Papago(N2MT) and Google Translation(GNMT) for legal texts, and evaluation was conducted by applying an automatic assessment model and human judgement model. The calculation date of the neural machine translation system was June 30, 2022 and three human evaluators participated in human judgement.
As a result of data analysis, both automatic assessment and human judgement except for Bleu score recorded 0.8 or higher. In terms of correlation, the correlation between automatic assessment was 0.60 to 0.85, showing a medium to high volume of correlation, and the correlation between human evaluators was 0.45 to 0.55, showing a weak or medium volume of correlation. The correlation between automatic assessment and human judgement showed the lowest correlation between Bleu score and human judgement with 0.22~0.26, Bert score and human judgement with 0.27~0.41 and Laser score with 0.00~0.18 respectively.
From the perspective of accuracy among expected standard, automatic assessment has advantages in accuracy because metrics have developed around accuracy, but errors have emerged that can’t completely judge from some sentences to contexts. In terms of human judgement, there were differences between evaluators in word layer, phrase layer, and text layer except sentence layer.
In terms of readability, automatic assessment and human judgement showed different patterns in judging the context. For some sentences that all human evaluators gave high scores, when vocabulary aspects or logical relationships changed, automatic assessment tended to judge them as errors.
For the consistency, in the case of automatic assessment, the same word is mixed and translated into different vocabulary even within a sentence, and it showed that it’s not unified with legal titles. In case of human judgement, overall consistency and coherency were judged with generally similar probabilities.
To recap, automatic assessment had advantages in the evaluation of ‘accuracy’ among the expected standard of legal texts, ‘accuracy’, ‘readability’ and ‘consistency,’ while ‘readability’ and ‘consistency’ were partially possible. In terms of human judgement, it had the advantage of being able to evaluate all of ‘accuracy’, ‘readability’, and ‘consistency,’ but there was a problem with the evaluator’s subjectivity. From the perspective of expected standard, to efficiently evaluate the machine translation of Chinese legal texts provisions, it’s necessary to consider items for ‘consistency’ and ‘weight factor’.
Analyzing the results of the present study, Korean neural machine translation is still difficult to completely meet the expected standard for Chinese legal texts, and it’s necessary to improve by technological development such as improving the similarity. In terms of human judgement, in order to use neural machine translation in the translation quality evaluation of Chinese legal texts done by human, methodological plan such as using statute books, analyzing, and synthesizing weak parts of neural machine translation and making evaluation criteria specialized in Chinese legal texts.