Few-shot translation evaluation

#translation #ai

In the context of a translation project, where we aim to utilize only the resources at hand, let’s consider a simplified approach to the evaluation stage. This approach would involve the following steps:

1. Start by taking the draft translation and identify the translation pairs that are most similar to the paired prediction.
2. Evaluate the draft translation for any potential misuse of words.
   1. One possible method could be to use a rules-based matcher that identifies any tokens in the source+prediction pair that only occur in EITHER the source OR the target in any of the similar examples.
   2. It could also be beneficial to examine whether any words seem to be mistranslated based on the gold standard examples.
   3. To aid this process, aligning these examples by phonetics, orthography, and apparent semantic similarities could be helpful.
3. Once the evaluation is complete, pass the task back to the translation bot, providing special notes about any words that appear to be mistranslated.

In essence, the process can be summarized as: [ most similar examples ] —> [ prediction ] —> [ most similar examples ]

The goal is to identify the most similar examples to the prediction, and then use those examples to evaluate the prediction. This process can be repeated iteratively, with the bot learning from the evaluation and then generating a new prediction and a new set of examples that will help the initial bot improve its output.