#evaluation

#translation #ai

In the context of a translation project, where we aim to utilize only the resources at hand, let’s consider a simplified approach to the evaluation stage. This approach would involve the following steps:

1. Start by taking the draft translation and identify the translation pairs that are most similar to the paired prediction.
2. Evaluate the draft translation for any potential misuse of words.
   1. One possible method could be to use a rules-based matcher that identifies any tokens in the source+prediction pair that only occur in EITHER the source OR the target in any of the similar examples.
   2. It could also be beneficial to examine whether any words seem to be mistranslated based on the gold standard examples.
   3. To aid this process, aligning these examples by phonetics, orthography, and apparent semantic similarities could be helpful.
3. Once the evaluation is complete, pass the task back to the translation bot, providing special notes about any words that appear to be mistranslated.

In essence, the process can be summarized as: [ most similar examples ] —> [ prediction ] —> [ most similar examples ]

The goal is to identify the most similar examples to the prediction, and then use those examples to evaluate the prediction. This process can be repeated iteratively, with the bot learning from the evaluation and then generating a new prediction and a new set of examples that will help the initial bot improve its output.

#translation #ai

Here’s an idea I had for translation quality checking.

The process begins by prompting for a translation using n examples and a directive to complete the translation pairs. The entire prompt, along with the AI’s completion, is then passed to the evaluator. The evaluator’s task is to determine which translation was generated by the AI. This may require fine-tuning a discriminator model on the gold-standard pairs.

The concept, inspired by Generative Adversarial Networks (GANs), involves an evaluator attempting to distinguish AI-drafted translations from a list of real translations. While it doesn’t function exactly like a GAN, the idea is to use a similar principle for evaluation.

The method could potentially be used to refine the prompts or the models themselves, by replacing the evaluator with a separate AI. Initial implementation will use Language Models (LLMs) to gauge the viability of the concept. If successful, this approach could be extended to train GANs for specific languages.

#translation #ai

In the realm of machine translation, achieving the highest level of accuracy is paramount. One approach to improve the precision of translations is through a method called “Token Matching Evaluation”. This method centres around the use of valid tokens, which are words or phrases that have appeared in sample translation pairs.