#low-resource

Key Features

• Compares languages using 11 different metrics
• Works with small datasets (even just a few translation pairs - this is huge for Bible translation into ultra-low resource languages)
• Language-agnostic approach
• Easy to install and use

The Road Ahead

It’s important to note that langsim is very much a work in progress. It’s not recommended for production use yet, and there’s plenty of room for improvement. The current metrics are just a starting point, and we’re actively seeking feedback and contributions.

One exciting direction for langsim’s future is leveraging Large Language Models (LLMs) to evolve its language comparison capabilities. Inspired by Stanford’s fascinating DrEureka project, we’re exploring ways to use LLMs to fine-tune the weights of our comparison metrics, potentially leading to more accurate and nuanced language comparisons.

Get Involved

If you’re interested in language comparison, computational linguistics, or just enjoy tinkering with code, we’d love your input. Try out langsim, open issues with your impressions, or contribute to its development.

Let’s push the boundaries of what’s possible in language comparison, one metric at a time!

Links

• Github Repository
• PyPi

1. Adding New Tokens to Large Language Models (LLMs)

The Problem: LLMs like the GPT models are trained on vast datasets, predominantly in high-resource languages. When encountering a low-resource language, these models struggle due to a lack of appropriate tokens and training data.

The Solution: Expanding the LLM’s vocabulary by training a tokenizer on new, low-resource language text and adding these tokens to the model. This approach has been exemplified by the SIL AI team’s experiments with the NLLB (No Language Left Behind) model (led by Michael Martin, details shared by Bethany Moore and Matthew Shannon as part of the presentation linked above). I can’t actually track down any links to their work here, but I suspect they will share more public information at some point. Some of the solution was based on forum discussions like this one.

How It Works:

• Train a tokenizer (e.g., a SentencePiece model) on new text from a low-resource language.
• Integrate these new tokens into an existing multilingual model. You need to make sure the model prioritizes these new tokens.
• Fine-tune the model with the new training data (in SIL’s case, use an existing New Testament translation in the target language for fine tuning).
• Predict some new text, such as the Old Testament.

This method apparently has improved the base NLLB results quite a bit, though I don’t have any numbers to share. I’m not sure if they’ve tried this approach with other models, but I suspect it would work with any multilingual model.

2. Cipher-Based Approach

The Problem: Standard tokenization methods may not capture the linguistic nuances of low-resource languages, leading to inaccurate or nonsensical translations, or the model might get stuck outputting some unknown token over and over.

The Solution: Utilizing a cipher-based approach to mask inputs and outputs, encouraging the model to rely on linguistic patterns via in-context learning (since the model is simply observing character level patterns rather than inappropriately associating the text in question with tokens mistakenly “recognized” from other languages in the training data). This technique can prevent dominant language probabilities or unknown-token nonsense from overshadowing the target language’s unique characteristics.

How It Works:

• Masking the text of a low-resource language using a cipher.
• Training the model to recognize and decode these patterns.
• Allowing the model to generate outputs based on learned patterns rather than defaulting to high-resource language structures.

This approach is particularly useful for languages that share syntactic or phonetic similarities with high-resource languages, reducing the risk of misinterpretation.

See the early experiments by Chris Priebe. I suspect there is a lot more that can be done on this approach, and the use of in-context learning seems like a huge win from my perspective (cf. this post).

3. Logits Warping

The Problem: LLMs may favor more common tokens from high-resource languages, leading to poor representation of low-resource languages.

The Solution: Logits warping involves adjusting the probabilities of certain tokens to bias the model towards a specific language or grammatical structure.

How It Works:

• Whether on the fly, or via a grammar definition, adjust the model’s logits, which are the model’s output predictions, to favor tokens from the target low-resource language.
• This biasing can force the model to generate text in the intended language, adhering to its unique grammatical rules.

This technique is particularly effective for languages with distinct grammatical structures or non-latin but in-training scripts, ensuring that the model’s output aligns more closely with the target language’s syntax and orthography.

I’ve done a few experiments with this approach, and it seems incredibly promising. I will probably write some more about how it might be combined with a system network in the flavour of Systemic Functional Linguistics (SFL) in the future.

Here’s a short video clip showing how you can specify a list of tokens (I just passed it [token1, token2, token3] as the possible outputs). The model then generates text that only includes those tokens.

Here’s a second short clip where I force the model to only output Japanese characters.

Conclusion

Tokenizing low-resource languages presents unique challenges in the field of NLP. However, by employing innovative approaches like adding new tokens, cipher-based methods, and logits warping, we can significantly improve the representation and understanding of these languages in LLMs. This not only enhances the accuracy of language models but also promotes linguistic diversity and inclusivity in the digital space. As we continue to advance in NLP, it’s crucial to keep exploring and refining these techniques to better serve the global community of diverse language speakers.

Opening up the black box

One of the biggest challenges with AI-based translation is that it works like a black box. You feed in some data, and you get some data out. But you don’t really know what’s going on inside the black box. You don’t know what the model is learning, or how it’s learning it. You don’t know what features it’s learning, or how it’s combining them to produce the output. This should rightly make translators somewhat nervous. How can you trust the output of a model that you don’t understand?

My vision for translation assistance with AI is thus to pry open the black box by using LLM-based predictions, rather than simple sequence-to-sequence predictions. This is a fundamentally different approach, and it relies on the concept of in-context learning, rather than a fine-tuning or transfer learning approach.

In-context learning

In short, in-context learning involves providing “training data” to the model on the fly, in the prompt. (Read more from Stanford AI Lab).

This is a fundamental difference that is really critical for translation assistance. It means we don’t need to take a bunch of data (e.g., a complete New Testament translation) in order to fine-tune a model for generating more data (e.g., a complete Old Testament translation). Instead, we can take a small amount of data (e.g., a few verses of the New Testament) and use it to generate more data (e.g., the next few verses of the New Testament, or some similar verses being drafted in the Old Testament).

We get two big benefits from this approach:

1. Bootstrapping: We can use a small amount of data to generate a large amount of data. This is a huge benefit for low-resource languages, since we can use the data we have (which is not enough for fine-tuning a model) to generate more data, and then use that new data (again, without re-fine-tuning our model) to generate even more data, and so on. This is a virtuous cycle that can help us bootstrap our way to a complete translation.
2. Instant feedback: We get instant updates, allowing the real translator (a human), to provide feedback to the model on the fly.

What do we need to make in-context learning work throughout the translation process?

To leverage in-context learning, we need to be able to draft new translations using few-shot examples (e.g., use a few already-translated verses to draft a new verse). We also need to be able to evaluate those translations using few-shot examples (e.g., use a few already-translated verses to evaluate a new verse). Optionally, we might want or need a back-translation in order to enable third party evaluation (such as a translation consultant).

Working backwards, then:

• To evaluate, I need
  • a metric
  • [optional] a back-translation
• To back-translate, I need
  • a translation
  • a non-hallucinating technique
    • translate back to Greek?
    • break down by punctuation?
    • Constrain using valid_tokens (see Token matching evaluation)
• To translate, I need
  • a chunk type (a unit of translation)
    • could be a Bible verse
    • could be a larger semantic unit that looks like a phrase
    • I would not recommend using a single word, since a word is a structural unit, and structures are precisely what gets left behind in translation
  • a source text
  • few-shot examples
  • [optional] supporting data
    • glossary
    • notes
    • evaluation feedback on prior drafts
  • [optional] constrain with valid_tokens
  • [optional] a concise, prosaic description of the most relevant source language data
  • [optional] a concise description of the language-typological differences between the source and target languages

Conclusion

Translation assistance can leverage LLMs and in-context learning to open up the black box of AI-based translation. This approach can help us bootstrap our way to a complete translation in a manner that cannot be achieved using traditional machine translation and typical AI-based approaches such as sequence-based model fine-tuning, and it can accept instant feedback from the translator, which means that every single correction or improvement made by the translator gets leveraged in the very next verse or chunk being drafted.

Note: you can also use an approach like this for a multi-agent simulation. See also the Social Approach to Low-Resource Language Translation.

The Social Approach

In the social approach, each AI agent takes on a role in the translation process:

• Translator: The translator agent’s role is to translate the text from the source language to the target language. It’s trained to generate translations that are as close as possible to the target language’s natural language usage.

• Back-Translator: The back-translator receives the output from the translator and translates it back to the original language. This helps ensure that the meaning of the text is preserved in the translation process.

• Evaluator: The evaluator compares the original text and the back-translated text to assess the quality of the translation. It might use metrics like BLEU or other language-specific criteria to provide a quantitative measure of the translation’s quality. It could also generate qualitative questions about the draft translation and attempt to answer those questions issuing a variety of tools (as in LangChain tools).

• Project Manager (Optional): This role could manage feedback from external stakeholders or experts, providing a higher-level, holistic evaluation of the translation. It’s responsible for integrating this feedback to guide the other agents.

Contrast with Swarm and Hive-Mind Approaches

Unlike the swarm-based and hive-mind approaches, which are inspired by natural systems and involve agents working in parallel, the social approach is structured around a sequential workflow, imitating human processes. Each agent has a distinct role and tasks are performed in a specific order.

In the swarm-based approach, multiple agents work on different aspects of the translation or different subsets of the data, combining their outputs for the final translation. In the hive-mind approach, each agent is fine-tuned on a specific aspect of the language, with the final translation being a combination of all outputs.

The social approach, on the other hand, leverages the adversarial nature of GANs to improve the translation process. It relies on the back-translation and evaluation steps to provide the necessary feedback for refining the translation.

Final Thoughts

The social approach to low-resource language translation provides a promising alternative to traditional methods. By imitating the roles and interactions of a human translation team, it offers a different perspective on how AI can be used for language translation tasks. As with any method, it has its strengths and challenges, but it adds another tool to the toolkit for tackling the complex problem of low-resource language translation.

Swarm-Based Approach

Drawing from the principles of collective intelligence, the swarm-based approach involves multiple AI agents working together. Each agent is trained on different aspects or subsets of the language data. Key strategies in this approach include:

• Dynamic Task Allocation: Agents dynamically choose tasks based on current system needs, similar to task distribution in a bee hive. This approach would require some mechanism for constantly tracking the state of the overall translation. If each component of the swarm is not just prompted but trained or fine tuned on a given state optimization task, then it would be possible to extrapolate from state optimization to new, unseen data.
• Distributed Training and Voting: Each agent makes independent predictions based on its training. The final translation is determined by a majority vote or another consensus mechanism. I suspect this option is likely to work about as well as democracy does, so take that for what it’s worth, but let’s not completely rule it out.
• Collaborative Learning: Agents learn from each other’s successes and failures to improve their performance. This one is probably the closest to the social approach to translation.
• Stigmergy:
  • Stigmergy is how termite mounds get built and blood cells know what do to despite knowing nothing about the broader context or big picture. Each action leaves a small trace in the environment, providing the impetus for the next action, regardless of who or what accomplishes that subsequent action. In some ways, a stigmergy approach can be likened to a chess board: you don’t need to have tracked the entire development of the game in order to know what the next right move is. It might be better to think of stigmergy as people rotating through a large set of chess games, always making the next right move on each particular game based on the state of the chess board as they find it.
  • In the context of translation, an agents’ outputs can serve as inputs for others, creating a progressive chain of translation that evolves and improves over time. Envision each AI agent (i.e., an LLM leveraging something like the ReAct approach) detecting “environmental” patterns that trigger particular action patterns, while leaving a trace for further iterations by some agent.
  • One might deploy single agents to focus on particular phenomena (e.g., proper names, or processes, or entities, or circumstances, or direct discourse, etc.) in the text, with prompts tailored with the best grammatical and translation information available so far for each respective phenomenon. Aletrnately (or alongside these bots), an agent could be deployed for translating one specific word or phrase that occurs, armed with glosses for that word and all gold-standard example translations available that contain that word. Each bot would be thus empowered to gloss a word based on contextualized examples wherever it occurs in the source texts. This overall approach seems very promising to me.

Hive-Mind Inspired Approach

The hive-mind approach is inspired by the decentralized, coordinated functioning of a bee hive. Here, each agent starts with a base model, like GPT, which is then fine-tuned on a specific task or aspect of the language. The final translation is a combination of outputs from all agents.

Final Thoughts

Swarm-based and hive-mind approaches offer potential for more accurate translations for low-resource languages. However, they require sophisticated coordination mechanisms and could be more resource-intensive than centralized models. Nonetheless, in the quest for precision in language translation, these approaches open up exciting new possibilities.

Update 2023-08-18: I just heard about Sakana.ai, which is aiming to use a swarm-based approach to rival the capabilities of the largest transformer models.