#machine-learning

The Established Approach: Understanding and Expert Validation

The established approach has been to use an abstract notion of “understanding” as that bridge.

Under this paradigm, we start with Language 1, internalize and comprehend it to arrive at an “understanding”, and then express that understanding in Language 2. However, validating this approach is challenging, because the interim step of “understanding” does not exist as concrete, discrete units. As a consequence, the approach is not easily scalable, and the internalization process is protracted in an attempt to assure the validity of the “understanding”.

Validity, in turn, is assessed by experts in the target language, the subject matter, and by authorities and gatekeepers from within or without the respective communities.

The New Approach: Multimodal Assets and Consensus Validation

An alternative approach that is easier to validate and scale leverages multimodal assets like images, audio, text, animations, videos, and simulations to serve as the bridge between languages.

With this method, Language 1 is represented as a set of multimodal assets. Then Language 2 is generated by interpreting and describing those assets. Validation becomes straightforward, because the mapping between each multimodal asset and its linguistic representation in Language 2 is distinct and unambiguous.

The validation process is simple and proven - present a user with a snippet of Language 2, and ask them to match it to the correct multimodal asset, similar to how a Captcha test works. This allows the approach to be easily scaled without extensive software training.

Once we have established a robust set of validated asset-language pairings for the new language, we can strategically select a subset of the corpus that will provide sufficient data to train a small language model or fine-tune a large language model. As Zhou (2023) demonstrated, a model can be trained to translate the Bible with 90% accuracy using only 3% of the Bible as training data. This suggests that eliciting translations for a carefully chosen subset of a corpus via multimodal assets and Captcha-like validation is feasible.

The key advantage of this approach is that it does not require any prior knowledge of the target language, making it massively scalable for translating low-resource languages that have been historically difficult to support. By focusing data collection efforts on the most information-dense portions of a corpus, we can efficiently gather the necessary training data to enable high-quality machine translation for the long tail of languages.

In summary, using multimodal representations as a bridge between languages offers a promising alternative to the “understanding” based approach that is easier to robustly validate and scale. Combined with strategic subsampling of corpora for data collection, this paradigm shift could dramatically expand our ability to develop machine translation capabilities for low-resource languages.

Reference:

Zhou, Z. (2023). [TBA].

Opening up the black box

One of the biggest challenges with AI-based translation is that it works like a black box. You feed in some data, and you get some data out. But you don’t really know what’s going on inside the black box. You don’t know what the model is learning, or how it’s learning it. You don’t know what features it’s learning, or how it’s combining them to produce the output. This should rightly make translators somewhat nervous. How can you trust the output of a model that you don’t understand?

My vision for translation assistance with AI is thus to pry open the black box by using LLM-based predictions, rather than simple sequence-to-sequence predictions. This is a fundamentally different approach, and it relies on the concept of in-context learning, rather than a fine-tuning or transfer learning approach.

In-context learning

In short, in-context learning involves providing “training data” to the model on the fly, in the prompt. (Read more from Stanford AI Lab).

This is a fundamental difference that is really critical for translation assistance. It means we don’t need to take a bunch of data (e.g., a complete New Testament translation) in order to fine-tune a model for generating more data (e.g., a complete Old Testament translation). Instead, we can take a small amount of data (e.g., a few verses of the New Testament) and use it to generate more data (e.g., the next few verses of the New Testament, or some similar verses being drafted in the Old Testament).

We get two big benefits from this approach:

1. Bootstrapping: We can use a small amount of data to generate a large amount of data. This is a huge benefit for low-resource languages, since we can use the data we have (which is not enough for fine-tuning a model) to generate more data, and then use that new data (again, without re-fine-tuning our model) to generate even more data, and so on. This is a virtuous cycle that can help us bootstrap our way to a complete translation.
2. Instant feedback: We get instant updates, allowing the real translator (a human), to provide feedback to the model on the fly.

What do we need to make in-context learning work throughout the translation process?

To leverage in-context learning, we need to be able to draft new translations using few-shot examples (e.g., use a few already-translated verses to draft a new verse). We also need to be able to evaluate those translations using few-shot examples (e.g., use a few already-translated verses to evaluate a new verse). Optionally, we might want or need a back-translation in order to enable third party evaluation (such as a translation consultant).

Working backwards, then:

• To evaluate, I need
  • a metric
  • [optional] a back-translation
• To back-translate, I need
  • a translation
  • a non-hallucinating technique
    • translate back to Greek?
    • break down by punctuation?
    • Constrain using valid_tokens (see Token matching evaluation)
• To translate, I need
  • a chunk type (a unit of translation)
    • could be a Bible verse
    • could be a larger semantic unit that looks like a phrase
    • I would not recommend using a single word, since a word is a structural unit, and structures are precisely what gets left behind in translation
  • a source text
  • few-shot examples
  • [optional] supporting data
    • glossary
    • notes
    • evaluation feedback on prior drafts
  • [optional] constrain with valid_tokens
  • [optional] a concise, prosaic description of the most relevant source language data
  • [optional] a concise description of the language-typological differences between the source and target languages

Conclusion

Translation assistance can leverage LLMs and in-context learning to open up the black box of AI-based translation. This approach can help us bootstrap our way to a complete translation in a manner that cannot be achieved using traditional machine translation and typical AI-based approaches such as sequence-based model fine-tuning, and it can accept instant feedback from the translator, which means that every single correction or improvement made by the translator gets leveraged in the very next verse or chunk being drafted.

Note: you can also use an approach like this for a multi-agent simulation. See also the Social Approach to Low-Resource Language Translation.

1. Unraveling Context with Attention Mechanisms

The process begins by transforming the text data into global vectors, and then into contextual vectors using attention mechanisms. These mechanisms, a cornerstone of modern transformer models, allow us to capture both the inherent meaning of a word and the rich context surrounding it in the form of contextual embeddings.

What I have tried so far is the following:

• Start with global word2vec or fastText vectors
• For each unique lexeme in corpus
  • Gather all sentences this lexeme occurs in
  • For each gathered sentence
    • Calculate the cosign similarity between the target lexeme and every other lexeme in the sentence
    • Weight each similarity score by the distance from the target lexeme (further away = less similar; this is obviously a step that is amenable to multiple possible configurations)
  • Store all of the resulting contextual vectors (or “attention” vectors) in a dictionary with the lexeme as the key

2. Identifying Contextual Patterns with Clustering

The next step involves clustering these contextual vectors. Using unsupervised machine learning methods such as K-means, DBSCAN, or hierarchical clustering, I group together similar vectors. These clusters form centroids that represent typical patterns of contextual variation, rather than separate, isolated word senses.

What I have tried so far:

• For each lexeme
  • Cluster all associated contextual vectors using some clustering technique
  • Identify centroid vectors within each cluster
  • Allow the centroid vectors to represent the ‘word senses’

3. Strengthening Clustering with Ensemble Techniques

To enhance the reliability of the clusters, I implement ensemble clustering techniques. By merging the results of multiple clustering algorithms, one could create a consensus on the cluster assignments. This approach capitalizes on the strengths of each algorithm, providing a more resilient understanding of contextual modulation.

In reality, I haven’t yet worked through any means of evaluating the resulting clusters. The trickiest stage in any clustering process (in my experience) is determining how many clusters you need. This is always a lossy process, so there seem to be drawbacks no matter how you slice up the semantic space.

4. Advanced Techniques: Neighbour-Aware and Consensus Clustering

I further refine the clustering process with two more techniques: neighbour-aware and consensus clustering. Neighbour-aware clustering considers the nearest neighbours of each vector, providing a better representation of local contextual variation. In consensus clustering, I amalgamate the outcomes from several neighbour-aware clustering methods to determine the most accurate cluster assignments.

Conclusion

Through the synergistic combination of attention mechanisms and multiple clustering techniques, I am developing a method to represent lexical meanings in a monosemic manner that transparently accounts for the role of contextual modulation in arriving at ‘typical’ word senses.

I’ve always been prone to challenge the common assumption of multiple word senses being inherent features of lexemes (rather than emergent features of larger contexts). This work in progress offers one attempt at representing and analyzing text data to this end. This opens up new possibilities for text understanding and semantic analysis in NLP, paving the way for more accurate and contextually-aware models of lexical semantics that recognize decontextualized lexemes as meaning potentials, and contextualized lexemes as specifications of those potentials.