#data

3 notes

588,000 Audio Clips and Counting

2026-07-09 · 4 min read

LangQuest just crossed 588,000 community-contributed audio clips. I checked the number twice before writing it down, partly out of habit and partly because of the dashboard incident below.

The largest single language in the set is past 230,000 clips on its own. Everything else is a long tail—dozens of languages, most with a few hundred to a few thousand clips, contributed by people recording in the language they actually speak, not a language a data vendor decided was worth paying for.

That long tail is the point. Some of the languages sitting in LangQuest right now have zero presence in any existing AI model. Not “underrepresented”—absent. There’s data in LangQuest already that isn’t even in AI models—languages that have never been done before. This isn’t just fuel for another translation feature; it’s the first entry in the ledger for languages that currently don’t have one.

Worth saying plainly: even the unlabeled audio counts. A clip nobody’s transcribed yet still tells you what a language sounds like—its phoneme inventory, its prosody, its rhythm. For languages with this little existing data, corpus value doesn’t wait on annotation. Every recording is evidence a model could eventually learn from, transcribed or not.

Now the part that taught us something. For a while, LangQuest’s internal dashboard had a “most common language” widget, and it confidently reported “English” as our top language. This was wrong on its face—English is, structurally, the least interesting language in this dataset—but it took embarrassingly long to notice, because the number next to it looked plausible enough not to question.

The cause, though, wasn’t a query bug. The widget was reading the project title, and a lot of those titles had a language name typed straight into them—“English,” over and over. That wasn’t boilerplate; it was contributors who didn’t yet know what the language dropdowns were for, so they wrote the target language into the project name instead. The dropdown that should have carried the real language often sat untouched. The widget wasn’t misreading the data so much as faithfully reporting what people had typed—and what they’d typed was a signal about our onboarding, not about the audio.

Nothing was lost, and nothing here is even fixed yet. No clips were mislabeled at the source, and you can still pull up the project-title breakdown to see exactly what the data comprises. But there’s no one-line query change that makes this go away—it’s a UI problem, and the real fix is upstream: people need to understand what those dropdowns mean before they name a project. The takeaway wasn’t “patch the widget,” it was “we need better onboarding.” Still, it’s a good reminder that a dashboard is itself a claim about your data—here, quietly, a claim about our interface—and claims deserve the same skepticism whether they come from a model or from your own UI. I trust the 588,000 number because I went and checked it against the raw table, not the widget.

More on the data-scarcity problem this is meant to chip away at soon.

Permalink →

#translation #ai

The landscape of Bible translation is rapidly evolving with AI advancements. Our role must shift accordingly. Tech giants are developing powerful translation tools, but they lack one crucial element: data for low-resource languages.

AI’s Potential in Translation

AI can now create accurate, creative Bible translations in major languages within minutes. For low-resource languages, the technology is ready - it just needs data.

Our Unique Contribution

Our mission-driven focus on low-resource languages positions us uniquely to provide the missing piece: language data.

The Real Challenge

The primary obstacle to universal Bible access is no longer translation itself, but gathering sufficient language data for AI translation into each language. Our task is to collect and publish this data in standard, open AI repositories.

A New Direction

To maximize our impact, we must:

  1. Prioritize data collection for low-resource languages
  2. Standardize data formats for AI compatibility (think JSON, not USFM)
  3. Develop user-friendly, offline-first language data collection tools
  4. Streamline our data infrastructure for efficient gathering and sharing (more data, less decisions about data)

The Path Forward

By focusing on our role as data collectors and curators, we can accelerate Bible translation in unprecedented ways. If we dedicate our efforts to data collection now, we’ll enable AI models to translate the Bible into every language faster than ever imagined.

This shift in focus would allow us to leverage our unique strengths and the power of AI, bringing us closer to our goal of making God’s Word accessible to all.

Permalink →

#translation

If you are working with Bible translation data, you know how challenging it can be to deal with different types of data sources, formats, and structures. You may have to spend a lot of time and effort to model your data in a way that makes sense for your specific use case, whether it is creating a graph, a relational database, or a document database. 😒 But what if there were a better way? What if you could focus on gathering and integrating more data, without having to make all the decisions up front about how to model it?

A simple database-like interface

That’s the idea behind a simple database-like interface that we have been exploring recently. It is based on the principle of 🔥 “more data, and less decisions about data”. ✨ The interface allows you to link together many types of wide-column datasets in various formats, such as JSON, XML, plaintext, and even audio, image, etc. Every file can be a first-class citizen 🫡, and files may make reference to other files or indexes within other files. ☝️ This way, you can create a rich and flexible data-lake that can store and access any kind of data you need.

The advantage of this approach is that you don’t have to worry about how to schematize everything in advance. You can start with the raw data you have, and then build on top of it as you go. You can create graphs or relational databases on top of this NoSQL data, using queries that can join and build complex structured implementations. And the best part is that you can reuse these queries and store them forever, so that you don’t have to repeat the same work over and over again. For example, you can create an alignment between two languages, and then include IDs or vectors that allow you to tunnel through to image data, audio Bible segments, or key paragraphs in commentary texts.

Persisting joined data

How do you keep the joined or queried data around indefinitely? You just save it in a new JSON file and plunk it into your folder along with all the original data.

While the idea of saving the joined or queried data in a new JSON file and adding it to your folder along with all the original data might seem straightforward, it’s important to consider the potential confusion this could cause in the long run. Just as saving different versions of a document with abstract names can lead to confusion about which version is the most recent or correct, the same can happen with data files.

To avoid this, one might utilize some simple metadata in a README file and mapping the origins of data. A simple solution could be to always include the script that created the data, with the script pulling its sources from a specific commit of another repository. This way, the “version” of the data can always be tracked. If a core repository is updated, all other repositories using the old commit can regenerate their data. This process could potentially be automated as a GitHub action that appends a list of added/updated files to the README (which would in theory include the creation script and associated output files).

These updates could be documented in the README, ordered by recent updates. This would provide a clear record of all new files added, optional descriptions, and timestamps for when files were created or updated, along with the commit hash.

Summary

This is not a new vision of a kind of database—it’s probably just a NoSQL database. Maybe we could call it an Agile Data Hub. 🤔 Whatever you call it, the application for Bible translation data is important, since we’re talking about many kinds of structured and unstructured data, especially in light of our ability to inexpensively extract structured data using LLMs. By using this simple database-like interface, you can speed up your delivery times and workflows, and leverage more data without making more decisions. Isn’t that what we all want?

Note: This post was inspired by recent experiences trying to create an alignment POC for Tok Pisin, and a subsequent schema-brainstorming session with Ben Scholtens. Thanks, Ben!

Permalink →