AI4Bharat's Ten Trillion Token Project: Powering AI for Indian Languages

AI4Bharat's Ambitious Data Collection Initiative

AI4Bharat, an IIT Madras-incubated artificial intelligence lab, has embarked on a groundbreaking project to collect ten trillion tokens of language data from across India. This massive undertaking aims to power the next generation of AI services tailored for Indian languages 1

1

. The initiative, known as the "Ten Trillion Token" project, seeks to address the unique challenges posed by India's linguistic diversity and create AI models that can effectively understand and process Indian languages.

Comprehensive Data Collection Across India

Over the past three years, AI4Bharat has conducted an extensive data collection campaign, covering almost every district in the country and encompassing all 22 official languages of India. The collected data includes:

• 200 million spoken words
• Voice samples from diverse demographics and professions
• Data from everyday conversations to technical documents

Mitesh Khapra, co-founder of AI4Bharat, emphasized the importance of this diverse dataset: "We have ensured that we collect voice samples split across several demographics, across different professions, blue collar and white collar" 1

1

.

Applications and Use Cases

The collected data is expected to have wide-ranging applications, including:

• Supporting farmers
• Assisting children
• Facilitating digital payments
• Aiding in agriculture

These use cases demonstrate the potential impact of language-aware AI on various sectors of Indian society 2

2

.

Open-Source Approach and Collaboration

AI4Bharat has adopted an open-source approach to accelerate the development and adoption of language technologies. Khapra stated, "Our data, models and scripts are open sourced. You can build on top of that" 1

1

. This approach has enabled various stakeholders, including startups, academic institutions, and deep tech companies, to utilize the collected data for building their own models.

Addressing the Language Gap in AI

The Ten Trillion Token project aims to address a critical gap in current AI technologies. While English-language data is abundant on the internet, making it easy to train AI models, the same is not true for Indian languages. Each of India's 22 major languages has its own script, grammar rules, and cultural context, presenting unique challenges for AI development 1

1

.

Future Implications

The successful completion of this project could have far-reaching implications for AI accessibility in India. It could enable:

• More natural and effective human-AI interactions in local languages
• Improved government services and form-filling assistance
• Enhanced agricultural advisory systems for farmers

By building native Indic models that support Indian languages "not as an afterthought," AI4Bharat aims to create AI systems that truly work for India's diverse population 2

2

.