Article on Digital Repository

Establishment of a Digital Repository to Make All Ancient Indian Palm-Leaf Manuscripts and Inscriptional Literature Accessible

Bhasa Mahakavi was an ancient Indian playwright who lived in the 2nd or 3rd century BCE. Scholars say that his plays laid the groundwork for the dramas of Kalidasa, the foremost among poets. Although Bhasa was widely referred to by ancient scholarly poets, copies of his plays disappeared into obscurity for several hundred years. By divine providence, in 1910, palm-leaf manuscripts containing thirteen plays of Bhasa were discovered by ‘Mahamahopadhyaya’ T. Ganapati Sastri in a traditional library in a village near Thiruvananthapuram in Kerala.

In this manner, thousands upon thousands of ancient Indian scientific texts, literature relating to the arts, and palm-leaf manuscripts containing an immense diversity of literary works have come to light through our research over the past 150 years. Yet, several hundred thousand more manuscripts remain to be brought into print. In addition to these, we have thousands of inscriptions dating back thousands of years. Scholars and researchers continue to study, edit and interpret them. However, even in today’s digital age, none of these materials has become accessible to ordinary people who are interested in them. Addressing this shortcoming, the Manuscript and Inscriptions Digitalization Foundation (MIDF) has now undertaken a major digital project, envisioned as a vast repository that will make these materials accessible to all Indians.

There is no shortage of libraries in India. However, this organisation aspires to achieve something beyond that. It wants to build a single, universally accessible search engine hosted on local servers—a place where anyone can actually discover, read and compare India’s written history.

It is estimated that India possesses approximately one crore (10 million) palm-leaf manuscripts and more than three lakh (300,000) stone inscriptions. These were written over a period of five thousand years in more than a hundred scripts. In terms of sheer volume, this is the largest written record created by any civilisation. Nevertheless, less than one per cent of it can be read or searched.

MIDF describes our country as a civilisational billionaire caught in a data-poverty trap. This is a very apt description. When it comes to preserving old manuscripts, our first natural response is to save the deteriorating pages before they are destroyed. This very thought has driven the massive digital movement of the past decade. However, although it generated a great deal of work, it only partially addressed a very small part of the actual problem. Those organisations merely took high-resolution photographs and scans of the manuscripts and declared the work complete.

“Anyone who takes a high-quality image of a palm-leaf manuscript can promote it as a digitised copy. But we question the usefulness of that image,” says Pooja Porwal, CEO of MIDF. If it cannot be searched or made accessible, it remains merely a file on a hard drive in a locked warehouse, beyond anyone’s ability to question it. Nevertheless, high-quality scanning is an important first step.

The organisation describes the problem it is trying to solve not as a preservation problem, but as an infrastructure problem. A researcher who wants to study even a single piece of historical documentary heritage has to physically locate the archive that holds it, travel there, and hope that the material has been catalogued. But most materials have not been catalogued in this manner. Knowledge exists, yet it remains invisible. In practice, this is equivalent to its not existing at all.

The cost of this invisibility is first borne by scholars. In the absence of a central index, all research in this field takes place in separate compartments. For example, multiple handwritten copies of the same ancient text are held in repositories in different parts of the country. A scholar who wants to research this text must personally travel to each of these repositories to obtain the manuscripts, leaf through outdated catalogues, or struggle with antiquated technology.

This problem is not merely an academic issue; it concerns sovereignty. In the absence of a domestic repository, Indian scholars depend on foreign-owned databases to study their own history. Porwal points out the consequence of this: “Whenever a geopolitical issue arises, these repositories can be closed to Indian users.” Access to the history of its own civilisation would then depend on servers that the country cannot control or guarantee. A country that cannot read its own past without another country’s permission faces a serious vulnerability.

MIDF explains the urgency of this need as follows: “In an artificial intelligence (AI)-based economy, information that a machine cannot read is practically extinct.” With every year that passes without this infrastructure, the losses resulting from inaction continue to increase. This means that researchers publish their findings based on incomplete information, AI systems learn about India only from foreign archives, and India’s history ends up being written entirely according to the terms set by others.

Europe has digitised more than 200 million searchable pages. It is licensing its heritage as training data for large language models. China has increased the accuracy of optical character recognition (OCR) for traditional scripts to more than 97 per cent.

The models now being developed to organise and answer questions about human history are learning from those who have made that history machine-readable. India, which possesses the largest written record, is providing almost none of this information in a comparable form. The history of Indian medicine, astronomy, law and administration is being interpreted largely through archives built, indexed and controlled by others. While the rest of the world’s knowledge is becoming accessible to machines, India’s knowledge is not. Consequently, this gap continues to widen every year.

In contrast to all these approaches, what MIDF proposes appears highly ambitious. It is building an integrated platform called ‘Smriti’. This platform already exists as a working prototype. It has been designed not merely to store data, but specifically to make it searchable. Its features include keyword searching, cross-linking between related texts, and semi-automatic transcription that converts an image into searchable data.

This transcription tool is not ready-made, off-the-shelf software. A scholar either accepts or corrects the transcriptions. Every correction trains the model, making it more intelligent and accurate over time.

The second part of this design is missing from many digitisation initiatives. MIDF does not propose to undertake extensive fieldwork on its own. Instead, it operates as a funding and standards organisation. It issues grants to custodians and partner institutions that hold collections, on the condition that they digitise them according to a common set of specifications. These specifications form the quiet centre of the entire project.

Across India, hundreds of different digitisation methods are in use. One repository captures five metadata fields, another captures seven different field, while yet another captures twenty-three fields. As a result, the outputs are structurally incompatible. MIDF intends to address this problem through an ISO-style standards handbook that specifies comprehensive metadata fields for inscriptions and manuscripts. Its objective is to ensure that every digitisation initiative in India meets this standard, enabling nationwide interoperability.

The government has not undertaken this work because it needs to be carried out through grants and standards rather than legislation. Traditional custodians, who have protected this heritage across generations, will not relinquish ownership unconditionally. The MIDF model leaves the responsibility for preservation where it already rests, allowing custodians to retain the manuscripts and the rights to their digital copies. At the same time, MIDF provides the capital, the standards and a place where the materials can be easily discovered. In a field where institutions are generally suspicious of outsiders, often for good reasons, this approach builds trust. By treating them as partners rather than people to be exploited, the organisation can persuade collectors who are reluctant to share their archives to participate.

Following this approach, the MIDF team built a working prototype in less than a month, without raising any funds. It is already doing for India what international legacy platforms have not done. It incorporates cross-links and carries structured metadata. It has been designed to be searchable, rather than merely browsable. In just four or five months, with the support of only three grant partners, this major initiative has transferred data from approximately fourteen hard drives. This includes more than two crore (20 million) pages, 91,000 palm-leaf manuscripts, 1,000 inscriptions, and 50 lakh (5 million) delicate palm-leaf manuscripts. This achievement was accomplished with borrowed labour and just one part-time engineer. MIDF is now raising a fund of ₹15 crore to make this work permanent.

The first phase requires ₹5 crore for technology, hosting costs and grants to digitisation partners. The Foundation states that, once the platform is built, it will be hosted entirely on Indian servers and will serve as permanent infrastructure for generations, allowing anyone to use it free of charge. At a cost lower than the marketing budget of a medium-sized film, India can finally stop depending on foreign countries to preserve and access its own history. (Those who wish to support this initiative may make a donation.)

Many highly important texts currently lie in unlisted piles, deteriorating without any index to establish their existence and without any search facility through which they can ever be found. This Foundation is making efforts to bring these materials within the reach of Indians for examination and study.

                                                                                                                                                                                      — Sri Koundinya