Methods for Automatic Information Extraction in Application to Typological Databases

The year 2024 calls for promising developments in the field of zero-shot machine translation, due to results of recent research allowing Large Language Models (LLMs) to process longer token sequences as prompts: more than 2 million tokens (Ding et al. 2024), potentially infinite context (Munkhdalai et al. 2024). For low-resource languages that lack not only parallel, but also monolingual data on the Internet, the only source of information is books written by linguists - descriptive grammars. The approach associated with providing whole books - grammars - as instructions in a prompt for LLM with the aim of zero-shot machine translation for low-resource languages was described in (Tanzer et al., 2023) and (Zhang et al., 2024). However, the variability and ambiguity of the terminology used in grammars, as well as the disparate nature of the information, create problems. A possible solution to these problems is the creation of a scalable pipeline for processing and systematizing grammars. This paper aims to address these issues, providing a systematic approach for processing descriptive grammars and creating a scalable pipeline for systematizing them. Apart from the pipeline based on Retrieval Augmented Generation (RAG), the paper presents two benchmarks for evaluating RAG components. The benchmarks consist of paragraphs from descriptive grammars, annotated according to their relevance to typological characteristics. Based on the results obtained on the presented benchmarks, the most optimal approach to RAG on the linguistic domain with regard to both quality and predictability appears to involve the following steps: taking a standalone typological feature and passing it to the RAG pipeline with both the reranker and the Chain-of-Thought prompt.