Exploring an LLM (ChatGPT 4) for Estimating Democracy: Case of African States

Over the past decade, scholars in social sciences have increasingly leveraged the vast advancements in big data and machine learning, significantly expanding their research capabilities and enabling them to address previously intractable hypotheses empirically (Cioffi-Revilla et al., 2010; Mason et al., 2013). The introduction of Large Language Models (LLMs) and Generative AIs is poised to further enhance these capabilities, potentially revolutionising tasks like text annotation, sentiment analysis, and topic detection, where these models are already making significant impacts (Ziems et al., 2024; Rathje et al., 2023; Hegselmann et al., 2022). Building on this momentum, Wu and colleagues (2023) have explored the potential of LLMs in analysing complex socio-political constructs. They developed a method that addresses the limitations of current LLMs and strives to ensure the reliability and validity of the generated estimates. Specifically, their study utilised GPT-3 to conduct pairwise comparisons of U.S. senators regarding their ideological stances, support for gun control, and abortion rights to derive relative scores for these constructs. This methodology proved effective, with GPT-3 generating reliable and valid estimates that correlated well with existing measures and revealed new dimensions of the American political landscape previously unexplored by traditional metrics. Our study seeks to extend Wu and colleagues' innovative approach to a different context by applying it to assess authoritarian-democracy scores and opposition silencing tendencies of African countries and presidents. By employing GPT-4 for this analysis, the study aims to explore the applicability of this methodology across different settings and constructs. The dual focus on democracy and opposition silencing through LaMP scores not only enriches the academic dialogue on these critical issues but also challenges the limits of what LLMs can achieve in social science research. This exploration may provide fresh insights into political dynamics and governance in African nations, testing the boundaries and potential of LLMs in global social science applications.