'You Find Yourself Thinking about Your Research 24/7'

Daria Gracheva, expert at the Institute of Education, turned to science in pursuit of methods to discern between a poorly constructed test and a well-designed one. In this interview with the HSE Young Scientists project, she discusses the '4K test,' which is intended to measure critical thinking and communication skills, shares insights on the model named after Georg Rasch, and explains how setting up an aquarium for a crab can help assess soft skills.

Why I Decided to Pursue Science

I have always had a thirst for knowledge. However, before enrolling in the master's programme at HSE University, I lacked a clear understanding of a career in science and couldn't envision myself pursuing it. But since studying at the Institute of Education, where one classroom in two doubles up as a research centre or laboratory, it was impossible for me not to be swept up by this atmosphere.

Once I realised that I truly enjoyed what was going on there, I took on the role of a research assistant at the Centre for Psychometrics and Measurement in Education. The example of my colleagues who combine teaching with research has been empowering. They initially inspired me to pursue a career in science and subsequently motivated me to enrol in the doctoral programme at the Institute of Education.

Before my master's at HSE University, I studied Mathematical Methods in Economics at Plekhanov Russian University of Economics. While I found statistics and mathematical modelling captivating, economics did not interest me as much. Therefore, I sought to apply my expertise elsewhere and opted for a field at the intersection of education and psychology. Currently, I construct models, analyse educational and psychological data, and draw conclusions from the findings.

The Subject of My Research

Researchers in the fields of education and particularly psychology often want to measure and quantify various phenomena, be it educational outcomes, a child's self-esteem, or intelligence. To accomplish this, one requires an appropriate measuring instrument. I enrolled in the master's programme focusing on measurements because I was interested in the process of designing such instruments and wanted to know how to distinguish a poorly conceived test from a reliable one. Designing tests and analysing their results requires a specific methodology. This falls within the realm of psychometrics, the science of measurement that seeks to establish a connection between observable behaviour (like test responses) and something unobservable or latent, such as intelligence or digital literacy. Personally, I am involved in measuring what are commonly referred to as 21st-century skills, which include critical thinking, creativity, communication, and collaboration. It is often believed that cultivating these soft skills can significantly contribute to a child's future success in life.

My work has two main aspects. First, we must identify a suitable instrument for assessing complex skills like critical thinking or creativity. Traditional multiple-choice tests are not well-suited for this purpose. What's required are interactive test systems known as scenario- or story-based tasks, where you construct a testing environment, place a respondent within it, and record behavioural cues that indicate their level of creativity, ability to collaborate, and other relevant skills. My team and I were tasked with designing such a test system, employing an evidence-centred design approach and drawing upon existing theoretical models of the relevant skills.

The second aspect of my work involves modelling the data reflecting the respondent's behaviour within the test environment. We use statistical models to evaluate the respondent's abilities with as much accuracy as possible and then to provide meaningful feedback.

How the System We Have Designed Is Applied

It goes by the name of the '4K test' and resembles a computer game. Respondents access the online test via a link, where a story unfolds in either a real or fantastical setting. They can click on elements and complete diverse tasks, showcasing their skills in critical analysis, creative problem-solving, and more.

For example, our approach to critical thinking assessment involves a simulation of an internet browser, where the respondent needs to find useful information for solving a problem, but first they must select the most reliable source of such information. Indeed, a hallmark of a critical thinker is the ability to discern a reliable source of information from one that is less trustworthy.

Currently, we are in the process of refining our '4K test' to transform it into a product that we can offer to schools.

Example of an Assignment

We have devised a test assignment tailored for elementary school students, requiring the test taker to contemplate the appropriate course of action if they opt to keep a crab as a pet. First, a pet crab needs a home, such as an aquarium.

We provide the child with an electronic text outlining the necessary items for setting up the aquarium. The child reads the text and highlights the key points. Following that, we present them with a computer simulation of an aquarium on the screen, allowing them to use a mouse to drag and drop items like substrate, a shelter, driftwood, and plants into the aquarium. Additionally, there are other items on the screen that haven't been mentioned in the text or have explicitly been indicated as items that cannot be placed in the aquarium, such as plastic. We then observe as the child constructs their own aquarium based on the information they have analysed.

In collaboration with colleagues, we have published a paper in an international peer-reviewed journal in which we describe our approach to designing this assignment and provide evidence of its psychometric validity.

My Dream

My dream is to engage in an international project. Measuring 21st century skills is a global challenge. Many of the papers we reference have been authored by researchers from other countries. Our work aligns with the international agenda, and naturally, we would very much like to have our papers translated into English to facilitate international comparative studies.

If I Hadn't Become a Scientist

I would still be working with data. I enjoy applying my skills in a research setting, yet could also venture into business and apply them to product development and marketing tasks.

Scientists I Would Like to Meet

In psychometrics, we frequently use statistical models based on modern testing theory. These models depict the probability of a correct response to a question as contingent on both the respondent's ability and the task's characteristics, such as its level of difficulty. This formulation of the theory was introduced by Georg Rasch. I would be curious to ask him how he came up with this idea and whether he anticipated that the Rasch model would emerge as the gold standard in this field.

A Typical Day for Me

I'm a morning person and I get up early. This may not be the best for my health, but I often kickstart my day diving enthusiastically into work. Later, I have breakfast and then resume my work.

It appears to me that the life of a researcher is structured in such a way that you find yourself thinking about your research 24/7. If you're not occupied with routine tasks, you're probably either finalising an article or formulating a new hypothesis. I cannot assert that I have a clear-cut work-life balance, but as long as I have the energy, I am content with my current lifestyle.

How I Deal with Burnout

Scientists at research institutions are often susceptible to burnout because we invest a part of our soul into our work; we genuinely care about both the outcome and the process.

How do I handle such situations? When I reach a critical point, I remind myself that 'enough is enough' and take a day or two off so I can rest and do nothing at all. Such an abrupt shift towards complete rest usually helps. It might not necessarily be an official day off, but I trust my supervisors will forgive me. If I suddenly feel overwhelmed in the middle of the week, I try to distribute my workload across other days as much as possible and take some time to relax. Since researchers typically have irregular schedules, I can always finish my article on a Saturday. The key here is to recognise the moment when you sense your energy waning and motivation dwindling, as motivation plays such a crucial role in our work.

My Interests Besides Science

At present, my primary hobby is playing board games. Since I spend a significant amount of time in front of the screen for my work, I appreciate that board games don't require a computer. Yet at the same time, board games keep your brain active, they are intellectually stimulating, enjoyable, tactile, and don't strain your eyes. We have an extensive collection of board games at home and continually add to it. Ticket to Ride is my favourite at the moment.

What I Have Been Watching Lately

Barbie, and it did not disappoint. This is a film that blends humour with social commentary. At first glance, it may appear to be a children's story about dolls, but the film addresses significant themes related to women's rights, gender equality and opportunities, which are also relevant to us in science.

What I Enjoy Reading

I'm particularly drawn to a compelling plot, and I enjoy detective stories and thrillers, such as those by Agatha Christie or Stephen King. I also went through a phase of fascination with Dan Brown's books, particularly The Da Vinci Code and Angels and Demons.

My Favourite Place in Moscow

Chistye Prudy, where our Institute of Education is located. In spring or autumn, it is lovely to step out of the Institute and take a stroll along Chistoprudny Boulevard. As you walk to the Metro, you become keenly aware that you're not just working at a research institute, but at one located in the heart of Moscow.

Advice for Aspiring Scientists

Do not give up. An academic career is not an easy journey, especially considering that novice scientists often grapple with doubts about their work. The crucial thing is not to face your doubts alone. If you are uncertain about the relevance and usefulness of your research, discuss it with your supervisor and colleagues. I can assure you that you will receive tremendous support, which you will find incredibly inspiring.