The importance of cognitive interviews in measurement practices

By Fernanda Gándara, Director of Research, Monitoring & Reporting for Room to Read's Girls' Education Program

Sunita Shakya, Officer, Research, Monitoring & Reporting for Room to Read Nepal

Mini Joshi, Senior Officer, Research, Monitoring & Reporting for Room to Read Nepal

A cognitive interview is a process designed to capture information about students’ processing while they complete a task, such as responding to a question. For example, a cognitive interview may ask students why they selected or provided a given answer to a question, and through that process, provide insight as to whether students are understanding and engaging with questions as intended.

In particular, cognitive interviews are widely used in the development and validation of educational assessments, as they provide evidence based on response process, one of the five sources of validity evidence [1].

In Nepal, Room to Read recently conducted a large number of cognitive interviews as part of our piloting and validation process for our Adolescent Life Skills Assessment (ALSA). Room to Read Nepal’s ALSA was redesigned, and developed from bottom to top, to properly capture the voices and realities of the girls we support in Room to Read’s Girls’ Education Program. In many ways, Nepal’s ALSA pilot and validation efforts should serve as an example for other Room to Read country offices and our partners in how to effectively enhance the contextual relevance of Room to Read’s measurement and evaluation tools.

In this blog post, I ask my colleagues Sunita Shakya and Mini Joshi from Room to Read Nepal’s Research, Monitoring & Evaluation team to share some insights about their experience implementing and using cognitive interviews with girls in Grade 6 through Grade 9. We hope that their reflections help other teams tackle similar challenges as they engage in meaningful measurement work.

[1] American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for educational and psychological testing. American Educational Research Association.

Before the Nepal 2023 pilot experience, what was your level of comfort conducting cognitive interviews?

Sunita: It was the first time that I heard about cognitive interviews. I was enthusiastic to learn about them in-depth. My level of curiosity went up when I was involved in the item construction and translation of the ALSA. I noticed that items were unique compared to other studies. I thought: “if the self-administering questionnaire was constructed in such a way, then what could be in the cognitive interview, and how that would be done?” I understood that cognitive is not only related to “knowing something” but to “knowing something from the participants’ perception” on a subject which involves lots of mental processes, and that is a very challenging area to dive into. But despite this type of activity being outside of my comfort zone, I got involved in the process. I was confident that no matter how difficult it was, I could do it, and I did not want to lose that opportunity to learn something new.

I gradually developed my confidence while discussing with the team and did not hesitate to ask even simple questions. I went to several times and searched for additional resources to better comprehend. I closely followed the team leaders’ instructions on how they phrased the questions and what could be the next leading questions to create a network of responses in a meaningful way. Furthermore, working with two different team leaders also helped me enhance my understanding and skills during the interviews.


Mini: Before the Nepal 2023 pilot experience, I was not that confident about conducting cognitive interviews. However, as soon as I started conducting them, it built up my confidence. It was my first time conducting such a task, and I must admit that I felt a mixture of nervousness and excitement. When I first reviewed the protocol (i.e., questions) and the expected responses, I was concerned that the girls might struggle to provide meaningful answers. Thanks to the support and guidance from Room to Read’s Research, Monitoring & Evaluation Team, I successfully conducted the interviews, and this experience greatly boosted my confidence. I now feel fully capable of conducting any cognitive interview on my own.

What was the most important learning that arose from conducting several cognitive interviews?

Sunita: The more I was involved, the more I gained the knowledge and confidence to conduct them. The interviews gave me insight into the multiple dimensions underlying similar issues and how different issues or problems are viewed differently depending on girls’ circumstances and their family or social backgrounds. For instance, some girls consider even a small argument violence, whereas others viewed beating or slapping as appropriate corrective behavior needed to maintain harmony in the family.

Further, rapport-building is crucial to creating a comfort zone for the girls in the discussions. I noticed that when the girls felt comfortable during the rapport building, they did not hesitate to share and discuss their ideas. However, despite trying hard during rapport building, it was difficult and took more effort to connect with the introverted girls who had never experienced such an interview before.


Mini: The most important learning that I gained from conducting multiple cognitive interviews was that getting the responses from the girls in a single interaction can be a challenging task. To gain a deeper understanding of their perspectives, it is essential to probe several times until we can uncover the full scope and nuances of their thoughts and emotions.

Was there anything that surprised you from girls’ responses?

Sunita: Yes, in unexpected ways. Some girls from Room to Read’s Girls’ Education Program were shyer than girls from outside of the program while talking about certain topics like menstruation, harassment and so on. Then, some program participants held very progressive ideas, like openly accepting that there are some people who act differently than their socially prescribed gender roles in their community. And we observed some interesting reactions among students to more explicit questions about gender and sexuality.

Most notably, it was nice to hear from girls, even younger girls in Grade 6, that they are aware of the many castes and ethnicities in their communities, and they need to live together in harmony.


Mini: As I said, when I first reviewed the questionnaire and the expected responses, I was concerned that the girls might struggle to provide meaningful answers. However, to my surprise, the girls provided very interesting responses that challenged my own notions about themselves.

For example, let’s think about questions about menstruation. In many parts of Nepal, menstruation continues to be associated with various taboos, such as refraining from entering the kitchen, participating in prayers, touching male family members and staying separate. During the cognitive interviews, some of the girls displayed confidence when discussing menstruation in ways that surprised me. They explained that they no longer adhere to these traditional beliefs, and their parents are supportive of their decision.

How essential would you say are cognitive interviews to contextually relevant measurement?

Sunita: To make the ALSA contextually relevant, cognitive interviews provided ample insights. For example, they provided information on whether the questions were easy to understand, whether the items carried double meaning, about the best item types to get the desired responses, on how to improve the items and so on. With all these critical perspectives, the items were refined in ways which I found important.


Mini: Cognitive interviews play a crucial role in the development of questionnaires. They play a significant part in ensuring that the tools are not only contextually relevant and clear but also capable of generating data that is both valid and reliable. These interviews make a substantial contribution to enhancing the accuracy, trustworthiness and authenticity of measurements, making them an essential component of precise and valuable data collection. This significance arises from the fact that the validity of measurements is compromised when respondents misinterpret items. By detecting and resolving potential problems, cognitive interviews enhance the overall quality of research and the precision of measurements across diverse fields.

What would be your advice to people who need to conduct cognitive interviews to pilot a questionnaire?

Sunita: First, it is necessary to understand the cognitive interview guidelines and purpose of the activity. The proper guidelines play a key role in maintaining consistency among the interviewers. Therefore, sufficient preliminary discussions and the development of a common understanding on these issues between interviewers is a must.

Second, cognitive interviews mean being in someone else’s shoes to feel and understand their perception on our concerns and questions. So, to make them feel comfortable at the discussion table is an essential task for the interviewer. For this to happen, sufficient rapport-building skills are required depending on the girl’s nature or response. For example, some girls may be introverted, and some may be extroverted. I found that it was easier to start discussing with the extroverted girls, but it was difficult to get openness from the introverted girls. The only solution for such a situation is to talk more informally to create a comfort zone.

Lastly, cognitive interviews require patience to keep the girls on track, especially when the girls are especially talkative. Cognitive interviewers also need to be ready to face a range of highly emotional situations, have skills for comforting and calming students, and return conversations back to the original discussion points.


Mini: To begin, it's crucial to establish a strong connection with the students by engaging them in meaningful conversation, ensuring their comfort and being friendly with them, while asking about something that is important to them but that does not feel threatening, such as their favorite activity. There may be cases during the cognitive interviews when students become emotional, so it's important to be prepared to handle such situations effectively. Initially, students may provide superficial answers, or their responses during the interview may seem random. But through thoughtful probing on given topics, they do provide insight into the underlying emotions and thoughts.

Additionally, we should determine whether the language used at school is their first language or not. During my experience conducting interviews, I encountered one girl for whom Nepali was not her first language. She exhibited hesitation and discomfort, likely because I was an outsider. This is likely to happen again, because Room to Read’s staff — myself included — works across various districts, and we meet girls from diverse linguistic backgrounds within unique community settings. It's challenging to interview girls whose first language isn't Nepali. To address this, we can hire local individuals proficient in the languages spoken by these girls. Using translation tools in their language is a good idea if several girls share the same language. However, it could be challenging if the girls speak different local languages. Therefore, it's advisable to also establish a friendly and comfortable rapport with the participants before proceeding with the interview process.

Cognitive interviews remain essential as we continue to improve our measures and to amplify the voices of girls in their development process. The insight that cognitive interviews provide is essential to understand the level and type of engagement that the girls have with tools. In the case of the ALSA, adequate engagement is not just essential for validation purposes but also to our gender transformative commitment. The ALSA is not just a measurement tool, but it creates opportunities for girls to reflect upon themselves and their lives. In addition, the ALSA items, and its results, should support future dialogues on the topics of gender and life skills. Overall, the type of experiences that we aim to create through the ALSA go far beyond ticking a box. And cognitive interviews play a critical role in our ability to achieve that.

Discover more Room to Read insights