Skip to main content

Employing cognitive interviewing to evaluate, improve and validate items for measuring the health-related quality of life of women diagnosed with ovarian cancer



Use of patient-reported outcome measures in clinical settings facilitate the delivery of better health care to improve patient health outcomes. Previously collected qualitative data indicated themes that could inform items for a health-related quality of life measure. This study investigated the content validity of items for inclusion in a new health-related quality of life measure suitable for patients with ovarian cancer.


Cognitive interviewing techniques were used with fourteen women diagnosed with ovarian cancer and at different times since diagnosis, to evaluate items derived from the previously collected qualitative dataset. A set of draft items was administered via telephone, Zoom and WhatsApp app together with questions on item meaning and wording. Interviews were transcribed and thematically analysed.


Four broad themes emerged in relation to the questionnaire construction and comprehension of items: intent and clarity, wording, relevance and context, and overall questionnaire construct. All draft items were adjusted based on the interview findings. A final set of 38 health-related quality of life items comprised 7 items describing physical health and functioning, 21 describing emotional wellbeing and 10 items describing social wellbeing; each rated on a five-point frequency response scale.


The items reflected a range of personal experiences associated with the patient clinical journey, creating a health-related quality of life tool specific to women diagnosed with ovarian cancer. The cognitive interviewing process established content validity for the tool, thereby, preparing it for field testing and evaluation of its psychometric properties. This study highlighted the fundamental role of cognitive interviewing during health-related quality of life questionnaire development to ensure that item content is grounded in patient feelings, functioning and meaning.

Peer Review reports


The goal of this study was to examine the meaning and coherence of items which formed a health-related quality of life questionnaire for patients with ovarian cancer. Ovarian cancer (OC) is the most lethal gynaecological malignancy. Globally, 230 000 women are diagnosed with OC and 150 000 die of the disease each year [1]. In Australia, OC is the eighth most commonly diagnosed cancer among women with approximately 1500 new cases diagnosed every year and a five-year relative survival of only 46% [2]. OC survivors remain at high risk of relapse and fear of cancer recurrence which may lead to significant anxiety and psychological morbidity [3]. Many OC survivors report moderate to severe symptoms such as peripheral neuropathy and fatigue two years after completing treatment, and may experience a disease recurrence, or developing a new primary cancer, all of which can influence health-related quality of life (HRQOL) [4, 5].

Patient reported outcomes are defined as “any report of the status of a patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else” [6, 7]. Patient reported outcomes are collected using tools and/or instruments called patient reported outcome measures (PROMs) [8]. Incorporating PROMs into clinical settings is believed to enhance the delivery of health care [9] and achievement of patient health outcomes [10].

Clinical trials involving cancer patients often utilize general PROMs, (for example, the Short Form Survey 36 (SF-36)) and validated tools for specific cancer types, including gynaecologic cancers [11]. Whilst commonly used in research, PROMs are less often used in routine clinical practice [9], and important information on functional recovery and HRQOL could be missed.

The European Organization for Research and Treatment of Cancer Quality of Life Questionnaire for Cancer Patients (EORTC QLQ-C30) and Functional Assessment of Chronic Illness Therapy Functional Assessment of Cancer Therapy-General (FACIT FACT-G) are the two most widely used cancer-specific HRQOL questionnaires and can be augmented by site- and/or treatment-specific modules [11]. Both EORTC and FACIT have OC-specific modules including EORTC QLQ-OV28, FACT-O and FACT-ovarian cancer symptom index (FOSI) [12]. The EORTC QLQ-C30 questionnaire contains 30 items which assess functioning, global HRQOL, and cancer-related symptoms. It is complemented by a OC module, QLQ-OV28, which contains a further 28 items including body image, abdominal/gastrointestinal symptoms, hormonal/menopausal symptoms [13]. The FACT-O questionnaire comprises 27 items from FACIT FACT-G to cover four core domains of wellbeing with an additional 12 items specific to OC. The FOSI is a shorter, more focused subset of the FACT-O items that includes three subscales: disease/treatment-related symptoms and general function/wellbeing [14]. More recently, King et al. developed a tool, the Measure of Ovarian Symptoms and Treatment concerns (MOST), to assess patient-reported symptom burden as an end point in clinical trials. This tool consists of 24 items including abdominal/disease/treatment-related symptoms, psychological symptoms and MOST-Well-being [15].

Whilst each of these tools seek to describe experiences of women with OC, each were developed using different methodological approaches and this could explain variations in their content [16]. For example, the FACT-O was developed to assess symptoms and quality of life and semi-structured interviews were conducted with five health professionals (nurses) and 17 OC patients with varying disease severity. Items were then reviewed by a panel of experts [17]. Additional to symptoms, the items reflect psychosocial aspects such as “appearance of my body,” “able to feel like a woman” [17]. Similarly, to the FACT-O, the QLQ-OV28 focused on measuring symptoms, both general and disease specific symptoms. Unlike the MOST which primarily focused on symptom benefit in women with symptomatic OC, neither QLQ-OV28 nor the FACT-O were specifically developed and validated in patients with platinum resistant recurrent OC, where the aim of treatment is symptom benefit and palliation. Similarly, differences exist in the level and stage of patient involvement in the development of PROMs [13, 14]. The importance of this is seen in the work of Kirwan [18, 19] and illustrated by Friedlander who showed clear differences between the level of importance of symptoms reported by symptom benefit in women with symptomatic OC.

We previously conducted semi-structed interviews and focus groups with women with OC and identified key experiences and priorities for women diagnosed with OC. Findings included challenges related to diagnosis and treatment, adjustments in their relationships with family and/or friends, financial issues, some difficulties in their relationships with health professionals and comment on useful coping strategies. Through the use of template thematic analysis, these findings were further developed into a set of items that could be useful for the development of a values-based OC PROM (Additional file 1) [20].

Establishing content validity is a fundamental first step in determining whether an outcome measure is fit for purpose [21]. Using cognitive interviewing (CI), the purpose of this study was to refine, and content validate the items/statements derived from the qualitative data collected in our previous study. We also aimed to examine whether additional items from the dataset could contribute to a broader questionnaire on factors related to HRQOL (Additional file 2).


This study employed cognitive interviewing (CI) with the integration of concurrent think-aloud (CTA) procedure. CI is a method whereby items and contents and response processes can be assessed and validated [22], ensuring the content clarity and relevance of the items and response categories [23, 24]. The CTA procedure is a research method in which respondents speak aloud about their thoughts as they complete each questionnaire item in regard to personal understanding of the items [25, 26]. Thus, this interview procedure was implemented to identify items where respondent interpretation and the developer’s intentions were dissimilar and to identify ways in which those items can be modified based on the responses given [23, 25]. The questionnaire and interview guide used in the study was developed for this study and is provided as Additional file 2 and in Table 1. This study was granted ethics approval by the Human Research Ethics Committee at University of Notre Dame Australia (UNDA) (2020-010F) and our study complied with the Declaration of Helsinki.

Table 1 Probing questions used during the CTA procedure

Recruitment procedures and study population

Our previous study incorporated input and guidance from relevant community support organisations, primarily Ovarian Cancer Australia (OCA) and Cancer Council Western Australia (CCWA). This study also worked in partnership with CCWA, OCA and the Australia New Zealand Gynaecological Oncology Group-Survivors Teaching Students, in recruiting participants through advertisements distributed through the media and relevant agencies. The recruitment process was slow as it coincided with the COVID-19 pandemic. Additional participants were recruited through King Edward Memorial Hospital and Solaris Cancer Care, a patient support organisation, in Perth, Western Australia. All participants who expressed interest in participating in the study were contacted directly by the UNDA researcher (SB) to schedule an interview at a mutually convenient time.

Purposive sampling technique which utilized a maximum variation strategy, aiming to represent variation in the stage of OC, treatment received, demographic and socioeconomic characteristics was employed. This sampling technique provides insights and in-depth knowledge regarding individuals’ experience in different circumstances [27]. Participants were women diagnosed with OC, older than 18 years and were proficient in English.

Data collection

Participants were provided with all relevant documents including a consent form, participant information sheet and interview schedule (provided as Additional file 3), and signed consent was received prior to interview commencement. Due to the COVID-19 pandemic, interviews were conducted using telephone or video (Zoom/Skype/WhatsApp App) according to the participant’s preference. Identified statements from our previous qualitative study were tested by conducting CI using the CTA procedure (Table 1). The focus of applying CTA was not limited to determining certain words or response categories of the items but also to identify how and why respondents answered each item. Prior to the interview, participants were briefed that they would be asked to share their thoughts and opinions on the statements that were unclear or difficult to comprehend [25]. Participants were encouraged to engage with the researcher by reacting to pieces of the document and explain if they found the items and/or content confusing or unfamiliar.

Data analysis

Collected data was audio-recorded and transcribed verbatim and was imported into QSR NVivo version 12 for data management and analysis [28]. In the last five interviews conducted, no new codes or themes were generated and thus saturation of data was achieved. The CI data were analysed in a multistep process. The initial analysis comprised open coding, with the intention of establishing codes based on participant feedback and suggestions. Thereafter, a second analysis involved axial coding to identify patterns in the codes for each item based on participant feedback and to categorise it under broader themes. A template thematic analysis technique was utilised to extract themes from the date collected via the CTA procedure. Template thematic analysis uses a priori code frames to analyse and report on the data. A priori codes are usually taken from interview questions as a way of initially organising the data. Template analysis commonly uses the questions as a skeleton code frame for thematic analysis [29]. Items generated from all interviews (including the initial patient interviews) and the literature relating to coping with cancer were collated into the interview statements. Additional items were also generated based on emerging themes identified during the CI data analysis. The identified themes were then refined and operationalised into statements corresponding to each theme.

Investigative team review

During multiple sittings, the supervisory team from clinical and research backgrounds, reviewed the modified statements to establish a final set of items. A qualitative researcher who had collaborated closely with community groups, a health researcher with research expertise in qualitative study designs who had expertise in developing quality of life measures, a health researcher whose expertise lies in the areas of health service redesign and translation, a gynaecologist, and a higher degree research student. Team meetings formed a fundamental part of the questionnaire development phase. In compliance with all collective feedback and suggestions, supervisory team meetings were held to determine necessary changes and achieve a consensus on the modifications.



Fourteen participants took part in individual telephone or video interviews with a mean duration of one hour and fifteen minutes (range of 30 min to one hour 50 min). Interviews for one participant were completed over two occasions for their convenience, and the initial interview which provided pilot data was also incorporated into the dataset. Feedback collected from the pilot interview informed modification of the statements’ response scale to be used in the subsequent interviews. Participants varied in age, employment and marital status but all lived in a metropolitan setting. Seven participants had been diagnosed over five years previously and two participants had received their diagnosis within a year of the interview. Among the 14 interviewees, five participants were undergoing active treatment because of recurrence of their cancer (Table 2).

Table 2 Description of study participants (n = 14)

Draft questionnaire examined using CI and CTA

Draft items were classified into the domains identified in our previous qualitative study. Diagnosis and Treatment-related (chemotherapy, surgery and complementary therapies), relationships with family/friends, financial aspects, health services and interactions with health professionals, and coping strategies. Challenges related to diagnosis and treatments were documented. Key themes such as physical wellbeing, emotional wellbeing, relationships with family/friends, health services and interactions with health professionals, and coping strategies were identified as related to living with an OC diagnosis across the clinical journey. A 5-point Likert response scale was created for the items.

The draft questionnaire acknowledged all data from the original qualitative study and included a set of questions to collect participant demographic and cancer history information. The next four sections focused on challenges related to diagnosis and treatment and included skip questions which directed participants to appropriate sections based on the responses given, with the response scale measuring the severity of challenges. The remaining sections with items related to HRQOL, satisfaction with services and coping strategies were provided with a response scale of frequency (Table 3).

Table 3 Total number of items per questionnaire section prior to modification based on CI data

Refinement of HRQOL items

A multistep analysis process was then used to classify how the items could be modified, using four broad themes. Refinement of HRQOL items is illustrated with examples below.

Item intent and clarity

Responses for several items were inconsistent with the item intent, failing to interpret the researcher’s objectives. Specifically, for some items, participants reported, “I do not understand what this means” or “I couldn’t get to the meaning of the question”. For example, one original item was “I felt frustrated during and/or after receiving a treatment”. A majority of the participants comprehended “frustrated” in terms of the treatment received, but the researcher’s intent was to measure “frustrated” in relation to a participant’s activities whilst undergoing treatment. With participant input, the item was modified to “I felt frustrated that I could not take part in usual activities during and/or after treatments”.

Item wording

Difficulties with comprehension of the meaning of some items was identified. For instance, the statement “I feel valued because I can still contribute to the workforce” created confusion in the participant. One particular response was “I am not working, but I still feel valued. I would put not applicable for that one. But I would put strongly disagree because I feel valued because of work”. To better structure it, the item was modified to “I have felt valued because of the work that I can do (home, workforce)”.

Other items were considered vague by participants and difficulties arose in communicating the researcher’s intent of the items to the participants. For example, the item “My family/friends have reacted unexpectedly to my illness” could have been interpreted as a positive or negative experience. With participant feedback, the item was modified to “My family/friends have reacted unexpectedly (in a negative way) to my illness”.

Item relevance and context

Some items had little relevance to the participants’ age. These items included: “I have been embarrassed by the way my body has changed” and “I have felt less feminine because of my illness”. Moreover, several items specific to emotional wellbeing were not relevant to participants who were under surveillance and not receiving treatment. For example, when asked about “I have felt sick and unwell due to the side-effects of treatments I have experienced”, participants replied saying “that’s hard because during the chemo, I felt unwell and sick but since the chemo, I have had no problem”.

Finally, the complexities of the wording some items made it difficult for participants to respond precisely. For example, one original item was “My family/friends are generally supportive of me at this time”. However, based on the participant’s perspective, the support received from family and friends could have been different. Thus, the item was separated into two items. In particular, one participant couldn’t comprehend the context of the item within the social domain. Initially, the statement was developed as “I have felt isolated socially because of my illness. The participant could not, however, answer in what context the word "socially" meant. The item was subsequently changed to “I have found it difficult to connect socially with people because of my illness (e.g. at work, in public)”.

Refinement of items describing contextual factors for HRQOL

The qualitative dataset contained additional themes of disease/treatment and financial issues, communication with health professionals and coping strategies. These themes were developed into three sets of questions that reflected contextual factors for HRQOL: patient symptoms, satisfaction with health services and strategies for self-sufficiency and resilience. These items were refined using the same multistep analysis process and examples are presented below.

Item wording

Some items related to technical terms that were complex to understand. For example, concerns were raised on specific terminologies such as “cancer recurrence”, “mucositis”, “full cycle of chemotherapy”, “complementary therapies”. In response, clear definitions were constructed (Additional file 3).

Item relevance and context

Relevance of specific items was age dependent. For instance, questions regarding “sudden onset of menopause” and “inability to have children” had no impact on the wellbeing of many participants because they had already undergone menopause prior to receiving their OC diagnosis. Thus, a “not applicable” response column was included to the patient symptom section. Since there were items of little relevance across the questionnaire and upon joint agreement amongst the supervisory team, a timeline was provided for every section.

The context of the item “There is a lack of financial assistance with practical support” was unclear as one participant expressed, “I haven’t had to seek that out. So, I don’t really know. I haven’t needed it but to listening other ladies I think there probably is a need for it”. However, the researcher’s intent was to know whether the participants had any challenges accessing the services or not. The item was modified as “There has been a lack of practical support offered to me”. In addition, the item “I maintained a sense of gratitude” seemed out of context to a participant. The item was modified to “I have maintained a sense of gratitude for what I am able to do/achieve”.

Questionnaire construct

Some participants mentioned that the response format type for HRQOL and wellbeing items could be improved as responses could have reflected upon frequency, rather than merely agreeing/disagreeing to a statement. For instance, participants responded by using expressions such as “at times” or “sometimes” and found it difficult to merely agree/disagree to the items. Thus, a frequency format of Always/Often/Sometimes/Rarely/Never was applied to sections that measured aspects of HRQOL and wellbeing.

In summary, the CI and CTA processes informed substantial item reconstruction to achieve acceptable content validity for this participant group. The questionnaire in its entirety was restructured: including 67 items which were modified, 10 items were condensed and merged into appropriate sections, 66 items were deleted, and 15 items were added. The items describing HRQOL were grouped, forming an HRQOL instrument named the OVArian cancer health related Quality Of Life (OVAQOL) scale. This scale comprises items that contributed conceptually to three HRQOL domains: physical wellbeing (n = 7), emotional wellbeing (n = 21) and social wellbeing (n = 10). The rich original qualitative dataset and the comprehensive interview and analysis processes informed the development of accompanying sets of questions on demographic characteristics, disease and treatment status, patient satisfaction with healthcare services and patient resilience (Table 4).

Table 4 Final structure and number of items in the questionnaire with examples of OVAQOL items


Using a CI approach, the purpose of this study was to refine and validate the wording of the items extracted from the data collected during a previous qualitative study with 14 women diagnosed with ovarian cancer. The draft items were administered through discussion, along with questions about the meaning and wording of the items. All items were adjusted based on participant input. The final set of items comprised 7 physical wellbeing items, 21 emotional wellbeing items and 10 social wellbeing items were identified as having meaning to the participants. They reflected a range of personal experiences, creating a new tool specific to women diagnosed with ovarian cancer. The CI process established content validity for the tool and highlighted the fundamental role of CI during questionnaire development. Feedback and suggestions provided by the participants. Following modification and evaluation, evidence for the content validity of the items was generated. Going forward, this new questionnaire has capacity to measure outcomes in women with OC and contribute to improving their health outcomes across the survivorship trajectory [30], and is ready for further validation.

Originally, the draft instrument consisted of 52 HRQOL items, and thereafter, has been reduced to 38 items. Since the questionnaire seeks to measure disease and treatment specific HRQOL outcomes of women diagnosed with OC, the items and its contents should reflect the purpose of measuring HRQOL. HRQOL is a multidimensional construct that measures the impact of a disease on physical, psychological and social relations aspects on a person’s life [31] as defined by the World Health Organisation [32]. Upon modification and evaluation, the items were examined and categorised into physical/functional, emotional and social domains consistent with this definition. The concept of HRQOL does not measure aspects such as job or income security [33]. Thus, items pertaining to accessing information in relation to financial needs were included in the patient satisfaction section with health services, while an item that measures psychological distress in relation to loss of income was merged with HRQOL emotional domain.

Our study findings illustrate how disease/treatment related symptoms had substantial impacts on the HRQOL of our study population, consistent with previous studies [34, 35]. As a consequence, these symptoms in turn affect the ability and capability to perform tasks either it be usual activities or professional [36]. Emergence of physical wellbeing constituted one of the important HRQOL domain. Included statements pertained to both physical health and functioning status of patients diagnosed with OC. The statements illustrated the impacts of symptoms from the patient perspective. Items related to “fatigue” and “difficulty sleeping” measure patient health while “participation in usual activities” focuses to measure patient functional ability. Similar to our findings, other studies have also revealed that people with chronic illness struggle with daily life tasks by being dependent on others [37, 38], revealing that it is important to measure physical domain of HRQOL for patients with OC.

Emotional wellbeing is a fundamental component of HRQOL instruments. Items including “depression”, “anxiety”, “fear of recurrence” were included in OVAQOL. Inclusion of such items enable evaluation of psychological distress experienced by women whether it be disease and/or treatment related by how severely it has impacted their HRQOL. By measuring psychological and emotional well-being and if difficulties are identified, clinicians could refer their patients to appropriate services and supports including counselling. Responses to PROMs questions help hospitals and healthcare services provide the care that patients need and want. These measures aim to fill a vital gap in our knowledge about outcomes that matter to patients [39]. In addition, it enables the researcher to identify whether such items impact other domains of HRQOL as previous research shows that psychological distress is related to poor performance status [40].

Personalized medicine should also focus on a patients’ cognitive, psychological, familial, and societal factors that influence clinical decisions. We believe that assessing the psycho-emotional status of OC patients and evaluating their resilience using OVAQOL will assist healthcare providers in conveying information in a tailored manner based on patients' characteristics, thereby providing suggestions and alerts; and potentially increase patient participation in the consultation process, as well as their satisfaction and involvement in their treatment decision-making processes.

Social functioning is also an essential component of HRQOL, particularly in relation to support provided by the family and friends which is emphasized in other similar qualitative studies [41]. In turn, social relations and mental wellbeing domains are interconnected. In a 2001 study of individuals with breast cancer, Kornblith and colleagues identified that women with low levels of support either it be through family, friends or professional, had higher levels of psychological distress throughout their clinical journey [42].

The initial qualitative dataset included important information describing factors related to the women’s HRQOL, including their satisfaction with health services, help-seeking throughout their clinical journey and resilience. In a recent study, it was indicated that HRQOL for those affected with systemic lupus erythematosus found a positive association between patient health care satisfaction and health status, possibly due to supports in the physical, emotional and social domains [33]. Recent studies also show that there is a direct relationship between resilience, life satisfaction and wellbeing on those living with chronic illness [43,44,45] and indicates the importance of measuring the impact of such variables on wellbeing and HRQOL. Thus, based on our findings and supporting information from various literature, relevant items associated with health services and informational challenges were collected into a module of questions describing ‘satisfaction with health care services’. Items that measured various coping strategies and resilience were collated into a ‘resilience’ module of questions to enable measurement of the self-empowerment and self-sufficiency strategies used by women during these difficult times.

Based on the development processes, the researchers believe that the content of OVAQOL truly reflected the consumer voice as was captured through the preceding qualitative study and the current CI procedures, each with patient involvement. This had not been performed to this extent in the development of existing OC HRQOL instruments suggesting limited patient involvement during the developmental stages of these PROMs. Therefore, the number of items in each HRQOL domain is larger compared to QLQ-OV28 and FACT-O. For example, EORTC QLQ-C30 has only four items and FACIT FACT-G has six items in the emotional domain, while OVAQOL measures 21 different aspects of emotional well-being, all of which were obtained by the individual experiences of women diagnosed with OC. In addition, there is some uncertainty regarding their factor structures. For example, low correlations are observed in several studies suggest that the social subscales of the EORTC-QLQ-C30 and FACIT FACT-G do not measure the same construct [46].

Consumer involvement in PROM development is essential as it is most appropriately women who have experienced OC who can determine outcome relevance and comprehensibility of the instrument [47, 48]. Previous studies have indicated that lack of patient involvement affects the sensitivity, validity and response of the questionnaire tool [49, 50], in which a 2017 study showed patient involvement had not increased in PROM development over time [19].


The study methodology itself is one of the main strengths of this study. Utilization of CI not only aided in the refinement of the statements in the tool, but also enabled identification of limitations which in turn assisted in the modification of the questionnaire construct that defines HRQOL. This study also had certain limitations. The onset of COVID-19 during the course of this study hindered the participant recruitment process, thereby impacting the study progress. Administering CI to a small sample size contributed to the second limitation as sampling variation could have increased with even more participant recruitment where evaluating and reviewing the contents of the items could have improved. Even though research has suggested that it is ideal to recruit between seven to 10 participants to check and confirm participant’s item comprehension, the variability in the participant number depends upon factors such as maximum sampling variation, questionnaire complexity and participant understanding of the items [51].


The current study utilized a systematic process to develop an OC specific PROM and highlights the value of CI for questionnaire item modification and content validity. This tool has potential to monitor patients' psycho-emotional condition in relation to their quality of life with greater granularity which could help healthcare practitioners more effectively identify and manage patients’ symptoms and concerns and track their progress over time. Validation of the PROM in a larger sample and evaluation of its psychometric properties is an essential next research project.

Availability of data and materials

The datasets used and/or analysed during the current study are currently safely stored in NVivo software within researcher’s work laptop and are available from the corresponding author on reasonable request.



Cancer Council Western Australia


Cognitive Interviewing


Concurrent Think-Aloud


European Organization for Research and Treatment of Cancer Quality of Life Questionnaire for Cancer Patients


European Organization for Research and Treatment of Cancer Quality of Life Questionnaire-Ovarian


Functional Assessment of Chronic Illness Therapy Functional Assessment of Cancer Therapy–General


Functional Assessment of Cancer Therapy–Ovarian


Functional Assessment of Cancer Therapy–ovarian cancer symptom index


Health-Related Quality of Life


Measure of Ovarian Symptoms and Treatment concerns


Ovarian Cancer


Ovarian Cancer Australia


OVArian cancer health related Quality Of Life scale


Patient Reported Outcome Measures


University of Notre Dame Australia


  1. Zhang Y, Luo G, Li M, Guo P, Xiao Y, Ji H, et al. Global patterns and trends in ovarian cancer incidence: age, period and birth cohort analysis. BMC Cancer. 2019;19(1):984.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Cancer Australia. Ovarian cancer statistics Available from: [Accessed 30 Sept 2019].

  3. Jones J, Howell D, Grunfeld E. Cancer survivorship: a local and global issue in cancer control. Lancet Glob Health. 2018.

  4. Ashley L, Velikova G, Downing A, Morris E, Wright P. Health-related quality of life in cancer survivorship: predictive power of the Social Difficulties Inventory. Psychooncology. 2017;26(11):1994–7.

    Article  PubMed  Google Scholar 

  5. Subramaniam S, Kong Y-C, Chinna K, Kimman M, Ho Y-Z, Saat N, et al. Health-related quality of life and psychological distress among cancer survivors in a middle-income country. Psychooncology. 2018;27(9):2172–9.

    Article  PubMed  Google Scholar 

  6. Fayers PM, Machin D. Quality of life: the assessment, analysis, and reporting of patient-reported outcomes. 3rd ed. Wiley Blackwell;2016.

  7. Nixon A, Wild D, Muehlhausen W. Patient reported outcomes : an overview. First ed: SEEd;2015.

  8. Jokstad A. Patient-reported outcomes (PROs) versus patient-reported outcome measures (PROMs)—is there a difference? Clin Exp Dent Res. 2018;4(3):61–2.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Lavallee DC, Chenok KE, Love RM, Petersen C, Holve E, Segal CD, et al. Incorporating patient-reported outcomes into health care to engage patients and enhance care. Health Aff (Project Hope). 2016;35(4):575–82.

    Article  Google Scholar 

  10. Roydhouse JK, Gutman R, Keating NL, Mor V, Wilson IB. Proxy and patient reports of health-related quality of life in a national cancer survey. Health Qual Life Outcomes. 2018;16(1):6–11.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Moss HA, Havrilesky LJ. The use of patient-reported outcome tools in Gynecologic Oncology research, clinical practice, and value-based care. Gynecol Oncol. 2018;148(1):12–8.

    Article  PubMed  Google Scholar 

  12. Friedlander ML, King MT. Patient-reported outcomes in ovarian cancer clinical trials. Ann Oncol. 2013;24(Suppl 10):x64–8.

    Article  PubMed  Google Scholar 

  13. Questionnaires. EORTC Qualtiy of Life Available from: [Accessed 29 Apr 2019].

  14. Questionnaires. FACIT Available from: [Accessed 29 Apr 2019].

  15. King MT, Stockler MR, O’Connell RL, Buizen L, Joly F, Lanceley A, et al. Measuring what matters MOST: validation of the Measure of Ovarian Symptoms and Treatment, a patient-reported outcome measure of symptom burden and impact of chemotherapy in recurrent ovarian cancer. Qual Life Res. 2018;27(1):59–74.

    Article  PubMed  Google Scholar 

  16. Kemmler G, Holzner B, Kopp M, Dünser M, Margreiter R, Greil R, et al. Comparison of two quality-of-life instruments for cancer patients: the functional assessment of cancer therapy-general and the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire-C30. J Clin Oncol. 1999;17(9):2932–40.

    Article  CAS  PubMed  Google Scholar 

  17. Agarwal S, Bodurka DC. Symptom research in gynecologic oncology: a review of available measurement tools. Gynecol Oncol. 2010;119(2):384–9.

    Article  PubMed  Google Scholar 

  18. Kirwan JR, Fries JF, Hewlett S, Osborne RH. Patient perspective: choosing or developing instruments. J Rheumatol Suppl. 2011;38(8):1716–9.

    Article  Google Scholar 

  19. Wiering BM, De Boer D, Delnoij D. Patient involvement in the development of patient-reported outcome measures: The developers’ perspective. BMC Health Serv Res. 2017;17(1):635.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Boban S, Downs J, Codde J, Cohen PA, Bulsara C. Women diagnosed with ovarian cancer: patient and carer experiences and perspectives. Patient Relat Outcome Meas. 2021;12:33–43.

    Article  PubMed  PubMed Central  Google Scholar 

  21. United States Food & Drug Administration. Clinical Outcome Assessment (COA) Qualification Program Available from: [Accessed 25 Oct 2020].

  22. Ryan K, Gannon-Slater N, Culbertson MJ. Improving survey methods with cognitive interviews in small- and medium-scale evaluations. Am J Eval. 2012;33(3):414–30.

    Article  Google Scholar 

  23. Peterson CH, Peterson NA, Powell KG. Cognitive interviewing for item development: validity evidence based on content and response processes. Meas Eval Couns Dev. 2017;50(4):217–23.

    Article  Google Scholar 

  24. Castillo-Díaz M, Padilla J-L. How cognitive interviewing can provide validity evidence of the response processes to scale items. Soc Indic Res. 2013;114(3):963–75.

    Article  Google Scholar 

  25. Charters E. The use of think-aloud methods in qualitative research: an introduction to think-aloud methods. Brock Educ. 2003;12(2):68–82.

    Article  Google Scholar 

  26. Alhadreti O, Mayhew P. To intervene or not to intervene: An investigation of three think-aloud protocols in usability testing. J Usability Stud. 2017;12(3):111–32.

    Google Scholar 

  27. Polit DF, Beck CT. Essentials of nursing research: appraising evidence for nursing practice. Ninth ed: Wolters Kluwer Health;2018.

  28. Frey B. NVivo. The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation: SAGE Publications;2018.

  29. Brooks J, McCluskey S, Turley E, King N. The utility of template analysis in qualitative psychology research. Qual Res Psychol. 2015;12(2):202–22.

    Article  PubMed  Google Scholar 

  30. Basch E, Spertus J, Adams Dudley R, Wu A, Chuahan C, Cohen P, et al. Methods for developing patient-reported outcome-based performance measures (PRO-PMs). Value Health. 2015;18(4):493–504.

    Article  PubMed  Google Scholar 

  31. Taylor RM, Gibson F, Franck LS. A concept analysis of health-related quality of life in young people with chronic illness. J Clin Nurs. 2008;17(14):1823–33.

    Article  PubMed  Google Scholar 

  32. World Health Organization. Constitution of the World Health Organization. Am J Public Health Nations Health. 1946;36(11):1315–23.

    Article  Google Scholar 

  33. McElhone K, Abbott J, Teh LS. A review of health related quality of life in systemic lupus erythematosus. Lupus. 2016;15(10):633–43.

    Article  Google Scholar 

  34. Sun CC, Bodurka DC, Weaver CB, Rasu R, Wolf JK, Bevers MW, et al. Rankings and symptom assessments of side effects from chemotherapy: insights from experienced patients with ovarian cancer. Support Care Cancer. 2004;13(4):219–27.

    Article  PubMed  Google Scholar 

  35. Lindley CM, Hirsch JD, O’Neill CV, Transau MC, Gilbert CS, Osterhaus JT. Quality of life consequences of chemotherapy-induced emesis. Qual Life Res. 1992;1(5):331–40.

    Article  CAS  PubMed  Google Scholar 

  36. O’Brien BJ, Rusthoven J, Rocchi A, Latreille J, Fine S, Vandenberg T, et al. Impact of chemotherapy-associated nausea and vomiting on patients’ functional status and on costs: survey of five Canadian centres. CMAJ. 1993;149(3):296–302.

    CAS  PubMed  PubMed Central  Google Scholar 

  37. Paap MCS, Bode C, Lenferink LIM, Groen LC, Terwee CB, Ahmed S, et al. Identifying key domains of health-related quality of life for patients with Chronic Obstructive Pulmonary Disease: the patient perspective. Health Qual Life Outcomes. 2014;12(1):106–18.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Lee GL, Pang GSY, Akhileswaran R, Ow MYL, Fan GKT, Wong CCF, et al. Understanding domains of health-related quality of life concerns of Singapore Chinese patients with advanced cancer: a qualitative analysis. Support Care Cancer. 2016;24(3):1107–18.

    Article  CAS  PubMed  Google Scholar 

  39. Cohen PA, Webb PM, King M, Obermair A, Gebski V, Butow P, et al. Getting the MOST out of follow-up: a randomized controlled trial comparing 3 monthly nurse led follow-up via telehealth, including monitoring CA125 and patient reported outcomes using the MOST (Measure of Ovarian Symptoms and Treatment concerns) with routine clinic based or telehealth follow-up, after completion of first line chemotherapy in patients with epithelial ovarian cancer. Int J Gynecol Cancer. 2022;32(4):560–5.

    Article  PubMed  Google Scholar 

  40. Bodurka-Bevers D, Basen-Engquist K, Carmack CL, Fitzgerald MA, Wolf JK, de Moor C, et al. Depression, anxiety, and quality of life in patients with epithelial ovarian cancer. Gynecol Oncol. 2000;78(3):302–8.

    Article  CAS  PubMed  Google Scholar 

  41. Ferrell BR, Smith SL, Ervin KS, Itano J, Melancon C. A qualitative analysis of social concerns of women with ovarian cancer. Psychooncology. 2003;12(7):647–63.

    Article  PubMed  Google Scholar 

  42. Kornblith AB, Herndon nJE, Zuckerman E, Viscoli CM, Horwitz RI, Cooper MR, et al. Social support as a buffer to the psychological impact of stressful life events in women with breast cancer. Cancer. 2001;91(2):443–54.

  43. Tecson KM, Wilkinson LR, Smith B, Ko JM. Association between psychological resilience and subjective well-being in older adults living with chronic illness. Proc (Bayl Univ Med Cent). 2019;32(4):520–4.

    Article  Google Scholar 

  44. Bajaj B, Pande N. Mediating role of resilience in the impact of mindfulness on life satisfaction and affect as indices of subjective well-being. Pers Individ Dif. 2016;93:63–7.

    Article  Google Scholar 

  45. Seiler A, Jenewein J. Resilience in cancer patients. Front Psychiatry. 2019;10:208.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Holzner B, Bode RK, Hahn EA, Cella D, Kopp M, Sperner-Unterweger B, et al. Equating EORTC QLQ-C30 and FACT-G scores and its use in oncological research. Eur J Cancer. 2006;42(18):3169–77.

    Article  CAS  PubMed  Google Scholar 

  47. Trujols J, Portella MJ, Iraurgi I, Campins MJ, Siñol N, Cobos JPdL. Patient-reported outcome measures: Are they patient-generated, patient-centred or patient-valued? J Ment Health. 2013;22(6):555–62.

  48. Staniszewska S, Adebajo A, Barber R, Beresford P, Brady LM, Brett J, et al. Developing the evidence base of patient and public involvement in health and social care research: the case for measuring impact. Int J Consum Stud. 2011;35(6):628–32.

    Article  Google Scholar 

  49. Fossey EM, Harvey CA. A conceptual review of functioning: implications for the development of consumer outcome measures. Aust N Z J Psychiatry. 2001;35(1):91–8.

    Article  CAS  PubMed  Google Scholar 

  50. Meadows KA. Patient-reported outcome measures: an overview. Br J Community Nurs. 2011;16(3):146–51.

  51. Leidy NK, Vernon M. Perspectives on patient-reported outcomes : content validity and qualitative research in a changing clinical trial environment. Pharmacoeconomics. 2008;26(5):363–70.

    Article  Google Scholar 

Download references


The authors wish to thank all the patients who participated in this study. This study was supported by the Continuous Improvement in Care-Cancer Project. Ms. Sharolin Boban is a Masters’ of Science student at the University of Notre Dame Australia, Fremantle and this work forms part of her thesis.


This study was done with the support of a Cancer Research Trust grant and is part of the Continuous Improvement in Care-Cancer Initiative, a multi-disciplinary research program, aiming to provide value-based healthcare in clinical settings in Western Australia. JD was supported by Department of Health Western Australia Merit Award. The funding body played no role in the design of the study, its analysis, interpretation of the data or in the writing of the manuscript.

Author information

Authors and Affiliations



All authors made significant contributions to the study's conception and design. SB and JD contributed to the study execution and performance of the research. SB acquired the data and conducted data analysis. Interpretation of data was conducted by SB and JD. SB contributed to creating the first draft. JD, CB, JC and PC collaborated in further iterations, revising it critically for important intellectual content; agreed to submit it to the current journal. All authors have read and approved the final manuscript; gave approval of the final version to be published; and agreed to hold all the authors transparent.

Corresponding author

Correspondence to Sharolin Ann Boban.

Ethics declarations

Ethics approval and consent to participate

This study was granted ethics approval by the Human Research Ethics Committee at University of Notre Dame Australia (UNDA) (2020-010F) and our study complied with the Declaration of Helsinki. All participants were asked to return their signed informed consent form prior to the commencement of the interview. Participants were informed the intent of the study which included agreement to the release of study results.

Consent for publication

The participant informed consent form included a statement related to the authorization for publishing study results.

Competing interests

There are no competing interests in this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1

: Published work of preceding research. Prior to this study, a qualitative study with women with OC was conducted, which identified key experiences and priorities such as challenges related to diagnosis and treatment, adjustments in their relationships with family and/or friends, and comments on useful coping strategies. These findings were expanded into a set of items that could be useful in developing a values-based OC PROM.

Additional file 2

: Cognitive Interviewing Questionnaire. The questionnaire was created specifically for this study. CI using the CTA procedure was used to test identified statements from our previous qualitative study. Set of draft items was administered via telephone, Zoom and WhatsApp app on fourteen women with OC.

Additional file 3

: Consolidated criteria for reporting qualitative studies (COREQ): 32-item checklist. The study was carried out in accordance with the COREQ 32-item checklist, which assisted researchers in reporting important elements of the qualitative research, study design, study context, findings, evaluation, and interpretations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Boban, S.A., Bulsara, C., Codde, J. et al. Employing cognitive interviewing to evaluate, improve and validate items for measuring the health-related quality of life of women diagnosed with ovarian cancer. BMC Women's Health 22, 391 (2022).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI:


  • Cognitive interviewing
  • Health-related quality of life
  • Ovarian cancer
  • Patient reported outcome measures