Skip to main content

Acceptability of quality indicators for the management of endometrial, cervical and ovarian cancer: results of an online survey



Measuring quality indicators (QI’s) is a tool to improve the quality of care. The aim of this study was to evaluate the acceptability of 36 QI’s, defined after a literature search for the management of endometrial, cervical and ovarian cancer. Relevant specialists in the field of interest were surveyed.


To quantify the opinions of these specialists, an online survey was sent out via mailing to members of gynaecological or oncological societies. The relevance of each QI was questioned on a scale from one to five (1 = irrelevant, 2 = less relevant, 3 = no opinion/neutral, 4 = relevant, 5 = very relevant). If a QI received a score of 4 or 5 in 65% or more of the answers, we state that the respondents consider this QI to be sufficiently relevant to use in daily practice.


The survey was visited 238 times and resulted in 53 complete responses (29 Belgian, 24 other European countries). The majority of the specialists were gynaecologists (45%). Five of the 36 QI’s (13,9%) did not reach the cut-off of 65%: referral to a tertiary center, preoperative staging of endometrial cancer by MRI, preoperative staging of cervical cancer by positron-emission tomography, incorporation of intracavitary brachytherapy in the treatment of cervical cancer, reporting ASA and WHO score for each patient. After removing the 5 QI’s that were not considered as relevant by the specialists and 3 additional 3 QI’s that we were considered to be superfluous, we obtained an optimized QI list.


As QI’s gain importance in gynecological oncology, their use can only be of value if they are universally interpreted in the same manner. We propose an optimized list of 28 QI’s for the management of endometrial, cervical and ovarian cancer which responders of our survey found relevant. Further validation is needed to finalize and define a set of QI’s that can be used in future studies, audits and benchmarking.

Peer Review reports


During the last decade, there is a growing interest in measuring the quality of care as it has been proven that this has a positive impact on the survival of the patients [1]. In order to improve the standards of care, it is of paramount importance to identify suboptimal elements in the clinical trajectory of a patient. These may be related to factors implicated in the diagnosis, decision making and/or treatment of patients. A tool to expose these deficiencies are quality indicators (QI’s). These are norms, criteria, standards or other direct qualitative and quantitative measures of health care quality that make use of readily available hospital inpatient administrative data [2]. They can highlight potential quality concerns, identify areas that need further study and investigation, track changes over time, and enable physicians to compare their practices with the guidelines and with each other, in order to strive for optimal health care [3, 4]. Recent literature shows that less than 50% of patients suffering from ovarian cancer in Europe are treated according to the guidelines and that there is a correlation between treatment adherent to the guidelines and survival [5]. This is a striking example demonstrating that there is a high need for quality control and benchmarking in our health care systems [4]. Unfortunately, many physicians and hospitals consider measuring QIs an obstacle of their practice and functioning, mainly because of the administrative burden and costs involved [4, 6].

We recently performed a systematic literature search to identify QI’s for the management of patients with endometrial, cervical and ovarian cancer and could select a list of 36 relevant QI’s [7]. We focused on structural, process and outcome measures of health care predominantly for the management of these three cancers. Structural measures refer to health care facility resources, e.g. the number and qualification of staff, case volume of the hospital and specialist, supplies, access to specific technologies etc. [8, 9]. Process measures refer to specific actions implemented to achieve an optimal result, where outcome measures are indicators of the total health of treated patients and the quality of given care [7,8,9]. Our search showed that important QI’s related to the structural organization of health care were referral to high volume specialists in high volume hospitals, treatment by specialized gynaecological oncologists and discussion of a treatment plan by a multidisciplinary team according to current guidelines. Process measures were adequate pretreatment staging, a patient report of high quality and adherence to treatment guidelines, while outcome indicators were treatment related morbidity and increased survival. In order to be relevant in a clinical setting, the list of 36 selected QI’s list should adequately measure the quality of care, improve the quality of care in daily practice, be cost-effective and contribute to create uniformity.

Optimally, consensus guidelines and QI’s are formulated after a systematic assessment of the literature looking for evidence based healthcare recommendations on how to treat or diagnose a disease with the aim of better patient outcomes [10]. In order to obtain a widely supported workable framework the Delphi method is often used to further refine and define these guidelines and QI’s in a second stage. The Delphi method is a forecasting process framework whereby questionnaires on the subject of interest are send out to a panel of experts asking them to score the relevance of the guidelines. When opinions differ on certain issues, the experts are allowed to adjust their answers anonymously in subsequent rounds based on how they interpret the group response, thereby aiming to work towards a mutual agreement. This was achieved successfully for many tumor types including breast cancer [11, 12]. Although international organizations such as ESMO and ESGO have provided guidelines on the treatment of gynaecological cancers, up to now no agreement has been reached on the proper QI’s which could be used to measure and benchmark the quality of the provided care [7, 13,14,15]. Therefore, we decided to perform an online survey for specialists involved in the treatment of patients with endometrial, cervical and ovarian cancer to question the acceptability and practicability in daily practice of the QI’s which were distilled out of our literature search [5].


Study population

To evaluate the relevance of the 36 QI’s [10], we designed an online survey (Additional file 1). The questionnaire was sent by electronic mail to all members of the Flemish and Dutch Society of Obstetricians and Gynaecologists, and the Belgian Society of Medical Oncology. A link to the questionnaire with an invitation to participate was also put on the website of the European Society of Gynecological Oncology, and was also available in our paper published in the European Journal of Surgical Oncology [7]. The full survey can be found in the Additional file 1. Participating specialists were informed that they answered the list voluntarily without any rewards and that their anonymized answers would be used for the further course of the study. Since we received less than 50 responses in first 3 months, we sent reminders to all above mentioned specialists.


This survey was created using an online form ( and was accessible from 1 September 2018 to 28 February 2019. The participants were asked to indicate their specialism (gynaecological oncologist, general gynaecologist, medical oncologist, radiotherapist or other) and demographics. The answers of non-European specialists were excluded. The relevance of each QI was questioned on a scale from one to five (1 = irrelevant, 2 = less relevant, 3 = no opinion/neutral, 4 = relevant, 5 = very relevant). Physicians evaluating a QI not belonging to their discipline (eg radiotherapist on surgical issues), had the possibility to not score that QI. In contrast to process measures which are cancer specific, structural and outcome measures applicable for all three cancer types were not questioned separately per cancer type. At the end of the questionnaire, there was room to give remarks and suggestions. For each QI, the ‘Survio’ software recorded all given answers and the frequency of each particular score (1, 2, 3, 4 and 5). Subsequently, the corresponding percentages were calculated. If a QI received a score of 4 or 5 in 65% or more of the answers, we state that the respondents consider this QI to be sufficiently relevant to use in daily practice. The percentage of 65 has been set by mutual agreement (AB, AL, LW, PVD, XBT).

Statistical analysis

Statistical analysis was performed using the IBM software package, SPSS V.25, and results were graphically displayed with GraphPad prism V.6.0 h.

Ethical considerations

Ethical approval for the study was obtained by the Ethical Committee of the Antwerp University Hospital (nr. 19/16/215). No human subjects were involved in this study. The physicians responding to the questionnaires agreed that their answers would be used for statistical analysis (see Additional file 1) and publication.


Respondents characteristics

Sending out the survey to the specialists resulted in 238 visits and 53 responses. Twenty-nine respondents (54,7%) are working in Belgium and 24 respondents (45.3%) are employed in another European country. The majority of these specialists are gynaecologists (45.3%). The demographic characteristics of the respondents are displayed in Table 1.

Table 1 Characteristics of respondents

Percentages of given scores

As mentioned above, the percentages of given scores were calculated for each QI and are displayed in Figs. 1, 2, 3, 4 and 5. Figure 1 contains the results of the structural and outcome QI’s for both endometrium, cervix and ovarian cancer. Figure 2 contains the results of the common process QI’s for the three cancers. The results of the specific process QI’s are displayed in a separate chart per cancer type (Figs. 3, 4 and 5).

Fig. 1
figure 1

Relevance of structural and outcome quality indicators for endometrial, cervical and ovarian cancer

Fig. 2
figure 2

Relevance of common process quality indicators for endometrial, cervical and ovarian cancer

Fig. 3
figure 3

Relevance of specific process quality indicators for endometrial cancer

Fig. 4
figure 4

Relevance of specific process quality indicators for cervical cancer

Fig. 5
figure 5

Relevance of specific process quality indicators for ovarian cancer

Identification of insufficiently relevant quality indicators

To consider a quality indicator as sufficiently relevant to use in daily practice, we stated that the total percentage of scores 4 and 5 had to be 65% or more. Thirty one of the 36 QI’s (86,1%) reached this cut-off. Seven QI’s were indicated as “very relevant” in 65% or more of the given answers and thus achieved this cut-off with only the percentage of score 5. This concerns QI 5, QI 14, QI 20, QI 21, QI 22, QI 23, QI 34. Five QI’s (13,9%) did not reach the cut-off of 65%: QI 3, QI 6, QI 12, QI 19, QI32. Therefore, we conclude that the specialists considered these QI’s as insufficiently relevant. These five QI’s are discussed in detail below.


Gynaecological cancers are challenging and complex diseases. According to the GLOBOCAN 2018 estimates on the global cancer burden, produced by the International Agency for Research, the incidence and mortality of cervical cancer, cancer of the corpus uteri and ovarian cancer respectively are 569.847, 382.069, 299.414 and 311.365, 89.929, 184.799 [16]. In developed countries, endometrial cancer is the most common of these gynecological cancers and ovarian cancer is the leading cause of death [17, 18]. Although this study mainly focuses on the quality of care to ultimately increase the survival of gynaecological cancer patients, the quality of life should not be omitted. These women have to deal with physical changes, but also with mental and social challenges influencing their overall well-being. The physical changes comprise menopausal symptoms, a changed sexual life, bowel and/or urinary tract problems etc. An important part of the rehabilitation and follow-up of these patients needs to focus on conversations with health professionals around these challenges [19, 20]. In the study of Vitale et al. it is stated that not only young women, but also the elderly are entitled to a standard treatment. Elderly patients are often undertreated because of their many morbidities and low life expectancy, but treatment according to the guidelines can increase their quality of life [21].

Several studies have been performed listing relevant QI’s to measure the quality of cancer care. We were the first to create one list for endometrial, cervical and ovarian cancer [7]. The present survey examined the relevance of the 36 quality indicators which were retained in the above-mentioned literature search. Using an online survey, sent to physicians active in the field, we have shown that 5 of these indicators are not considered to be relevant to measure the quality of care for these three cancers.

“Referral to a tertiary center (QI 3)” is a structural quality indicator that was questioned for all three cancer types by one single statement in our survey. Although the literature supports that there is a survival advantage for patients treated for ovarian and cervical cancers in high volume centers by high volume surgeons, the evidence is much less clear for endometrial cancer [22]. At the end of the questionnaire the specialists had the opportunity to give remarks and suggestions. Some correspondents correctly regretted that they could not score this QI separately for the different tumors types as they considered it relevant for ovarian and cervical cancer. Although surgery is often less complex and the need for adjuvant treatment is often not indicated in endometrial cancer patients with low risk histology, this is not the case in patients with high risk histology. Recent evidence is emerging that in this second group of patients perioperative mortality is higher and survival is reduced in low volume hospitals [22, 23]. It is likely that in a true Delphi procedure this QI would have been debated and rephrased in a second round and accepted for cancer of the ovary, cervix and high risk endometrial cancer.

“Preoperative assessment by MRI and/or computed tomography in patients with endometrial cancer (QI 6)” and “Pretreatment positron emission tomography-computed tomography imaging (PET-CT) should be performed when primary definitive surgery is not appropriate in cervical cancer (QI 12)” are staging process quality indicators. These were based on the opinions of scientific societies but are not feasible in all centers. Although they may have an impact on referral patterns of patients, there is no hard evidence that these issues have a direct impact on patient survival. We therefore dropped them as mandatory QI’s.

According to QI 19, another process indicator, “intracavitary brachytherapy should be incorporated into the treatment of patients with locally advanced cervical cancer”. Our respondents did not accept this QI as mandatory, probably because this technology is not available in all units. It should be mentioned that this QI was the most endorsed QI among specialized radiotherapists, as individual tailored radiotherapy treatment incorporating brachytherapy results in a better outcome of patients [24, 25]. This underscores the need to refer these patients to tertiary centers offering this treatment modality to patients. We therefore would advise to incorporate this QI in future quality control programs.

“Reporting of an ASA and/or WHO score for each patient” (QI 32) is a common process quality indicator for all three cancer types. Although associations between ASA and WHO scores and specific surgical complications or outcomes have been reported in the literature, they can vary over the treatment trajectory of a patient. It was considered complicated by some respondents to register them systematically. To maintain the practicality, this QI was not incorporated in the optimized QI list [26].

The common QI’s must be assessed separately per cancer type. Therefore, they are mentioned separately for each cancer type in the optimized list. Additionally, we have split the outcome QI into a separate QI for 1, 3 and 5-year survival so this can be investigated more accurately.

One of the limitations of the present study is the limited response rate. This is comparable to the proportion of answers recorded after similar electronic surveys on other subjects [27, 28]. Apparently, it is difficult to motivate clinicians to fill in questionnaires voluntarily amid the many other occupations of daily and professional life. However, our respondents seemed to be very motivated. They could make general remarks regarding the entire survey and often did so. We considered every opinion given by a specialist in this field of interest useful as it reflects the vision of clinicians in the real world, which may differ from the view of “experts”. Some of them stated that a few of the QI’s on the list are rather vague, unclear or quite obvious and that some similar QI’s should be combined into one unambiguous QI. We have gone through all QI’s and consequently concluded to remove three additional QI’s from our QI list. The first one is QI2: “Proportion of patients with endometrial, cervical and ovarian cancer treated according to the guidelines”. This QI is indeed superfluous since it is very general, overarching, difficult to measure and it is not specified which guidelines are involved. We want to emphasize that we do think this QI is theoretically very relevant if it is well defined (e.g. according to ESMO guidelines), but we doubt its usefulness in daily practice due to its complexity. In addition, several retained QI’s measure this indirectly. The other two QI’s that we do not preserve in our final list are QI23 and QI25. These process QI’s concern the surgical staging of ovarian tumors. After discussion, we felt that these QI’s are unnecessary because other QI’s cover their content (e.g. QI21 and QI26).


Our study contributes to the standardizing of healthcare practices, which is essential to deliver consistent and cost-effective healthcare. After removing the 5 QI’s that are not considered as relevant by the specialists and the additional 3 QI’s that we have decided to be superfluous, we obtained an optimized QI list for the management of endometrial, cervical and ovarian cancer (Additional file 2). We hope this paper can initiate a Delphi meeting by experts on this subject finalizing a definite set of QI’s that can be used in a subsequent studies and audits in daily practice.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.



Quality Indicator


European Society of Medical Oncology


European Society of Gynaecological Oncology


Global Cancer Observatory


Magnetic Resonance Imaging


American Society of Anesthesiologists


World Health Organization


  1. Querleu D, Ray-Coquard I, Classe JM, Aucouturier JS, Bonnet F, Bonnier P, et al. Quality indicators in ovarian cancer surgery: report from the French Society of Gynecologic Oncology (Societe Francaise d'Oncologie Gynecologique, SFOG). Ann Oncol. 2013;24(11):2732–9.

    Article  CAS  PubMed  Google Scholar 

  2. Quality Indicators, Health Care 1998 [Available from:

    Google Scholar 

  3. Chenoz L, Phelippeau J, Barranger E, Bourdel N, Brun JL, Chereau E, et al. Evaluation and selection of quality Indicators for the Management of Endometrial Cancer. Int J Gynecol Cancer. 2017;27(5):979–86.

    Article  PubMed  Google Scholar 

  4. De Vos M, Graafmans W, Kooistra M, Meijboom B. Using quality indicators to improve hospital care: a review of the literature. Int J Qual Health Care. 2009;21(2):119–29.

    Article  PubMed  Google Scholar 

  5. Sobrero S, Pagano E, Piovano E, Bono L, Ceccarelli M, Ferrero A, et al. Is ovarian Cancer being managed according to clinical guidelines? Evidence from a population-based clinical audit. Int J Gynecol Cancer. 2016;26(9):1615–23.

    Article  PubMed  Google Scholar 

  6. Aletti GD, Peiretti M. Quality control in ovarian cancer surgery. Best Pract Res Clin Obstet Gynaecol. 2017;41:96–107.

    Article  PubMed  Google Scholar 

  7. Bonte AS, Luyckx A, Wyckmans L, Trinh XB, van Dam PA. Quality indicators for the management of endometrial, cervical and ovarian cancer. Eur J Surg Oncol. 2018;45:528–37.

    Article  PubMed  Google Scholar 

  8. Verleye L, Ottevanger PB, van der Graaf W, Reed NS, Vergote I, Gynaecological Cancer Group of European Organisation for R, et al. EORTC-GCG process quality indicators for ovarian cancer surgery. Eur J Cancer. 2009;45(4):517–26.

    Article  CAS  PubMed  Google Scholar 

  9. Verleye L, Vergote I, Reed N, Ottevanger PB. Quality assurance for radical hysterectomy for cervical cancer: the view of the European Organization for Research and Treatment of Cancer--gynecological Cancer group (EORTC-GCG). Ann Oncol. 2009;20(10):1631–8.

    Article  CAS  PubMed  Google Scholar 

  10. Chinmayee J. Consensus Guidelines: a tool for establishing evidence-based practices. Medical affairs blog; 2017. [Available from:

  11. van Dam PA, Tomatis M, Marotti L, Heil J, Wilson R, Rosselli Del Turco M, et al. The effect of EUSOMA certification on quality of breast cancer care. Eur J Surg Oncol. 2015;41(10):1423–9.

    Article  PubMed  Google Scholar 

  12. van Dam PA, Tomatis M, Marotti L, Heil J, Mansel RE, Rosselli Del Turco M, et al. Time trends (2006–2015) of quality indicators in EUSOMA-certified breast centres. Eur J Cancer. 2017;85:15–22.

    Article  PubMed  Google Scholar 

  13. Marth C, Landoni F, Mahner S, McCormack M, Gonzalez-Martin A, Colombo N. Cervical cancer: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Ann Oncol. 2017;28(suppl_4):iv72–83.

    Article  CAS  PubMed  Google Scholar 

  14. Ledermann JA, Raja FA. Newly diagnosed and relapsed epithelial ovarian carcinoma: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Ann Oncol. 2013;24:vi24–32.

    Article  PubMed  Google Scholar 

  15. Colombo N, Preti E, Landoni F, Carinelli S, Colombo A, Marini C, et al. Endometrial cancer: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Ann Oncol. 2013;24(Suppl 6):vi33–8.

    Article  PubMed  Google Scholar 

  16. Kankerregister S. Cancer Fact sheets Brussel: stichting Kankerregister; 2017 [Available from:

    Google Scholar 

  17. Harter P, Muallem ZM, Buhrmann C, Lorenz D, Kaub C, Hils R, et al. Impact of a structured quality management program on surgical outcome in primary advanced ovarian cancer. Gynecol Oncol. 2011;121(3):615–9.

    Article  PubMed  Google Scholar 

  18. du Bois A, Strutas D, Buhrmann C, Traut A, Ewald-Riegler N, Kommoss S, et al. Impact of treatment guidelines and implementation of a quality assurance program on quality of care in endometrial cancer. Onkologie. 2009;32(8–9):493–8.

    Article  PubMed  Google Scholar 

  19. Sekse RJT, Dunberger G, Olesen ML, Osterbye M, Seibaek L. Lived experiences and quality of life after gynaecological cancer-an integrative review. J Clin Nurs. 2019;28(9–10):1393–421.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Oliva A, Garcia-Cebrian JM, Calatayud EF, Serrano-Garcia I, Herraiz MA, Coronado PJ. A comparison of quality of life and resilience in menopausal women with and without a history of gynaecological cancer. Maturitas. 2019;120:35–9.

    Article  PubMed  Google Scholar 

  21. Vitale SG, Capriglione S, Zito G, Lopez S, Gulino FA, Di Guardo F, et al. Management of endometrial, ovarian and cervical cancer in the elderly: current approach to a challenging condition. Arch Gynecol Obstet. 2019;299(2):299–315.

    Article  PubMed  Google Scholar 

  22. Buskwofie A, Huang Y, Tergas AI, Hou JY, Ananth CV, Neugut AI, et al. Impact of hospital volume on racial disparities and outcomes for endometrial cancer. Gynecol Oncol. 2018;149(2):329–36.

    Article  PubMed  PubMed Central  Google Scholar 

  23. David I, Shalowitz EAJ, Buckingham L, Ko EM, Giuntoli RL 2nd. Survival implications of time to surgical treatment of endometrial cancers. Am J Obstet Gynecol. 2017;216(3):268 e1–e18.

    Article  Google Scholar 

  24. Viswanathan AN, Moughan J, Small W Jr, Levenback C, Iyer R, Hymes S, et al. The quality of cervical cancer brachytherapy implantation and the impact on local recurrence and disease-free survival in radiation therapy oncology group prospective trials 0116 and 0128. Int J Gynecol Cancer. 2012;22(1):123–31.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Croke J, Fyles A, Barbera L, D'Souza D, Pearcey R, Stuckless T, et al. Radiation therapy quality-of-care indicators for locally advanced cervical cancer: a consensus guideline. Pract Radiat Oncol. 2016;6(5):315–23.

    Article  PubMed  Google Scholar 

  26. Daabiss M. American Society of Anaesthesiologists physical status classification. Indian J Anaesthesia. 2011;55(2):111–5.

    Article  Google Scholar 

  27. Kordes M, Gerling M. Variations in the management of diarrhoea induced by cancer therapy: results from an international, cross-sectional survey among European oncologists. ESMO Open. 2019;4(6):e000607.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Richtig G, Richtig E, Bohm A, Oing C, Bozorgmehr F, Kruger S, et al. Awareness of predatory journals and open access among medical oncologists: results of an online survey. ESMO Open. 2019;4(6):e000580.

    Article  PubMed  PubMed Central  Google Scholar 

Download references


An Van Goethem for all the administrative work.


No funding.

Author information

Authors and Affiliations



AL, LW and AB designed the survey, performed the data analyses and wrote the manuscript. PVD and XBT supervised the above and made contributions to the analysis and interpretation of data. All authors have read and approved the submitted manuscript and are accountable for the author’s own contributions.

Corresponding author

Correspondence to Peter A. van Dam.

Ethics declarations

Ethics approval and consent to participate

The study was approved by the Local Ethical Committee of the Antwerp University (number 19/16/215). The physicians responding to the survey were informed that their answers would be used for statistical analysis (see Additional file 1) and that replying to the questionnaire implied that a written informed consent was given.

Consent for publication

Not applicable.

Competing interests

No competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1: Appendix 1.

Survey “Relevance of quality indicators in the management of endometrial, cervical and ovarian cancer”.

Additional file 2: Appendix 2.

Optimised QI list.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luyckx, A., Wyckmans, L., Bonte, AS. et al. Acceptability of quality indicators for the management of endometrial, cervical and ovarian cancer: results of an online survey. BMC Women's Health 20, 151 (2020).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: