Expert consensus on the development of a health-related questionnaire for the pediatric field of Korean medicine: a Delphi study
- 48 Downloads
Although a variety of patient-reported outcome measures (PROMs) for children have been developed, there is no pediatric PROM specific to Korean medicine (KM) that is validated by experts in the field. The aim of this study was to collate the opinions of specialists in KM pediatrics on the development of a generic PROM that can be used by Korean medical doctors to assess the health status of children.
A three-round Delphi survey was conducted to determine the level of consensus on the development of a new PROM. Delphi questionnaires were sent by e-mail to 91 KM pediatricians on January 24, 2018. The Delphi questionnaire was composed of four sections: conceptualization, construction, items, and sources of content for a PROM. A nine-point Likert scale was used, and if more than two-thirds of the panels agreed or disagreed with a given sentence, they were considered to have reached a consensus. A draft of a PROM for the pediatric field of KM was developed in accordance with the preliminary conceptual framework.
Out of 91 experts, 18 finished three rounds of the Delphi survey. The experts reached a consensus on the necessity of a KM pediatric PROM for measuring various areas including child health, and using Likert scales with a recall period of 3 months. They also agreed on specific items and sources of content. A new draft of a health questionnaire for KM pediatrics was developed based on the Delphi consensus. It contains 44 items covering 7 domains: i) functions of the digestive system, ii) functions of the respiratory system, iii) mental functions, iv) skin functions, v) pain, vi) functions of the metabolic and endocrine systems, and vii) demographic details.
This research represents the first step in developing a health questionnaire for the pediatric field of KM. The questionnaire can be used in clinical and research settings after verifying several types of validity and reliability.
KeywordsPatient-reported outcome measure Patient health questionnaire Questionnaire development Pediatrics Child health Children Korean medicine Delphi study Consensus
Food and Drug Administration
Five Viscera Weak Children Questionnaire
International classification of functioning, disability and health: children and youth version
Korean medicine doctors
patient-reported outcome measure
traditional Chinese medicine
Korean medicine (KM) has certain similarities with traditional Chinese medicine (TCM) and Kampo medicine in Japan; however, it was developed with considerable independent and distinctive features . In the Republic of Korea, Korean medicine doctors (KMDs) and medical doctors obtain independent licenses and use different treatment methods. KMDs mainly use herbal medicine, acupuncture, moxibustion, and manual therapy for treating patients . In 2016, 23,460 licensed KMDs were practicing at 14,142 KM clinics or KM hospitals in the Republic of Korea. Each year, approximately seven hundred and fifty KM students complete the six-year university course and take the required examinations for obtaining the national license . The KM specialist training system was implemented in 1999 by the Ministry of Health and Welfare of Korea. After 4 years of training (one-year internship and a three-year residence course) in a KM hospital, KMDs are then qualified to take examinations for a specialist certification. The specialty subjects include pediatrics, gynecology, neuropsychiatry, ophthalmology-otorhinolaryngology-dermatology, acupuncture and moxibustion, internal medicine, rehabilitation medicine, and Sasang constitutional medicine . Pediatrics is one of the specialties of KM, and it deals with the physiopathological characteristics and the treatment of ailments in children and adolescents from a KM perspective . In 2016, there were 101 KM pediatricians in the Republic of Korea, and they accounted for 3.7% of all KMDs with specialist licenses .
One of the signature methods of approaching treatment in KM is to analyze, diagnose, and treat diseases from a holistic perspective. In KM, the human body is regarded as an organic whole, and when the internal organs are not functioning properly, it can manifest externally on the body surface. When Korean medicine doctors examine patients, they collect information about the whole body through four diagnostic methods including inspection, listening and smelling examinations, inquiry, and palpation; they then perform a comprehensive analysis of the signs and symptoms . In addition to the four diagnostic methods that are based on the physician’s subjective judgment, KMDs use various questionnaires as reference materials for collecting clinical information; this method enables patients or caregivers to properly outline the details of their child’s health condition . When a parent-reported questionnaire is used, herbal medicine treatment has been shown to be effective in improving stamina, appetite, digestion, and quality of sleep in children [8, 9]. In the case of TCM, clinical research evaluation guidelines recommend that researchers should evaluate aspects of a child’s overall health status such as digestion, appetite, urine, fatigue, sweat, or sleep, using TCM syndrome quantitative classification, in addition to the assessment of the main symptoms [10, 11, 12].
According to the US Food and Drug Administration (FDA), a patient-reported outcome measure (PROM) of treatment benefit is defined as data about a patient’s health condition that are reported directly by the patient without interpretation of the patient’s response by a clinician or anyone else . When assessing children’s health status, it is frequently measured by informants, such as parents, teachers, or other caregivers. When the PROM of very young children cannot be reliably measured because of their developmental stages or language ability, perspective of the informants may be useful for determining the condition of the children [14, 15, 16].
Although a variety of PROMs for children have been developed [17, 18, 19, 20, 21, 22, 23, 24], few studies have reported the development of a validated pediatric PROM that considers the overall health status of patients in a manner suitable for the unique methodologies of KM. When KMDs examine a pediatric patient, they identify the following items about systemic body functions, in addition to the main symptoms of the patient: sweat, heat and cold, headache, stomachache, eating, digestion, stool, urine, or sleep . Although PROMs for young children tend to focus on physical functioning [5, 18, 19, 23], other existing measures are applied to measure a health related quality of life based on the World Health Organization’s definition of health, which is “a state of complete physical, mental, and social well-being and not merely the absence of disease” [18, 19, 20, 21, 22, 23, 24]. Therefore, it is necessary to apply a PROM that includes the items that KMDs need to know about a child’s physical health and the indicators that are important in KM treatment.
Out of all the KM pediatric literature in the Republic of Korea, “Five Viscera Weak Children Questionnaire (FVWCQ)” is one of the most frequently used questionnaires for measuring a child’s health status [25, 26, 27, 28, 29, 30]. FVWCQ is a proxy-reported questionnaire containing from 5 [25, 26, 27, 28, 29] or 6  domains, and from 30  to 50 [25, 26, 28] or 55 [27, 29] items. It consists of items regarding circulatory, mental, neurological, digestive, respiratory, genitourinary, and metabolic problems. However, the FVWCQ has not been validated; items, domains, and response options are not equal to each other. There are certain challenges in answering questions that have several aspects in one item such as “sneezing, runny nose, and stuffy nose are frequent,” “the child has diarrhea or constipation,” or “the child has slow tooth development and frequently gets cavities.” Furthermore, there are questions that are confined to girls only, such as “In the case of a girl, is there vaginal discharge?” There are no indications for boys. In addition, there may be a methodological flaw in studies in which this questionnaire is used, since its reliability and validity have not been assessed [25, 26, 27, 28, 29, 30]. Moreover, the FVWCQ has been used without users’ consensus. There is no reason why a questionnaire for KM pediatrics should be made in the framework of the five viscera among various theories of KM.
The aim of this study was to investigate the level of consensus of experts on a new pediatric PROM that is suited to the methodologies of KM, that can ease the monitoring and evaluation of the benefits of KM treatments in children. The Delphi questionnaire included the levels of consensus among experts with regard to concept, structure, items, and sources of content for a KM pediatric PROM. This article reports the findings from the Delphi survey and the preliminary framework and draft of a new pediatric PROM.
The Delphi study
A three-round Delphi survey was conducted for this research. The Delphi method represents a relatively unbiased approach to decision-making method in health and social care and is used to determine the degree of consensus of experts on a topic . The method consists of repeated surveys and feedback from experts without face-to-face meetings . There is no restriction on the number of rounds that can be conducted , but two  or three rounds are most common ; the decision on the number of rounds required for any study is made at the discretion of the researcher . Anonymity is guaranteed and this is what makes the Delphi method different from other consensus methods such as the nominal group technique . Respondents can confirm their responses to the initial interrogation or reconsider their previous opinions. They may change their responses and can respond differently in light of replies from other respondents [32, 37]. Participants are provided with means, medians, standard deviation, and inter-quartile range, as well as collected opinions, as a reference for judgment .
The Delphi questionnaire was studied and examined by the research team. All of them are KMDs who completed six-year undergraduate courses and are specialists in KM pediatrics as well. One of them (CGT) is a professor of KM pediatrics at KM University with more than 20 years of clinical experience. The other two (LJ and LSH) are KMDs with more than 10 years of clinical experience.
Statements of the Delphi questionnaire
1. It is necessary to develop a standardized questionnaire that can be used by KMDs in KM treatment for children or in research.
2. The <KM pediatric questionnaire> should measure various areas constituting pediatric health.
3. The <KM pediatric questionnaire> should measure the body functions of children.
4. The <KM pediatric questionnaire> should measure activities and participation of children.
5. The <KM pediatric questionnaire> should be based on the <Five Viscera Weak Children Questionnaire>.
6. The <KM pediatric questionnaire> should provide a total score.
7. The <KM pediatric questionnaire> should provide a score for each area.
1. A < KM pediatric questionnaire> should be developed for children aged 1–5.
2. A < KM pediatric questionnaire> should be developed for children aged 6–9.
3. A < KM pediatric questionnaire> should be developed for children aged 0–20.
4. It is appropriate to use Likert scales when responding to the <KM pediatric questionnaire>.
5. It is appropriate to use visual analogue scales when responding to the <KM pediatric questionnaire>.
6. It is appropriate to use dichotomic (yes/no) response options when responding to the <KM pediatric questionnaire>.
7. It is appropriate to ask about the last 1 month.
8. It is appropriate to ask about the last 3 months.
9. It is appropriate to ask about the last 6 months.
10. It is appropriate to ask about the last 1 year.
How much do you agree that “the following items should be included in the questionnaire”? chills / cold hands and feet / hyperhidrosis / headache / dizziness and giddiness / arthralgia / chest discomfort / abdominal pain / vomiting / anorexia / thirst / diarrhea / constipation / frequent urination / sleeping disorder / fatigue / vitality / complexion / dry skin / frequent infections / rhinorrhea/nasal obstruction / epistaxis / being easily startled / anxiety / sensitivity
Sources of content
Sources of content
How important do you think the following materials are to the development of the questionnaire?
1. Focus group interview (child)
2. Focus group interview (parents)
3. Focus group interview (experts)
4. Existing pediatric PROM
5. KM pediatrics textbook
6. TCM pediatrics textbook
7. Conventional medicine pediatrics textbook
8. Articles using <Five Viscera Weak Children Questionnaire>
9. Articles on pediatric PROM
The research team targeted KM pediatricians and/or professors of KM pediatrics at KM universities in the Republic of Korea, because they mainly treat pediatric patients and will be the main users of the questionnaire. Participants were contacted via e-mail through the Society of Korean Medicine Pediatrics. Participants received full explanations of the study background, purpose, methods, and personal information protection in advance. Interested participants signed an informed consent form before joining the panel. The present study received ethical approval from the institutional review board of Kyung Hee University, Korean Medicine Hospital at Gangdong in the Republic of Korea (KHNMCOH 2017–12-008). The investigation adhered to the tenets of the Helsinki Declaration. Participants who did not submit the consent form were excluded from the study.
The appropriate number of the participants to be included in a Delphi study is not specified . The sample size was determined by taking into account the available expert resources and the scope of the issue [31, 41]. There was no consensus on the exact sample size; the range initially considered was from 15 to less than 50 . If the sample size in the Delphi study is too large, a low response rate and difficulties in the summarizing process can be a potential problem [31, 40]. Considering that the number of KM pediatricians was 101 in 2016 , and due to time constraints, we considered that 15–20 participants would be suitable .
In the first round, participants were given 2 weeks to complete the Delphi questionnaire. Participants were allowed to enter data into a Hangul program file (a word processor popular in the Republic of Korea for its specialized language writing abilities) so that they could be attached to an e-mail. If the subjects did not respond to or complete the questionnaire after more than a week, we considered that they were not willing to participate and made no attempt to contact them . In the second and third rounds, we considered the participants familiar with the study method and the content of the Delphi questionnaire, and gave them 1 week to complete the questionnaires of each round.
Data management and analyses
The authors determined whether consensus had been reached based on the extent to which experts agreed on a given issue. Participants were asked to rank agreements or disagreements with statements on a scale of 1 to 9. One meant “totally disagree” and 9 “totally agree.” In the case of section 4, participants were asked about the importance of the sources of content, of which 1 meant “not important at all” and 9 “absolutely important.” In the first round, respondents were provided with reference materials to ensure that they could make informed decisions. The authors analyzed the responses for consensus and repeated the same questionnaire in the subsequent rounds. The second and third rounds provided the results (mean, median, standard deviation, and additional comments from other experts) of the previous round.
In the Delphi method, the definition of consensus among the participants is not established. The researchers determine how to measure the degree of agreement and the cut-off used to define the consensus . Because the definition of the RAND group  is the most widely used, the authors adopted this definition. In a three-round survey, when more than two-thirds of the panels agreed or disagreed with a given sentence, they were considered to have reached a consensus. If they reached a consensus on a score of 7–9, it would be considered a “high agreement.” In case of an agreement score of 4–6, the author considered it a “middle agreement.” If they agreed on a score of 1–3, it was assumed a “low agreement.” If they failed to reach a consensus, it was considered “uncertain.” “Agreed” sentences were not presented in the next round [32, 37, 45].
Descriptive statistics were used and sociodemographic data were presented by frequencies (percentage) to describe the panelists. Microsoft Office Excel version 14.0 (Microsoft Corporation, Redmond, WA, USA) was used to analyze the results of the Delphi questionnaire.
Preliminary conceptual framework and the questionnaire draft
A conceptual framework defines the concepts measured by a PROM in a diagram that describes the relationship between domains and items . A preliminary conceptual framework based on the results of the Delphi consensus was developed. A questionnaire draft was designed based on this conceptual framework. In creating each question, the authors referred to articles on pediatric PROMs [15, 16, 18, 19, 20, 21, 22, 23, 24, 46]. A KM pediatrics textbook  and a conventional medicine pediatrics textbook  were also used as references, and were considered to be highly important by the respondents in the sources of content section in the Delphi survey questionnaire.
Sociodemographic characteristics of the Delphi panelists (n = 18) involved in the development of the patient-reported outcome measures for the Korean medicine pediatric field
Clinical experience (years)
Level of healthcare facility of institution he/she is currently affiliated to
Primary healthcare institution (Korean Medicine clinic)
Secondary healthcare institution (Korean Medicine hospital with 30 to 500 inpatient beds)
The Delphi result
Delphi – round one
Participant responses from part 1 to part 4 (round one)
In section 2, the panel highly agreed on the use of Likert scales when responding to the questionnaire (13/18 or 72.2%). In section 2, one respondent did not respond to four statements asking for feedback on the recall period; this is noted in Table 3. He/she wrote in the blank spaces provided instead; he/she suggested that it would be better to divide the recall period according to stage of childhood, such as infancy and early childhood. In the same way, four statements with this missing response were considered to have been agreed on when more than two-thirds of the 17 participants agreed. There was a strong consensus to inquire about the last 3 months (12/17 or 70.6%) and moderate agreement to inquire about the last 6 months (12/17 or 70.6%). The panel disagreed on seven statements, including statements covering the age range; these statements were presented again in the second round.
In section 3, a high level of consensus was reached on 13 of 25 items in the first round such as cold hands and feet, hyperhidrosis, abdominal pain, vomiting, anorexia, diarrhea, constipation, frequent urination, sleeping disorder, fatigue, frequent infections, rhinorrhea/nasal obstruction, and epistaxis. Agreement was not reached on 12 items.
In section 4, the panelists agreed that interviews (parents, experts) are highly important. Regarding the literature references, the panel agreed that the KM pediatrics textbook and conventional medicine pediatrics textbook have a high degree of importance. There was no consensus on a focus group interview with children, on the existing pediatric PROM, and on several references as well (TCM pediatrics textbook, articles using the FVWCQ, and articles on pediatric PROM).
Delphi – round two
Participant responses from part 1 to part 4 (round two)
Delphi – round three
Participant responses from part 1 to part 4 (round three)
Preliminary conceptual framework and draft of questionnaire
This study presents the results of the development of a PROM for KM pediatrics, which assesses a child’s health status in a manner specific to KM, using the Delphi method. The Delphi survey was conducted to investigate the consensus of experts on the new PROM. Results from the Delphi survey revealed that 17 experts (94.4%) thought that it was necessary to develop a standardized PROM for KM pediatrics. However, the experts could not reach an agreement as to whether a new PROM should be based on the FVWCQ, which was frequently used in the KM pediatrics field [25, 26, 27, 28, 29, 30]. When the authors asked the experts to give their opinions on the new PROM, four experts replied that the new questionnaire did not necessarily have to be in the same format as the FVWCQ, and after reviewing it, certain items may be used as needed. One expert replied that because the items of the FVWCQ were artificially developed to fit the five viscera, it was not necessary to refer to a new PROM. Because the FVWCQ has been used without expert consensus or validation [51, 52], if there are any useful items, they can be used restrictively only after critical review from experts.
Concerning the age range, the experts did not highly agree on any sentence in that regard. The experts suggested that separate questionnaires for all childhood age groups and adolescence are needed and should be developed to cover items according to the growth and development of each childhood and adolescence period. After a discussion session, the authors’ decision was to develop a questionnaire for children aged 1–5 years first, which according to a survey is the most common age group of all patients in the outpatient clinics of KM pediatrics [48, 49, 50]. It is known that there is no clear evidence of the validity or reliability of self-reporting measures in children younger than 5 years; it is necessary to use the observational reports of parents or other caregivers and special clinical measures when assessing the health status of children in this age range . PROMs covering children 5 years old or younger commonly provide proxy-reported versions [14, 17, 18, 21, 24]. Therefore, the authors developed a PROM that utilizes parent-proxy reported outcome measures. Furthermore, in order to fit the characteristics of a PROM for younger children (younger than 5 years), the questionnaire consisted of items regarding “body function” rather than “activities and participation” [18, 19, 22, 46].
Regarding response options, 77.2% of the experts agreed that the Likert scales are appropriate for a PROM for KM pediatrics. Likert scales are considered appropriate for a pediatric PROM and a considerable number of PROMs have applied this scale . In addition, a cognitive interview study established that a 5-point Likert scale is not difficult to understand even in the case of young children as young as 8 years old . In accordance with the recommendations of the International Society for Pharmacoeconomics and Outcomes Research guideline , a follow-up cognitive interview was conducted to determine whether a 5-point Likert scale is appropriate for the questionnaire.
The recall period was determined to be 3 months, according to the experts’ consensus (70.6%, high agreement). In the open-ended questionnaire, two experts suggested applying different recall periods for each item. The recall period is controversial because the longer the recall period, the more likely it is to recall errors. Therefore, possible errors should be investigated through future interviews with parents.
The questionnaire draft consisted of items the experts agreed on (i.e., cold hands and feet, hyperhidrosis, abdominal pain, vomiting, diarrhea) and items extracted from textbooks (halitosis, cough, snore, wheezing, short breath, night crying) based on the results of the Delphi study. For clear communication, the item terms and domain classifications followed the ICF-CY. The ICF-CY was developed to be used by clinicians, family members, and researchers to document characteristics of children’s health  and has been used to identify the properties of PROMs . For example, an item such as sleeping disorder that the experts agreed on in the Delphi study was divided into “difficulty falling asleep” and “waking up often during the night,” with reference to the ICF-CY.
Experts recommended a diversity of symptoms to be included. However, the purpose of this questionnaire was not to identify all the symptoms that may appear in children aged 1–5. The items of the questionnaire were intended to be the symptoms common to the physical functions of children and sensitive enough to detect change  according to KM treatment modalities. Items that a clinician should directly evaluate were excluded whereas items to be assessed by inquiry were included. Items that parents could directly observe and describe were also included. In addition, the authors focused on making it easy for symptoms to be checked at the clinics or in clinical researches. Items such as cognition, language, motor, or psychosocial development were not included in the questionnaire because these items were judged particularly important by the use of a disease-specific assessment tool and examination by a specialist.
The draft of the questionnaire will be modified through an iterative process into a more appropriate questionnaire, and then the next step is the content validation of the questionnaire. Content validity should first be established among several types of validities in the development of a questionnaire. Such validity is evidence that a PROM evaluates the concept of interest data from qualitative research, and that the domains and items are appropriate. Focus group interviews, individual interviews, or cognitive interviews are recommended for content validity .
The draft of the KM pediatric questionnaire is a generic measure of physical health of preschool children, which was developed as a monitoring and evaluation scale. It can be administered by specialists in the different fields of health, including KMDs. It can be used for healthy and ill children aged 1 to 5 years, and can be filled in by the parents. This questionnaire consists of 44 items covering 7 domains: i) functions of the digestive system (10 items), ii) functions of the respiratory system (10 items), iii) mental functions (9 items), iv) skin functions (3 items), v) pain (3 items), vi) functions of the metabolic and endocrine systems (4 items), and vii) demographic details (5 items). This questionnaire assesses the frequency of signs and symptoms. A 5-point Likert scale was applied on the formal items (0 means “never a problem”; 1 means “almost never a problem”; 2 means “sometimes a problem”; 3 means “often a problem”; 4 means “almost always a problem”). The recall period was 3 months. A higher score indicated that the child had many symptoms associated with physical functioning.
This questionnaire was drafted in accordance with FDA guidelines, based on expert agreement for the first time in the field of KM pediatrics . The questionnaire contains items considered important in KM that are not included in other PROMs, for example, items such as cold hands and feet, halitosis, hyperhidrosis when doing activity, and night sweat.
A potential limitation of this study was that only 18 experts were included in the Delphi survey. Because the number of experts was small, there may be concepts and items that were not sufficiently included. This is due to the nature of Delphi research, which requires anonymity and voluntary participation in the recruitment process . In addition, the nature of the research method did not allow free face-to-face discussions. However, the Delphi method is known to be a quick and reasonably easy way to carry out a survey, particularly via e-mail . Instead, there was a blank space for an open-ended answer in the Delphi questionnaire, and the participants were able to express their opinions freely. The authors collected it as a reference for the next round. However, there can be fewer exchanges of opinions in this method than with a direct meeting. In round 3, there was not much change in the opinions of the panels, and the experts agreed on 2 among the 19 statements. Another limitation is the narrow diversity of respondents. Of 18 panelists, 15 were professors of KM pediatrics at university, and they had expertise in the pediatrics of KM. However, they might view the questionnaire differently from clinicians who are not academics. There is a limitation that the opinions of KMDs working at the primary healthcare institutions have not been fully collected. Further limitation is that there is an unavoidable uncertainty that this PROM may have because it is a proxy-reported outcome measure. When a parent reports on a measure, the items are interpreted, and the informant intervenes in the subjective health status of a child .
In this study, the Delphi survey was used to construct the framework of a questionnaire for KM pediatrics. Consensus was reached by the Delphi panel on the concept and construction of the questionnaire. This is an important step in the development of a pediatric PROM for KM. The next step will be individual interviews and cognitive interviews with parents for evidence of content validity. Further support of validity and reliability by psychometric evaluation such as construct validity, internal consistency, and test-retest reliability is required. This questionnaire may be used as objective data in the medical examination of children in KM clinics, and the quality and reliability of the KM treatment may be improved through a validated assessment scale. In addition, once the validity and reliability of the questionnaire are established in the future, it may improve the quality of research in KM pediatrics, and enable better communication with healthcare professionals.
This research represents the first step in developing a PROM for KM pediatrics. The draft of the new questionnaire is a proxy-reported outcome measure and consists of 44 items and 7 domains, and a 3-month recall period. The content validity of this PROM will be confirmed later through interviews with parents; future psychometric evaluations of this PROM will make it a more reliable and valid questionnaire. If the PROM goes through a rigorous validation process, it will become a dependable method of accurately catching important and significant symptoms indicators in KM treatment. Moreover, it will be possible to get a better understanding of the health status of children.
The authors appreciate the Delphi panelists for valuable time for the development of the Korean Medicine pediatric questionnaire.
All authors constructed and revised the Delphi questionnaire. JL analyzed the data and wrote the manuscript. GTC and SHL reviewed and revised the manuscript. All authors read and approved the final manuscript.
This study was not supported by any source of funding.
Ethics approval and consent to participate
Ethical approval was obtained from the institutional review board of Kyung Hee University, Korean Medicine Hospital at Gangdong in the Republic of Korea (KHNMCOH 2017–12-008). The investigation adhered to the tenets of the Helsinki Declaration. Participants and signed written informed consent forms to participate, explaining the nature of study and participant’s rights.
Consent for publication
The authors declare that they have no competing interests.
- 2.Shin Y, Shin JS, Lee J, Lee YJ, Kim M, Ahn Y, Park KB, Shin BC, Lee MS, Kim JH, Cho JH, Ha IHA. Survey among Korea medicine doctors (KMDs) in Korea on patterns of integrative Korean medicine practice for lumbar intervertebral disc displacement: preliminary research for clinical practice guidelines. BMC Complement Altern Med. 2015;15:432.PubMedPubMedCentralCrossRefGoogle Scholar
- 3.2016 Yearbook of Traditional Korean Medicine. Korea institute of oriental medicine, 2018. https://www.kiom.re.kr/brdartcl/boardarticleView.do?menu_nix=WUNNW2Aq&brd_id=BDIDX_o9YEVvNb40b134N1Rt17aq. Accessed 6 Nov 2018.
- 5.Kim KB, Kim DG, Kim YH, Kim JH, Min SY, Park EJ, Baek JH, Sung HK, Yu SA, Lee SY, Lee JY, Lee HJ, Chang GT, Jeong MJ, Chai JW, Cheon JH, Han YJ, Han JK. Korean pediatrics & adolescent medicine. Ui Sung Dang: Seoul; 2015.Google Scholar
- 6.Choi SH. Korean medicine. Kyobo Book Center, 2016; http://bigbook.or.kr/bbs/bbs/board.php?bo_table=bo16&wr_id=26. Accessed 6 Nov 2018.
- 7.Hwang YJ, Kim KB, Min SY, Kim JH. Clinical investigation of survey and test in Korean oriental pediatrics. J Korean Orient Pediatr. 2008;22(1):25–34.Google Scholar
- 8.Paeck EK, Park JH, Park JH, Yoon YJ, Jeong SK, Lee HJ, Lim JS, Bahn GH, Moon JS. Influence of taking medicine for a period of time on the liver function and improvement of the development disordered children. Korean J Orient Physiol Pathol. 2006;20(2):494–7.Google Scholar
- 9.Bahn GH, Kim CJ, Chung JH, Kim YH, Paik EK, Park JH. A pilot study for developing an assessment scale for the effect of herbal medicine in healthy children: open-label study with Gami-Jiwhangtang. Korean J Orient Med. 2004;25(4):139–46.Google Scholar
- 10.Clinical Evaluation Unit, Pediatric Branch of China Association for Traditional Chinese Medicine. Guideline on evaluation of new drug of Chinese materia medica for treatment of recurrent respiratory tract infections in children. Drug Eval Res. 2015;38(3):238–43.Google Scholar
- 11.Clinical Evaluation Unit, Pediatric Branch of China Association for Traditional Chinese Medicine. Guideline on evaluation of Chinese medical research for treatment of anorexia in children. Drug Eval Res. 2015;38(3):244–50.Google Scholar
- 12.Clinical Evaluation Unit, Pediatric Branch of China Association for Traditional Chinese Medicine. Guideline on evaluation of Chinese medical research for treatment of eczema in children. Drug Eval Res. 2015;38(6):596–602.Google Scholar
- 13.US department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research Center, Center for Devices and Radiological Health. Guidance for industry: Patient-reported outcome measures: use in medical product development to support labeling claims, 2009.Google Scholar
- 14.Matza LS, Patrick DL, Riley AW, Alexander JJ, Rajmil L, Pleil AM, Bullinger M. Pediatric patient-reported outcome instruments for research to support medical product labeling: report of the ISPOR PRO good research practices for the assessment of children and adolescents task force. Value Health. 2013;16(4):461–79.PubMedCrossRefPubMedCentralGoogle Scholar
- 16.Varni JW, Limbers CA, Burwinkle TM. Parent proxy-report of their children's health-related quality of life: an analysis of 13,878 parents' reliability and validity across age subgroups using the PedsQL™ 4.0 Generic Core Scales. Health Qual Life Outcomes. 2007;5:2.PubMedPubMedCentralCrossRefGoogle Scholar
- 17.Janssens A, Thompson Coon J, Rogers M, Allen K, Green C, Jenkinson C, Tennant A, Logan S, Morris C. A systematic review of generic multidimensional patient-reported outcome measures for children, part I: descriptive characteristics. Value Health. 2015;18(2):315–33.PubMedCrossRefPubMedCentralGoogle Scholar
- 25.Lee S, Kim A, Jang H, Jeong M. Study on weak children information collection using personal health record (PHR). J Pediatr Korean Med. 2017;31(3):46–63.Google Scholar
- 26.Gok SY, Lyu SA, Lee SY. The correlation study between interference test with the weak children symptoms. J Korean Orient Pediatr. 2009;23(3):71–88.Google Scholar
- 30.Son MJ, Han JK, Kim YH. Oriental medical research about school life maladjustment by study of the weak children symptoms. J Korean Orient Pediatr. 2009;23(3):55–69.Google Scholar
- 31.Hasson F, Keeney S, Mckenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000. https://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x.PubMedPubMedCentralGoogle Scholar
- 36.Giannarou L, Zervas E. Using Delphi technique to build consensus in practice. Int J Bus Sci Appl Manag. 2014;9:65–82.Google Scholar
- 37.Herdman M, Rajmil L, Ravens-Sieberer U, Bullinger M, Power M, Alonso J, the European Kidscreen and Disabkids groups. Expert consensus in the development of a European health-related quality of life measure for children and adolescents: A Delphi study. Acta paediatr. 2002;91:1385–90.PubMedCrossRefPubMedCentralGoogle Scholar
- 38.Toupin April K, Stinson J, Boon H, Duffy CM, Huber AM, Gibbon M, Descarreaux M, Spiegel L, Vohra S, Tugwell P. Development and preliminary face and content validation of the "which health approaches and treatments are You using?" (WHAT) questionnaires assessing complementary and alternative medicine use in pediatric rheumatology. PLoS One. 2016;11(3):e0149809. https://doi.org/10.1371/journal.pone.0149809.CrossRefPubMedPubMedCentralGoogle Scholar
- 39.World Health Organization. International classification of functioning, disability and health : children & youth version : ICF-CY. 2007; http://apps.who.int/iris/bitstream/10665/43737/1/9789241547321_eng.pdf. Accessed 10 Nov 2018.
- 40.Hsu C, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess. 2007;12(Suppl 10):1–8.Google Scholar
- 42.Ludwig B. Predicting the future: have you considered using the Delphi methodology? J Ext. 1997;35:1–4.Google Scholar
- 44.Fitch K, Bernstein S, Aguilar M, Burnand B, LaCalle J, et al. The RAND/UCLA Appropriateness Method User’s Manual. https://www.rand.org/content/dam/rand/pubs/monograph_reports/2011/MR1269.pdf. Accessed 6 Nov 2018.
- 45.Kim SY, Choi MY, Shin SS, Ji SM, Park JJ, Yoo JH, Lyu DH, Park SH. Handbook for Clinical Practice Guideline Developer. National Evidence-based Healthcare Collaborating Agency. 2015. https://www.neca.re.kr/lay1/bbs/S1T11C102/F/39/view.do?article_seq=5347&cpage=1&rows=10&condition=&keyword=&show=&cat= Accessed 6 Nov 2018.
- 47.Ahn HS, Shin HY. Hong chang yee Pediatrics 11th ed. Mirae N;Seoul:2016.Google Scholar
- 49.Choi MH, Kin DG, Lee JY. A study on the chief complaint of pediatric outpatients in the Kyung Hee oriental medicine hospital. J Pediatr Korean Med. 2010;24(3):121–37.Google Scholar
- 50.Choi EY, Chang GT, Kim JH. A study of the chief complaint of pediatric outpatients in the oriental hospital. J Korean Orient Pediatr. 2005;19(2):197–213.Google Scholar
- 51.You TS, Kim YS, Bae JH, Kim DG. Literature review of the weak children. J Korean Orient Pediatr. 1990;4(1):67–78.Google Scholar
- 52.Pediatrics of Kampo medicine. Pediatrics of Kampo medicine. Pediatr Intern Med. 1984;1984:1.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.