Abstract
Objective. To determine the quality and content of English language Internet information about juvenile idiopathic arthritis (JIA) from the perspectives of consumers and healthcare professionals.
Methods. Key words relevant to JIA were searched across 10 search engines. Quality of information was appraised independently by 2 health professionals, 1 young adult with JIA, and a parent using the DISCERN tool. Concordance of the website content (i.e., accuracy and completeness) with available evidence about the management of JIA was determined. Readability was determined using Flesch-Kincaid grade level and Reading Ease Score.
Results. Out of the 3000 Web pages accessed, only 58 unique sites met the inclusion criteria. Of these sites only 16 had DISCERN scores above 50% (indicating fair quality). These sites were then rated by consumers. Most sites targeted parents and none were specifically developed for youth with JIA. The overall quality of website information was fair, with a mean DISCERN quality rating score of 48.92 out of 75 (± 6.56, range 34.0–59.5). Overall completeness of sites was 9.07 out of 16 (± 2.28, range 5.25–13.25) and accuracy was 3.09 out of 4 (± 0.86, range 2–4), indicating a moderate level of accuracy. Average Flesch-Kincaid grade level and Reading Ease Score were 11.48 (± 0.74, range 10.1–12.0) and 36.36 (± 10.86, range 6.30–48.1), respectively, indicating that the material was difficult to read.
Conclusion. Our study highlights the paucity of high quality Internet health information at an appropriate reading level for youth with JIA and their parents.
- ADOLESCENT
- JUVENILE IDIOPATHIC ARTHRITIS
- INTERNET
- SELF-MANAGEMENT
- PATIENT EDUCATION
- QUALITY OF INFORMATION
Juvenile idiopathic arthritis (JIA) is the most common pediatric rheumatic disease, and is also a common cause of disability1. The disease affects many aspects of a child or adolescent’s life, and can negatively affect quality of life. Disease management is often complex, involving diverse therapies, requiring frequent medical monitoring and involving a team of healthcare providers2. Parents of children with JIA require information about the disease and its management in order to fully participate in their children’s care. Adolescents with JIA need sources of information as they move into assuming greater responsibility for their own healthcare decision-making and disease management3,4.
The Internet has emerged as an important medium for providing health information, as the digital divide shrinks with improved computer literacy and access across the population5,6. The use of the Internet to inform and influence healthcare, termed e-health, is applied to healthcare decisions to improve health management and produce better health outcomes7. However, reviews of health information on the Internet have found that many health sites contain inappropriate, inaccurate, and misleading information8–18. While misinformation is prevalent, a systematic review of harm associated with the use of health information on the Internet found few reported cases of harm (physical, emotional, or financial)19.
Standards for medical websites have been proposed, and areas of importance include currency, accuracy, completeness and consistency of information, accessibility of relevant content, and literacy level5,9,20,21. Several initiatives have been introduced to improve the quality of Internet-based health information. For example, organizational controls such as Health on the Net (HON) code provide a “stamp of approval” for websites adhering to agreed quality principles22. An alternative form of control is to place greater responsibility on the Internet user by providing access to tools such as the DISCERN tool (http://www.discern.org.uk), which provides consumers with a mechanism for appraising Internet health information, empowering them to recognize valid and reliable information23. Finally it has been suggested that healthcare professionals direct patients to reliable sites that they can access for accurate information about their condition7. While assessments of quality of Internet health information on arthritis in adults have been conducted24–27, there has been no systematic review of the quality, content, or readability of English language Internet information about JIA for parents and patients. Our goal was to explore the quality of information about JIA on the Internet.
MATERIALS AND METHODS
We searched the Internet for information relevant to JIA using a standardized search methodology and terms. We then used a multifaceted approach to evaluate sites by assessing the quality of information, the completeness and accuracy of the content, and the general readability of the information. Lastly, we compared the evaluations of a young adult with JIA, a parent, and healthcare professionals.
Identification of websites
Figure 1 summarizes the study process. A key word search of the Internet was performed in July 2007. Key words were selected to represent terms that consumers may use when searching for information about JIA. Key words used in the search were: juvenile idiopathic arthritis, juvenile rheumatoid arthritis, juvenile chronic arthritis, pediatric arthritis, kids arthritis, and arthritis in children. JIA is the new classification system proposed by the International League of Associations for Rheumatology to categorize the different types of arthritis in children28, which was previously referred to as juvenile rheumatoid arthritis (JRA) or juvenile arthritis. We did not use juvenile arthritis as a search term, as these words are assumed in JIA and JRA and thus produced similar results. The appropriateness of these search terms was confirmed after a cursory search identified relevant websites. These key words and the selected 10 search engines (Google, Yahoo, Excite, Altavista, AOL search, AskJeeves, Hotbot, Looksmart, Lycos, MSN search) were identified as being commonly used by patients and families in our previous research exploring self-management needs of adolescents with arthritis when they search for JIA information on the Internet3. The first 50 Web pages returned by each search engine for each key term were examined (n = 3000), as individuals rarely go beyond 10 pages when sourcing online information29. Two undergraduate nursing students (HH, CL; searched 5 search engines each) and 1 medical student (LC; searched 10 search engines) independently conducted the searches.
Websites were considered relevant if they provided sufficient information to answer the question, “How is JIA diagnosed and treated?” Websites were excluded if (1) they were already examined under a different uniform resource locator (Website URL); (2) were presented in a language other than English; (3) denied direct access through password requirements or repeated server unavailable; (4) had no human data or pertinent information; (5) were a library, portal, or gateway; (6) sold arthritis therapies without providing any information on the topic itself; or (7) were designed for social networking or personal journaling (blogs), as the quality of information on such sites is difficult to assess and control. While library portals are an excellent source of information, we found that few patients and parents used them when seeking out health information on the Internet3.
Assessment of website quality
The quality of the website information was independently assessed by 2 undergraduate nursing students (HH, CL) using the DISCERN tool (http://www.discern.org.uk/)23. The DISCERN tool was designed to help consumers appraise health information on the Internet and has evidence of reliability and validity30–32. The tool examines website information against 15 key criteria: the first section (questions 1–8) evaluates the reliability of the information (trustworthiness of information) and the second section (questions 9–15) considers the quality of the information on treatment choices. Each item is rated on a 5-point Likert scale. A summary score is generated by summing the scores for each item and ranges from 15 to 75, with low scores indicating poor quality and high scores indicating good quality23. We set a minimum cutoff DISCERN score of > 50% (37.5/75) for websites to be further evaluated by 2 consumers [young adult with JIA (JLB) and a parent (DP)] as we did not want to burden them by rating all 58 unique sites found. Using the total DISCERN score, the websites were grouped into categories of excellent (63–75), good (51–62), fair (39–50), poor (27–38), and very poor (15–26)15. Each website was categorized as for-profit, not-for-profit, government, academic, or unidentified. The target audience of the website (consumer, health professional, or both) was also extracted and if it was not explicitly stated, classification was determined by consensus among raters.
Assessment of website content
The completeness of the content of the websites was independently assessed by 2 undergraduate nursing students (CL, HH) and 2 experienced pediatric rheumatologists (LT, AH). Content completeness was rated on 16 criteria based on materials that have recently been recommended to be covered to promote self-management of JIA in adolescents (see Table 4)3. The content of each website was also evaluated for accuracy by the pediatric rheumatologists. Each reviewer was asked to rate the accuracy of the information on the website, using a scale of 1 to 414; a rating of 1 represents that the reviewer agreed with less than 25% of the information, 2 represents 26% to 50% agreement, 3 represents 51% to 75% agreement, and 4 represents > 75% agreement.
Assessment of website readability
Literacy level of the reviewed websites was determined by calculating the Reading Ease Score and Flesch-Kincaid grade level. The Flesch-Kincaid grade level rates text on US school grade levels or year (range 3rd–12th grade) and it is based on average sentence length and number of syllables per word. A score of 8.0 or less is the recommended level for developers of patient education materials31,33. The Reading Ease ranges from 0 to 100, with a lower score being more difficult to read than a higher score. In our study, readability was tested on excerpts from the treatment section of each website (e.g., nonsteroidal antiinflammatory drugs), since most health consumers are interested in learning more about treatments3. Microsoft Word® was used to calculate the readability statistics. Given that the Flesch-Kincaid grade level only goes to grade 12, we also calculated the SMOG score, which is valid from grade 3 to grade 19 (postgraduate degree). Calculating the SMOG score consists of taking a sample of sentences from the text and calculating a score based on number of syllables per word, and can be performed online at http://www.harrymclaughlin.com/SMOG.htm.
Data analyses
Overall search engine efficiency was defined as the number of unique and relevant websites as a percentage of the overall number of websites returned. The data from DISCERN and content (completeness and accuracy) ratings were coded and entered into Microsoft Excel® and analyzed using SPSS, version 16.0 (Rel. 11.0.1., 2001; SPSS Inc., Chicago, IL, USA)34. These data were analyzed to determine measures of central tendency and the distribution of values. Intraclass correlation coefficients (ICC) were calculated to determine interrater reliability within (e.g., between 2 healthcare professionals) and between groups (e.g., between healthcare professionals and consumers). A p value < 0.05 was considered significant.
RESULTS
Using the standard search terms, a total of 3000 Web pages were retrieved, with only 58 unique websites containing information on the diagnosis and management of JIA (2%). Of the 58 individual, relevant websites identified, only 16 (28%) had quality scores above 50% on the DISCERN measure. Of the 16 sites evaluated, 6 were classified as for-profit, 7 as not-for-profit, 2 government, and 1 as academic. The majority (88%) of sites targeted parents of children with JIA (Table 1). None of the sites was specifically developed for youth with JIA. Accreditation of the websites was also examined, and only 6 of the 16 (37.5%) were accredited by HON and/or Utilization Review Accreditation Commission.
The overall mean DISCERN scores (for all 4 raters) for each website are shown in Table 2. No website received an excellent rating. Two-thirds of the sites were rated as fair (n = 10; 59.8%), and 35.3% (6 sites) were rated as good. The overall quality of the website information was fair, with the average DISCERN quality rating score being 48.92 (± 6.56, range 34.0–59.5). The highest quality site was from a government agency (http://www.bchealthguide.org) and the lowest was a not-for-profit site (http://www.kidsarthritis.org.au). The 3 websites with the highest DISCERN ratings were http://www.bchealthguide.org, http://www.webmd.com, and http://www.emedicinehealth.com, with average DISCERN scores between 55.5 and 59.5, indicating good quality.
The mean DISCERN item quality ratings by type of website are outlined in Table 3. The majority of DISCERN items were rated above a score of 3 out of 5, indicating a moderate quality rating. However, 4 of the 15 items were rated below a score of 3, indicating poor quality. These items included clarity in the source of information used to compile the publication, when the information was produced (or currency), description of outcomes if no treatment utilized, and provision of support for shared decision-making. On average the websites with higher DISCERN ratings were the 1 academic site and the 2 government sites, and the lower ratings were associated with for-profit and not-for-profit websites.
The average content ratings for each website are shown in Table 2. The overall completeness of websites in terms of the JIA content was of moderate quality, with a mean completeness score of 9.07 out of 16 (± 2.28; range 5.25–13.25). Table 4 shows the mean ratings for website content by item, and compares content by type of website. The websites associated with the highest content ratings were the not-for-profit sites and the lowest were the for-profit sites. The items most frequently present were description of the types of arthritis, how arthritis is diagnosed, and treatment, specifically medications. Items relating to psychological treatments, lifestyle issues, transition to adulthood, and transition to adult healthcare were not mentioned in most instances. There was excellent agreement as determined by ICC between the 2 undergraduate nursing students (ICC ranged from 0.83 to 0.99) and the rheumatologists (ICC ranged from 0.82 to 0.99), as well as between the nursing students and rheumatologists (ICC 0.86–0.99) on content rating scores across the websites.
The overall accuracy score of the 16 websites was 3.09 (± 0.86) and ranged from 2 to 4. Out of the 16 websites, 6 (37.5%) had high accuracy ratings, and 5 (31.3%) had low accuracy ratings (Table 2). Mean accuracy scores were highest for the not-for-profit websites at 3.36 (± 0.94) and lowest for the for-profit sites, 2.83 (± 0.75). Percentage agreement between the 2 raters on accuracy of websites was 75%.
None of the websites met the criteria that it be understandable to consumers with no more than an 8th-grade level. The average Flesch-Kincaid grade level was 11.48 (or 11th grade) with a standard deviation of ± 0.74 (range 10.1–12.0); 10 sites (62.5%) scored at the maximum readability of 12.0. The mean Reading Ease score was 36.36 (± 10.86; range 6.30–48.1), which would indicate that the website material was difficult to read. No site had a score between 61 and 70, which would be the standard reading level, indicating the material was easy to read. Further, the average SMOG score was 13.21 (± 1.31; range 10.64–16.04), highlighting that most sites were actually at a college level (or similar to reading The New York Times).
DISCUSSION
To our knowledge, this is the first study that evaluated the content, quality, and readability of English language JIA information on the Internet from the perspectives of consumers and healthcare professionals. Consumers invest more time and resources to access health information than any other type of information available online35, and parents commonly use the Internet to obtain information about their children’s health36. However, our study showed that there is a paucity of accurate, high-quality, and easy-to-read information, and the information is difficult to find. There have been reports of similar low yields for quality health information for other pediatric chronic illnesses12,15.
The Arthritis Foundation (http://www.arthritis.org/) and The Arthritis Society (http://www.arthritis.ca/) websites are commonly used by parents of youth with JIA3. While these 2 sites were identified, they received a DISCERN score just below the 50% cutoff mark for further evaluation by consumers. In addition, the Printo website (http://www.printo.it/pediatric-rheumatology/information/UK/ondex.htm) is another excellent source of health information for patients and their families; however, this website did not come up in our search strategy as it does not contain the key search words. Therefore, it is important when developing health information for patients and families that the website includes key terms that are familiar to the audience being targeted; or healthcare professionals need to direct consumers to these high-quality sites.
Youth with JIA may be more likely than parents to look to the Internet for information and support5,6. In our survey of JIA websites, very few addressed topics uniquely important to youth, such as the effect of JIA and medications on lifestyle choices and transition to adult healthcare. Specific information that would be useful to youth in promoting self-management of their JIA and how to transition to adult centers was a particular deficiency of most websites reviewed3,4.
In our study, the 1 academic and 2 government websites on average had higher DISCERN ratings compared to for-profit and not-for-profit sites. The lowest completeness and accuracy ratings were seen on the for-profit sites. Almost half (44%) of the sites were found to be for-profit, making it difficult to rule out the possibility of biased information on these sites. In a similar study of the quality of arthritis information for adults on the Internet, Ansani and colleagues found that government and academic websites had higher quality ratings than other types of sites26. Parents and youth may not consider the source of a website when searching and reading health information on the Internet, and clearly the source of the information is a predictor of quality.
Readability is an important aspect of Internet information, if the information is to be useful to large numbers of patients. Consistent with other studies15–18, we found that the reading level of the websites we reviewed was significantly higher than the predicted reading skill of many potential readers. A study of the general readability of Internet pediatric patient education material designed for parents also found that the material had an average Fry Formula reading level of grade 12, much higher than what would be recommended for the average adult36. The accessibility of language used on a website is also important. Sim, et al, in a survey of Internet information about pediatric surgical conditions, found that although 94% of parents found the Internet useful for accessing information, many parents found the information too difficult to read and highly technical37. It has been proposed that pediatric educational material should be aimed at a grade 6–8 level or lower, and that websites should post the reading level of their material as well as providing glossaries of common medical terms33.
Identifying the websites with accurate, complete, and potentially helpful information about JIA from among the many sites with uninformative or misleading information was challenging for our group of healthcare professionals. Unfortunately, most individuals are not aware of the characteristics that indicate the quality of information when they are searching the Internet14. Users of the Internet tend to explore only the first few links on general search engines when seeking health information, and rarely check out the “about us” sections of websites to find out who the authors or owners of the site are, or to read disclaimers or a disclosure statement25. Based on our results, we feel that it is of critical importance in our roles as health professionals to educate youth and parents regarding how to find and recognize valid health information on the Internet. Generally accepted guides for assessment of health information include indicators such as: (1) clear statement of the purpose and target audience for the website; (2) information about authorship, disclosure, and attribution; (3) citation of best evidence; (4) acknowledgment of risks, other treatments, and the effect of no treatment; (5) references to other sources; and (6) clear disclosure of sponsorship, affiliations, or conflicts of interest9,20,25.
There are several limitations of our study. First, the websites evaluated were retrieved from matches on 10 different search engines at one point in time, and may not be representative of matches with other search engines or searches done at other times. However, we used common search engines that were identified by patients and parents in our previous research on developing Internet self-management interventions3, and selected search terms seemed likely to be similar to what parents or youth seeking information might use. Further, while we could have used metasearch engines to save time by searching multiple search engines at one time, we did not use this approach as patients and families tend not to use this strategy when looking for health information on the Internet3. Second, there is no consensus regarding the best tool to evaluate the quality of websites from the perspective of the consumer. The DISCERN tool has been used in other studies and by both healthcare professionals and consumers15,23,30, and therefore has some level of validity. The consumer participants in our study were both university educated, and thus are not representative of the typical consumer population. Further, this tool does not evaluate the usability of the website, which is an important criterion to ensure that the site is easy to navigate. The ICC between healthcare providers and consumers would suggest that the sites were reliably rated, with minimal interobserver variability. Finally, we limited our search to the English language. Given that the Internet is international and multilingual, we would encourage similar evaluations in other languages.
Our study highlights the paucity of high-quality information at an appropriate reading level for youth with JIA and their parents. We also showed that the consumers in this study were able to use a quality tool to rate Internet information to help them make decisions about the information found; however, some patients may not have the background or ability to do this. Therefore, pediatric rheumatology educational materials should be developed to assist patients and parents in the assessment of the information they are accessing on the Internet. Last, there is a need to develop high-quality, easily accessible, current information about JIA for parents of children with JIA. In addition, information targeted to youth with JIA addressing their specific needs should be developed in order to promote self-management and transition to adult healthcare.
Footnotes
-
Supported by the Canadian Arthritis Network and The Arthritis Society. Dr. Stinson’s research is supported by a Canadian Institutes of Health Research (CIHR) Post-Doctoral Fellowship, Canadian Child Health Clinician Program, and the Canadian Arthritis Network. Dr. Tucker was supported by a Clinical Investigatorship from the Child and Family Research Institute, BC Children’s Hospital. H. Harris was supported by a CIHR Musculoskeletal Summer studentship. C. Lin was supported by a CIHR Summer Studentship in partnership with Lawrence S. Bloomberg Faculty of Nursing. Dr. Cohen was supported by the Canadian Rheumatology Association Abbott Summer Studentship.
- Received June 15, 2009.
- Accepted for publication March 2, 2009.