Abstract
BACKGROUND AND PURPOSE: The ubiquitous use of the Internet by the public in an attempt to better understand their health care requires the on-line resources written at an appropriate level to maximize comprehension for the average user. The National Institutes of Health and the American Medical Association recommend on-line patient education resources written at a third-to-seventh grade level. We evaluated the readability of the patient education resources provided on the Web site of the American Society of Neuroradiology (http://www.asnr.org/patientinfo/).
MATERIALS AND METHODS: All patient education material from the ASNR Web site and the Society of Neurointerventional Surgery Web site were downloaded and evaluated with the computer software, Readability Studio Professional Edition, by using 10 quantitative readability scales: the Flesch Reading Ease, Flesch-Kincaid Grade Level, Simple Measure of Gobbledygook, Coleman-Liau Index, Gunning Fog Index, New Dale-Chall, FORCAST Formula, Fry Graph, Raygor Reading Estimate, and New Fog Count. An unpaired t test was used to compare the readability level of resources available on the American Society of Neuroradiology and the Society of Neurointerventional Surgery Web sites.
RESULTS: The 20 individual patient education articles were written at a 13.9 ± 1.4 grade level with only 5% written at <11th grade level. There was no statistical difference between the level of readability of the resources on the American Society of Neuroradiology and Society of Neurointerventional Surgery Web sites.
CONCLUSIONS: The patient education resources on these Web sites fail to meet the guidelines of the National Institutes of Health and American Medical Association. Members of the public may fail to fully understand these resources and would benefit from revisions that result in more comprehensible information cast in simpler language.
ABBREVIATIONS:
- AMA
- American Medical Association
- ASNR
- American Society of Neuroradiology
- CLI
- Coleman-Liau Index
- FKGL
- Flesch-Kincaid Grade Level
- FRE
- Flesch Reading Ease
- GFI
- Gunning Fog Index
- NDC
- New Dale-Chall
- NFC
- New Fog Count
- RRE
- Raygor Reading Estimate
- SMOG
- Simple Measure of Gobbledygook
- SNIS
- Society of Neurointerventional Surgery
Easy accessibility and a seemingly unlimited supply of information on the Internet have made it a frequently accessed resource by the public. In fact, almost 80% of Americans are regularly on-line and up to 80% of them consult the Internet for information on health-related topics.1,2 Patients and their families are most apt to seek Internet materials about a new diagnosis, the side effects of medications, a diagnostic or therapeutic procedure, or other treatment options.3 The Internet is often accessed both before and after an initial visit to a health care provider. Perhaps not surprising, patients tend to value the Internet information. In one study, it was reported as the second most important resource, superseded in value only by consulting information coming directly from the physician.4 As a result of the known importance of on-line health care information, many organizations have published Internet-based resources pitched specifically to patients.
However, by itself the delivery of health care–related information does not necessarily mean that the patient or his or her family will comprehend it. The American Medical Association has noted that the average American reads at only an eighth grade level, while those enrolled in Medicaid read at an even lower fifth grade level.5 Limited health literacy, in particular, can be a barrier to care, leaving patients with inadequate knowledge for making informed health care decisions.6 Several studies have shown that limited health literacy has been associated with poor understanding of relatively simple instructions.7⇓–9 These studies found that of those with limited health literacy, 26% did not know when their next appointment was scheduled, 42% did not understand what it meant by “take medication on an empty stomach,” and 86% did not understand the Medicaid application rights and responsibilities section.7⇓–9 A 2003 report from the American Medical Association (AMA) that evaluated the health literacy of adults found that those who did not graduate high school, those older than 65 years of age, Hispanic adults, black adults, those who did not speak English before starting school, those without medical insurance, those with disabilities, and prison inmates were more likely to have below-basic health literacy.10 In an effort to broaden the reach of patient education materials, the AMA and the National Institutes of Health have recommended that they be written at a third-to-seventh grade level.5,11
Despite these guidelines, many of the Web sites of several national physician organizations, including medical, surgical, and subspecialty fields, have provided texts at a level too complex for most of the public to comprehend.12⇓⇓⇓⇓⇓⇓⇓⇓⇓⇓–23 Recent reports that evaluate the readability of patient education resources on radiology Web sites, sponsored by major organizations such as the Radiological Society of North America, the American College of Radiology, the Society of Interventional Radiology, and the Cardiovascular and Interventional Radiologic Society of Europe, demonstrated that the material offered to the public is written at a level well above the AMA and the National Institutes of Health recommendations.24,25
In this study, we investigated the level of readability of all patient education resources on the American Society of Neuroradiology (ASNR) Web site by using a variety of quantitative readability-assessment scales. Additionally, we analyzed patient education resources from the Society of Neurointerventional Surgery (SNIS) Web site because the ASNR patient education Web site has links directly to the SNIS site.
Materials and Methods
In September 2013, all patient education material available on the ASNR (http://www.asnr.org/patientinfo/) and SNIS (http://snisonline.org/patient-center) Web sites were downloaded into Microsoft Word (Microsoft Corporation, Redmond, Washington). Copyright information, references, and images were removed from the text. The ASNR Web site had 17 articles, which were subdivided into 3 categories: neuroradiology, procedures, and conditions. The patient education on the SNIS Web site was directly referenced from the ASNR Web site and included 3 additional articles. All 20 articles were individually analyzed for their level of readability, with 10 different quantitative readability scales by using the software program Readability Studio Professional Edition, Version 2012.1 (Oleander Software, Vandalia, Ohio). The readability scales included the Flesch Reading Ease (FRE),26 Flesch-Kincaid Grade Level (FKGL),27 Simple Measure of Gobbledygook (SMOG),28 Coleman-Liau Index (CLI),29 Gunning Fog Index (GFI),30 New Dale-Chall (NDC),31 FORCAST Formula,32 Fry Graph,33 Raygor Reading Estimate (RRE),34 and the New Fog Count (NFC).27 The FRE readability scale reports scores from 0 to 100, with higher scores indicating more readable text (Table 1). The 9 additional readability scores report a number that corresponds to an academic grade level (On-line Table).
The FRE readability scale scoring system
The 9 readability scales that report scores corresponding to an academic grade level were compared with a 1-way ANOVA test. A Tukey Honestly Statistically Different post hoc analysis was conducted for all ANOVA results with a P < .05. Additionally, the patient education material available on the ASNR and SNIS Web sites was compared by using an unpaired t test. Statistical analysis was performed by using OriginPro (OriginLab, Northampton, Massachusetts).
Results
The FRE found that collectively, the 20 articles were written at a difficult (30–50) level, given its score of 34.9 ± 12.9 (Fig 1 and Table 2). The 9 other readability scales (CLI, NDC, FKGL, FORCAST Formula, Fry Graph, GFI, NFC, RRE, and SMOG) determined the readability of all 20 articles to be at the 13.9 ± 1.4 grade level. Individually, the 9 scales found the level of readability to range from the 11.4 (FORCAST Formula) grade level to the 15.5 (both the GFI and SMOG) grade level (Fig 1 and Table 2). Only one (5%) of the articles, which was on MRI, was written below the 11th grade level, and it was still difficult text to understand with a readability score at the 10.3 grade level (Fig 1 and Table 2). According to assessments using the NFC, CLI, NDC, and RRE scales, the material was written at the 12.4 ± 2.9, 13.2 ± 2.0, 13.8 ± 1.8, and 13.9 ± 2.5 grade levels, respectively (Fig 1 and Table 2). Analysis with the FKGL and Fry Graph revealed the patient education material to be even more difficult, with scores at the 14.1 ± 2.6 and 15.2 ± 2.0 grade levels, respectively (Fig 1 and Table 2). Only 3 articles (15%) were written at a level appropriate for viewers who had some level of high school education (10.3, 11.2, and 12.0 grade levels) (Table 2). Most of the articles, 65% (13/20), were written for a reader with some level of college education (12.3–15.0 grade levels), while 20% (4/20) were written for someone with a graduate level of education (16.2–16.9 grade levels) (Table 2). There was no statistical difference between the patient education materials found on the ASNR Web site and those provided by the SNIS Web site (Fig 2).
The readability of all the articles as measured individually by the 10 readability scales. The 9 other readability scales include the CLI, NDC, FKGL, FORCAST Formula, Fry Graph, GFI, NFC, RRE, and SMOG; and their grade level is measured on the left y-axis (Grade Level). The FRE score is measured on the right y-axis (FRE Score). The yellow area corresponds to the grade level recommendations from the National Institutes of Health and the AMA. FRE scores of 0–30 indicate the patient education resources are very difficult, 30–50 are difficult, 50–60 are fairly difficult, 60–70 are standard, 70–80 are fairly easy, 80–90 are easy, and 90–100 are very easy.
The level of readability of each patient education article as measured by the 10 individual readability assessment scales
The readability of all articles from both the ASNR and SNIS patient education Web sites as measured by the 9 readability scales that correspond to academic grade level (CLI, NDC, FKGL, FORCAST Formula, Fry Graph, GFI, NFC, RRE, and SMOG). The yellow area corresponds to the grade level recommendations from the National Institutes of Health and the AMA.
There was no statistical difference (P = .69) between the level of difficulty of the patient education material when comparing the ASNR and SNIS Web sites as assessed by the FRE readability scale. Furthermore, there was no difference (P = .27) between these Web sites as assessed by the 9 other readability scales. The 1-way ANOVA demonstrated a statistical difference in the average readability reported among the 9 readability scales, F (8,171) = 8.59, P = .0001. The Tukey Honestly Statistically Different post hoc analysis found real differences with the CLI (13.2 ± 2.0 grade level) and the GFI (15.5 ± 2.1 grade level) and SMOG (15.5 ± 2.0 grade level) scales. It also found differences among the FORCAST Formula (11.4 ± 0.5 grade level) and the Fry Graph (15.2 ± 2.0 grade level), GFI (15.5 ± 2.1 grade level), NDC (13.8 ± 1.8 grade level), RRE (13.9 ± 2.5 grade level), SMOG (15.5 ± 2.0 grade level), and FKGL(14.1 ± 2.6 grade level). Last, it found differences between the NFC (12.4 ± 2.9 grade level) and the GFI (15.5 ± 2.1 grade level) and SMOG (15.5 ± 2.0 grade level) and Fry Graph (15.2 ± 2.0 grade level) scales. Despite these statistical differences among some of the readability scales, all patient education resources available on the ASNR and SNIS sites were still written at advanced levels above the National Institutes of Health and AMA recommendations.
Discussion
Each of the patient education resources from the ASNR and SNIS Web sites failed to meet the AMA and the National Institutes of Health guidelines recommending a seventh grade level of readability. This finding held true regardless of the readability scale used. Furthermore, the readability assessment scales determined that the overwhelming majority, 85%, of the material was written at the college level or higher. The complex nature of the textual narratives provided by the ASNR and SNIS will likely hinder widespread understanding of the material by the public. Their narrative complexity may prevent effective transmission of health care information. These discrepancies in content versus intent are not a new phenomenon inasmuch as other Web sites have been shown to have a disjunction between the level of complexity of their patient education materials and the ability of the average reader to understand them.13⇓–15,20,23⇓–25 A recent article from JAMA Internal Medicine revealed that 16 major national physician organizations had Web sites that presented patient education materials above the AMA and National Institutes of Health guidelines.22 That study did note that one organization had marginally acceptable levels of readability of their resources (American Academy of Family Physicians), meeting the AMA and the National Institutes of Health guidelines on some but not all readability scales.22
Approximately 70% of patients have said that the on-line information they review has impacted their health care decisions.35 In an effort to better help these patients make informed health care decisions, it is critical to revise current on-line resources so they are in accord with the capabilities of the consumers of the material. The AMA, National Institutes of Health, and Centers for Disease Control and Prevention have developed instructional guidelines on various ways to compose patient education narratives at or below a seventh grade level.5,11,36 Web site developers may also benefit from consulting organizations such as the Institute for Healthcare Improvement (www.ihi.org) when developing patient education resources for the Internet.
Specific suggestions for improvement include the following: 1) Simple identifying methods, such as using bold type to emphasize major terms and categories, can facilitate patient comprehension.37 2) Using a font size between 12 and 14 points and avoiding all capital letters, italics, and nontraditional fonts can augment patient use of the Web site material.38 3) Avoiding the use of medical terminology, unless absolutely necessary, could improve patient comprehension as well.39 4) A major tool is the use of videos. A study on the implementation of videos for patients with breast cancer undergoing surgery showed that those who viewed educational videos, despite low education levels, lack of insurance, unemployment, and cultural diversity, were still able to score >80% on questionnaires.40 When constructing these videos, one should also tailor the message to patients by including an introduction addressing the purpose of the video and how it can personally assist in their decision-making processes.41 5) Furthermore, the use of stories and pictures, which are often more memorable than statistics and recommendations, can help alleviate patient fears and address their emotional states.42 Recent studies have demonstrated that pictorial aids enhance a patient's recall, comprehension, and adherence to treatments.43 For individuals who perhaps understand the basic information and desire more advanced resources, links to scientific information could be provided.44
While the textual materials on the ASNR and SNIS Web sites demonstrate a high level of readability, there are likely other factors that contribute to understanding of patient education materials. Perhaps most important, as mentioned above, the use of images and videos can enhance the material and likely improve the reader's comprehension.40⇓–42 The impact of multimedia needs to be further studied to determine its impact on patients. Additionally, terminology related to neuroradiology is often complex. This may lead to unavoidably high readability scores. There are important limitations of this study. It is still imperative to address the complexity of the current text to improve its potential for patient appreciation. Goals for future work include evaluation of real patients to determine their understanding and comprehension of such resources.
Conclusions
Patient education resources available on the ASNR and SNIS Web sites are written well above the AMA and National Institutes of Health guidelines. Such information should be revised to the third-to-seventh grade reading levels. To achieve greater comprehension by the average Internet viewer, modification of this material into a simpler, easier-to-read format, would expand the population who could benefit from the information related to neuroradiology, its diagnostic procedures, and common conditions in which neuroradiology plays a vital role.
Footnotes
An abstract for this paper was accepted at: Annual Meeting of the American Society of Neuroradiology, May 17–22, 2014, Montreal, Canada, under the title “Online patient resources—the readability of neuroradiology-based education materials.”
References
- Received October 1, 2013.
- Accepted after revision November 4, 2013.
- © 2014 by American Journal of Neuroradiology
