DADOS-Survey: an open-source application for CHERRIES-compliant Web surveys
© Shah et al; licensee BioMed Central Ltd. 2006
Received: 13 June 2006
Accepted: 15 September 2006
Published: 15 September 2006
The Internet has been increasingly utilized in biomedical research. From online searching for literature to data sharing, the Internet has emerged as a primary means of research for many physicians and scientists. As a result, Web-based surveys have been employed as an alternative to traditional, paper-based surveys. We describe DADOS-Survey, an open-source Web-survey application developed at our institution that, to the best of our knowledge, is the first to be compliant with the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). DADOS-Survey was designed with usability as a priority, allowing investigators to design and execute their own studies with minimal technical difficulties in doing so.
To date, DADOS-Survey has been successfully implemented in five Institutional Review Board-approved studies conducted by various departments within our academic center. Each of these studies employed a Web-survey design as their primary methodology. Our initial experience indicates that DADOS-Survey has been used with relative ease by each of the investigators and survey recipients. This has been further demonstrated through formal and field usability testing, during which time suggestions for improvement were incorporated into the software design.
DADOS-Survey has the potential to have an important role in the future direction of Web-survey administration in biomedical research. This CHERRIES-compliant application is tailored to the emerging requirements of quality data collection in medicine.
The Internet has been increasingly utilized in biomedical research. From online searching for literature to data sharing, the Internet has emerged as a primary means of research for many physicians and scientists. More recently, Web-based surveys have been employed as an alternative to traditional, paper-based surveys. Several studies have demonstrated that Web-surveys have comparable and, in many instances, facilitate higher-quality results than traditional surveys [1–4]. Notably, Web-based surveys have several advantages: speed of data collection, automated data entry, and less potential for human errors, to name a few.
However, the use of Web-based surveys in the biomedical sciences presents several inherent challenges. From issues regarding anonymity of survey participants , generality and validity of the data, and multiple responses from a single participant, Web-surveys present challenges for researchers who choose to use them . Recently, the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) was formulated to address many of these concerns . Similar to the QUORUM guidelines for systematic reviews and CONSORT statement for randomized trials, the CHERRIES statement is the first checklist for ensuring the quality of biomedical research that utilizes Web-survey methodology. The CHERRIES statement has been described in detail elsewhere . In short, CHERRIES mandates that results from Web-surveys are reported with the following issues adequately addressed: survey design, Institutional Review Board (IRB) approval and informed consent process, development and pre-testing, recruitment process and description of the survey participant sample, survey administration, response rates, prevention of multiple entries from the same individual, and data analysis.
In this article, we describe DADOS-Survey, the first-ever open-source Web-survey software application to, the best of our knowledge, fully integrate the CHERRIES guidelines into its design. DADOS-Survey provides Web-survey administrators with the necessary means to collect and report data that is compliant with CHERRIES. Although multiple Web systems exist for collection of survey data using customizable forms (Survey Solutions Express, Inquisite, SurveyWriter, SelectSurveyASP, Vista, all last accessed February 2006), to our knowledge, no system has been designed to readily ensure CHERRIES compliance. We believe that DADOS-Survey addresses the specific concerns regarding the use of Web-surveys in biomedical research. We outline below the design objectives, software architecture, implementation, usability, and future directions for DADOS-Survey.
Design Objectives for DADOS-Survey
Ensuring that software functions match the user's needs, while maintaining compliance with CHERRIES guidelines
Users should be readily able to locate and understand which functions are available
Users should be able to easily navigate through the software website
Users should be able to provide rapid feedback to website administrators for constant improvements
The more simple the application, the greater its utilization
Within the context of these goals, we outlined several specific objectives before designing DADOS-Survey:
1. The program should be fully compliant with the CHERRIES Web-survey guidelines.
2. Non-anonymous surveys, by definition, contain protected health information (PHI) as specified by the Health Insurance Portability and Accountability Act (HIPAA), and thus require informed consent. The user should therefore have the ability to provide informed consent for non-anonymous surveys.
3. The program should be able to deliver anonymous surveys, not containing PHI, and therefore not in need of informed consent.
4. The program should have a mechanism to prevent entry of non-invited respondents for anonymous and non-anonymous surveys.
5. The user can deliver personalized e-mails to potential respondents, including the name of the principal investigator as the sender. Potential replies to e-mails should be captured and stored by the server.
6. The program should be able to record the exact date and time when each respondent answered the survey.
7. The Web-survey should be able to randomize the item sequence of a survey form. Item randomization avoids framing of item responses by precedent items.
8. The user should be able to determine the number of items per page, thus avoiding long pages or different pages with an unequal number of items.
9. Survey participants must perform a "completeness review" on the last page of the survey, giving respondents the opportunity to review and change their responses.
10. For non-anonymous surveys, respondents should be tracked through their login and password. Individuals who have already responded should be prevented from responding again.
11. The application should be able to count survey responders when they arrive at the first page of the survey, e-sign the informed consent, and submit the survey, all while determining the time between each of these activities. This information will be used to validate survey results and facilitate sensitivity analyses.
12. The application should be able to include pictures and flash movies.
13. The application should be able to collect private information data in a secure (HTTPS) environment in compliance with HIPAA regulations
For installation, Java >= 1.4.2 is required for APACHE Tomcat 5.X to run and as backend, MySQL 5.0.
Explanation of Survey Management Toolbar Options
Modify the Web-survey form
Import e-mail sheet
Import a Microsoft Excel list of e-mail addresses for survey recipients (a template is provided for users to download also)
Create the recipient e-mail that is sent with each survey URL. This e-mail contains an automatically-generated, customized greeting based on the e-mail list, while indicating the PI as sender of the e-mail
Extract real-time data collected from the survey, downloadable in Microsoft Excel format
Once the survey is created, the e-mail list can be uploaded and the body of the recipient e-mail can be defined. The user is then prompted to create the waves, or set automatic distribution dates, for the survey to be sent. These waves are automatically sent on the dates defined by the PI, without any additional intervention required.
Explanation of Options for Questions in Survey
Instructions or label for a group of questions
Free-text response that requires response of one line or less
Free-text response that requires more than one line
Free-text numeric response
More than one alternative
Response that can include more than once selection of user-defined alternative
Only one alternative
Response that is limited to only one user-defined alternative
Response that is limited to date format
Response that is limited to time format
As Web-survey participants generate data, it is stored on a password-protected, encrypted server at our institution. At any time during the Web-survey study, the PI is able to extract real-time response data that is compiled in a downloadable Microsoft Excel (Microsoft Corporation, Redmond, WA) spreadsheet.
To date, DADOS-Survey has been successfully implemented infive IRB-approved studies conducted by various departments within the Duke University Health System. Each of these studies employed a Web-survey design as their primary methodology. Our initial experience indicates that DADOS-Survey has been used with relative ease by each of the investigators
Usability testing was performed in accordance with previous novel software applications from our institution . As part of formal usability testing, ten physician-researchers were randomly selected from a departmental directory from the authors' two institutions. Prior to using the application for the first time and subsequently completing the usability survey, each investigator was given a 20-minute, live tutorial by one of the authors (AS) that described the application and all of its functions. These volunteers were first asked to use the program to design a survey and provide their anonymous feedback regarding their experience. In the next phase, these same ten researchers were the recipients of a non-anonymous Web-survey designed and launched using DADOS-Survey. This survey required the researcher to read and agree to an informed consent page. None of the researchers reported any issues with login, informed consent, or answering the brief survey.
Formal Usability Testing: Survey Questions and Responses
DADOS-Survey speed is excellent.
DADOS-Survey is extremely easy to learn
DADOS-Survey is extremely easy to use
It is very easy to understand all functionality available within DADOS-Survey (e.g., download files, upload files, etc)
The navigation in DADOS-Survey is highly intuitive
2. Field usability
The first four months of field usability were intended to identify and correct minor errors in the software source code, including navigational difficulties, non-functional URL address links, and the improvement of a consistent visual interface throughout the program. This phase of usability testing focused primarily on soliciting feedback and comments from research investigators from the first two research studies that utilized DADOS-Survey. This process allowed for early improvements, as research investigators brought to attention operational issues that could be addressed by software programmers. In doing so, both survey designers and recipients were asked to provide written feedback to one of the authors (MM). Specific feedback from users indicates that DADOS-Survey is "intuitively created" with "no problems" reported in completing the Web-survey design. Two investigators requested that "image capabilities be expanded to include MPEG video file formats."
Compliance with CHERRIES guidelines for Web-surveys
Detailed summary of DADOS-Survey compliance with CHERRIES
CHERRIES Item Category
CHERRIES Checklist Category
Describe survey design
√ Described in the recipient e-mail invitation
IRB approval and informed consent process
√ Information provided in the recipient e-mail invitation and first page of survey
Development and pre-testing
Development and testing
√ Described in the recipient e-mail invitation
Recruitment process and description of the sample having access to the questionnaire
Open survey versus closed survey
√ User selectable option based on IRB approval from host institution
√ Investigator must report
Advertising the survey
√ Survey can be posted on the Web or sent over e-mail. Responses are automatically captured by DADOS
√ Investigator must report
√ Investigator must report. Time/Date data is automatically captured by DADOS.
Randomization of items or questionnaires
√ DADOS is user configured to randomize/alternate items
√ DADOS is user configured to conditionally display items
Number of items
√ DADOS automatically captures and reports this data
Number of screens (pages)
√ DADOS allows user to configure a completeness check based on Java Script. A non-response option is provided to participants. One response and multiple response options are also available.
√ Review step allows participants to alter their responses before submission
Unique site visitor
√ DADOS captures IP address to determine unique site visitors
View rate (Ratio unique site visitors/unique survey visitors)
√DADOS computes this information and provides user with the results
Participation rate (Ratio unique survey page visitors/agreed to participate)
Completion rate (Ratio agreed to participate/finished survey)
Preventing multiple entries from the same individual
For non-anonymous surveys, a user can only answer a survey once. For anonymous surveys, if the user does not complete the survey at once, the data collected is not saved.
This was decided not to be implemented so that different users could use the same computer to answer the survey
Log file analysis
The registration is done by importing a spreadsheet with the subjects. Once a subject completes the survey he/she cannot log in using the same user again.
Handling of incomplete questionnaires
All surveys are analyzed, both completed and uncompleted; uncompleted questions are understood as missing.
Questionnaires submitted with an atypical timestamp
DADOS-Survey has been successfully implemented at our institution in a number of studies. Orthopedic specialists have used the software application in Web-surveys of surgeons to assess national and international practice variation in orthopedic surgical procedures. Several Web-surveys examining patient satisfaction are underway, both with and without private health information contained within the survey. Lastly, faculty members in the Department of Radiology at our institution have used DADOS-Survey to design inter- and intra-observer agreement studies using high-resolution clinical images.
We have demonstrated that DADOS-Survey has the potential to have an important role in the future direction of Web-survey administration in biomedical research. This CHERRIES-compliant application is tailored to the emerging requirements of quality data collection in medicine. Of note, the CONSORT guidelines have had a drastic impact on the design, implementation, and reporting of randomized trials [10–12]. Similarly, the authors feel that compliance with CHERRIES requires that its guidelines be implemented at the design stage of a Web-survey. DADOS-Survey allows for such compliance in its software design. By allowing the user to create and distribute CHERRIES-compliant surveys in minimal time, it is expected that the free distribution of DADOS-Survey will make it an efficacious and attractive alternative to traditional paper surveys for researchers worldwide to conduct studies compliant with the recently developed CHERRIES data reporting guidelines.
DADOS-Survey has several advantages over existing Web-based survey applications (Survey Solutions Express, Perseus Development Corporation, Braintree, MA; Inquisite, Austin, TX; Vista, Vanguard Software Corporation, Cary, NC). To the best of our knowledge, this is the first freely distributed open-source application that incorporates compliance with the CHERRIES data reporting guidelines into its design. In contrast to the commercial packages, which do not permit modification of the source code, we are confident that the modifiable source code of DADOS-Survey will encourage users to customize and further improve the features in this release. Importantly, DADOS-Survey can be installed on any academic institutional server, allowing users to securely store and manage data at their own institutions. One disadvantage of DADOS-Survey over commercial survey applications is that users will likely need to consult their institutional Information Technology division for assistance in installation and in modifying the source code.
1. DADOS-Survey allows researchers to develop randomized experiments on the Web in which different questions are assigned to different subjects. An example includes the presentation of statements about drug efficacy while randomly choosing whether or not to introduce a picture of a visual complication, with the goal of evaluating whether psychological factors play a role in the evaluation of medical evidence.
2. In contrast to most commercial Web-based survey applications, the source code for DADOS-Survey is user customizable and can be installed on an academic institutional server, allowing data to be collected in a manner consistent with institution-specific and HIPAA data privacy regulations. Patient data can be collected with a greater degree of confidence among patients regarding the security of their responses. This includes surveys in which patients are asked about their perception of risk factors for certain medical conditions or their ethical perceptions.
3. Inter and intra-observer reliability studies evaluating radiographic imaging with dynamic images, including fluoroscopy cine images. The freely modifiable source code allows institutional users to develop plug-ins for DADOS-Survey to utilize different formats of images and video in surveys. Although movies and Java-script can be incorporated in the current version at our own institution, such features require programming that is not accessible to general users at the present time. Future versions of DADOS-Survey should include the ability by general users to dynamically insert movies and Java-script code into their survey designs.
We believe that DADOS-Survey is the first freely distributed open-source Web-survey software application that facilitates compliance with the CHERRIES data reporting guidelines. The initial experiences among researchers (i.e. survey designers) and those of survey recipients in early research studies have been positive. It is hoped that additional features will continue to be added to DADOS-Survey to increase its functionality and customization for unique Web-survey methodologies.
Availability and requirements
Project Name: DADOS-Survey
Project Home Page: http://www.ceso.duke.edu/ (click link for "Free software")
Operating System: Linux/Windows
Programming Language: Java >= 1.4.2
Other requirements: Tomcat 5.x, MySQL 5.0
License: GNU General Public License
Any restrictions to use by non-academics: none
List of abbreviations used
Checklist for Reporting Results of Internet E-Surveys
Health Insurance Portability and Accountability Act
Protected Health Information
- Ritter P, Lorig K, Laurent D, Matthews K: Internet versus mailed questionnaires: a randomized comparison. J Med Internet Res. 2004, 6: e29-10.2196/jmir.6.3.e29.View ArticlePubMedPubMed CentralGoogle Scholar
- Buchanan T, Smith JL: Using the Internet for psychological research: personality testing on the World Wide Web. Br J Psychol. 1999, 90 ( Pt 1): 125-144. 10.1348/000712699161189.View ArticleGoogle Scholar
- Nathanson AT, Reinert SE: Windsurfing injuries: results of a paper- and Internet-based survey. Wilderness Environ Med. 1999, 10: 218-225.View ArticlePubMedGoogle Scholar
- Davis RN: Web-based administration of a personality questionnaire: comparison with traditional methods. Behav Res Methods Instrum Comput. 1999, 31: 572-577.View ArticlePubMedGoogle Scholar
- Gosling SD, Vazire S, Srivastava S, John OP: Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. Am Psychol. 2004, 59: 93-104. 10.1037/0003-066X.59.2.93.View ArticlePubMedGoogle Scholar
- Wyatt JC: When to use web-based surveys. J Am Med Inform Assoc. 2000, 7: 426-429.View ArticlePubMedPubMed CentralGoogle Scholar
- Eysenbach G: Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. 2004, 6: e34-View ArticlePubMedPubMed CentralGoogle Scholar
- Pietrobon R, Shah A, Kuo P, Harker M, McCready M, Butler C, Martins H, Moorman CT, Jacobs DO: Duke Surgery Research Central: an open-source Web application for the improvement of compliance with research regulation. BMC Med Inform Decis Mak. 2006, 6: 32-10.1186/1472-6947-6-32.View ArticlePubMedPubMed CentralGoogle Scholar
- Devereaux PJ, Manns BJ, Ghali WA, Quan H, Guyatt GH: The reporting of methodological factors in randomized controlled trials and the association with a journal policy to promote adherence to the Consolidated Standards of Reporting Trials (CONSORT) checklist. Control Clin Trials. 2002, 23: 380-388. 10.1016/S0197-2456(02)00214-3.View ArticlePubMedGoogle Scholar
- Egger M, Juni P, Bartlett C: Value of flow diagrams in reports of randomized controlled trials. Jama. 2001, 285: 1996-1999. 10.1001/jama.285.15.1996.View ArticlePubMedGoogle Scholar
- Moher D, Jones A, Lepage L: Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. Jama. 2001, 285: 1992-1995. 10.1001/jama.285.15.1992.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/6/34/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.