Program Evaluation of NIH Peer Review Processes: The Role of Anonymization

Contracting Office Address Department of Health and Human Services, National Institutes of Health, National Heart, Lung and Blood Institute, Rockledge Dr. Bethesda, MD, Office of Acquisitions 6701 Rockledge Dr RKL2/6145 MSC 7902, Bethesda, MD, 20892-7902, UNITED STATES Introduction This is a Small Business Sources Sought notice. This is NOT a solicitation for proposals, proposal abstracts, or quotations. The purpose of this notice is to obtain information regarding: (1) the availability and capability of qualified small business sources; (2) whether they are small businesses; HUBZone small businesses; service-disabled, veteran-owned small businesses; 8(a) small businesses; veteran-owned small businesses; woman-owned small businesses; or small disadvantaged businesses; and (3) their size classification relative to the North American Industry Classification System (NAICS) code for the proposed acquisition. Your responses to the information requested will assist the Government in determining the appropriate acquisition method, including whether a set-aside is possible. An organization that is not considered a small business under NAICS code 541611 should not submit a response to this notice. Background The NHLBI, on behalf of the Center for Scientific Review (CSR), is seeking small businesses with the ability to serve the purpose of conducting a program evaluation project on the effect of anonymizing grant applications on reviewer scoring behavior. The data on this topic of interest are not currently available from any other source. The Center for Scientific Review (CSR) is the portal for NIH grant applications and their review for scientific merit. We receive all research grant applications sent to NIH and handle the review of more than 70% of those by organizing peer review groups (study sections) to evaluate research grant applications. CSR has major responsibility for the quality of NIH's investments to advance science and health through research. Our mission is to see that NIH grant applications receive fair, independent, expert, and timely reviews - free from inappropriate influences - so NIH can fund the most promising research. Each year, CSR handles over 85,000 grant applications, employs nearly 500 staff (more than 50% PhD level scientists) and uses more than 17,000 reviewers. CSR has engaged in a new initiative to closely examine the peer review process. Aims include the identification of procedures and practices that are most beneficial in accomplishing the Center's mission as well as identifying any aspects that might make the review process vulnerable to bias. Should any potential weaknesses be identified, CSR will work with experts in the research community to develop policies that mitigate any potential problems. The proposed work will be a first step in this process. Recent data (Ginther et al, 2011; 2012) have shown that African American investigators are less likely than White investigators to receive NIH R01 grant funding by up to 10 percentage points. Although the highest scoring applications had equal likelihood of funding regardless of race of applicant, black applicants were less likely than other groups to receive scores that allow them to be discussed by study sections and, therefore, were not advanced for funding consideration. This raises the question of possible bias in the peer review process. Given that demographic information is not included in NIH grant applications, the notion of bias in peer review is based on the assumptions that (1) reviewers can infer or have direct knowledge of the race, ethnicity, and/or gender of grant applicants and (2) this information influences their ratings. To investigate these possibilities, reviewer perceptions of applicant demographics must be assessed. In addition, a comparison will be made of reviewer scores on grant applications submitted by black and white investigators where their identities are either masked or unmasked. If all other applicant characteristics are equivalent, a difference in scores for masked and unmasked applications from black applicants that is greater than the difference in scores for masked and unmasked applications from white applicants would provide support for the hypothesis of bias due to race of applicant. In addition to race, the effect on awards of gender, seniority and institutional affiliation is also of interest. Project Requirements The proposed project will test the effect on reviewer ratings of removing investigator and institutional information from grant applications. Specifically, the goal is to determine if anonymity in grant applications will reduce any effects on application scores attributable to reviewers' perception of race, gender, seniority, and institutional affiliation of the applicant. Services are needed to (1) anonymize grant application by removing identifying information from grant applications, (2) assign original and anonymized applications for review by CSR reviewers (3) analyze resulting scores to determine the effect of anonymizing on overall impact scores, and (4) evaluate the accuracy of reviewers' perceptions of applicant characteristics and assess the perceived grantsmanship of each application. CSR will supply 1200 grant applications to be reviewed in whole and anonymized versions. Each version will receive three independent reviews by reviewers that will judge between 4 and 6 applications each and then electronically submit their reviews. A critical component of this proposed project is data security; specifically, the security of grant and investigator related information and maintenance of the confidentiality of grant reviewers. Requirements include assurance of secure communication with potential reviewers, confidential transmission of grant related materials and related critiques, and the security of personally identifiable information. Potential vendors must have appropriate equipment, software, policies/procedures, and personnel to maximize data security. They must be prepared to work with applications and data permanently residing on NIH servers. In addition, a firm understanding of the NIH peer review processes, practices and regulations is needed so that procedures for selection of reviewers and assignment of applications for review can mirror NIH practices. Evidence of data analytic experience is required including measure development and testing of psychometric properties. Relevant data analytic skills are also needed to design an anonymizing experiment and answer the following research questions: (1) To what degree do reviewers believe they have awareness of the demographic characteristics of grant applicants? (Can grant applications be anonymized?) (2) How accurate are their assessments of the demographics of grant applicants? (3) To what degree do scores provided for anonymized applications differ from their original versions that include investigator and institution information? (4) Are the differences in scores between anonymous and original applications different for black versus white applicants?; and (5) Are there any differences in grantsmanship ratings for original and anonymized applications for black and white PIs? Capability Statement Small business concerns that believe they possess the capabilities necessary to undertake this work should submit complete documentation of their capabilities to the Contracting Officer. The capabilities statement must specifically address each project requirement separately. Additionally, the capability statement should include 1) the total number of employees, 2) the professional qualifications of scientists, medical experts, and technical personnel as it relates to the above outlined requirements, 3) a description of general and specific facilities and equipment available, including computer equipment and software, 4) an outline of previous research projects that are similar to the project requirements in which the organization and proposed personnel have participated, and 5) any other information considered relevant to this program. The capability statement must not exceed 10 single sided or 5 double sided pages in length and using an 11-point font size minimum. Interested small business organizations are required to identify their size standards in accordance with the Small Business Administration. The government requests that no proprietary or confidential business data be submitted in a response to this notice. However, responses that indicate the information therein is proprietary will be properly safeguarded for Government use only. Capability statements must include the name and telephone numbers of a point of contact having authority and knowledge to discuss responses with Government representatives. Capability statements in response to this program evaluation that do not provide sufficient information for evaluation will be considered non-responsive. When submitting this information, please reference the solicitation notice number. All capability statements sent in response to this Sources Sought Notice must be submitted electronically (via email) to Jenn Frazier, Contract Specialist, at jenn.frazier@nih.gov in either MS Word or Adobe Portable Document Format (PDF), within 15 calendar days of the date of this announcement. All responses must be received by the specified due date and time in order to be considered. This notice does not obligate the Government to award a contract or otherwise pay for the information provided in response. The Government reserves the right to use information provided by respondents for any purpose deemed necessary and legally appropriate. Any organization responding to this notice should ensure that its response is complete and sufficiently detailed to allow the Government to determine the organization's qualifications to perform the work. Respondents are advised that the Government is under no obligation to acknowledge receipt of the information received or provide feedback to respondents with respect to any information submitted. After a review of the responses received, a pre-solicitation synopsis and solicitation may be published in Federal Business Opportunities. However, responses to this notice will not be considered adequate responses to a solicitation. No proprietary, classified, confidential, or sensitive information should be included in your response. The Government reserves the right to use any non-proprietary technical information in any resultant solicitation(s). The solicitation release date is pending. The Government intends to negotiate one time-and-materials type contract, for a period of one year with an approximate award date of September 15, 2016. Original Point of Contact Jenn Frazier, Contract Specialist, NHLBI COAC Services Branch, Phone (301) 496-6092, Fax (301) 480-3338, Email: jenn.frazier@nih.gov Janet Mattson, Contracting Officer, NHLBI COAC Services Branch, Phone (301) 435-0357, Fax (301) 480-3338, Email: mattsonjm@nhlbi.nih.gov Department of Health and Human Services, National Institutes of Health, National Heart, Lung and Blood Institute, Rockledge Dr. Bethesda, MD, Office of Acquisitions 6701 Rockledge Dr RKL2/6145 MSC 7902, Bethesda, MD, 20892-7902, UNITED STATES