STScI is moving to further anonymize the HST peer review process and has developed a procedure in which authors’ identities are concealed to reviewers. Provided here are responses to common issues (presented as an FAQ) to provide general information to the community on these changes. See the Recommendations of the Working Group on Anonymizing Proposal Reviews site for information on the working group that compiled these guidelines, as well as the final report authored by the working group. See the Proposer Guidelines in Anonymous Reviews for help in preparing proposals; a similar page, Reviewer Guidelines in Anonymous Reviews, gives general guidelines to reviewers.
|Table of Contents|
Why are you moving to double-anonymous reviews?
Fairness to proposers: STScI places a high value on the equity and integrity of the proposal review process. The goal is to minimize potential sources of bias and enable reviewers to focus on the science case of a proposal. Several studies have shown that a reviewer's attitude toward a submission may be affected, either consciously or unconsciously, by the identity of the lead author or principal investigator (see reference papers). We have noted that over the last 15 cycles, HST proposals led by women have had systematically lower success rates than those lead by men (Reid 2014). While the results are complex and the exact cause is unknown, independent studies of our reviews suggest a double-anonymous process would significantly reduce the potential for bias, whether by gender, affiliation or country of origin. Such a process may also level the playing field between new and established researchers.
Are truly anonymous submissions even possible?
Even in our relatively small community, it's not as likely as one might believe that one would correctly guess the authorship of a proposal. While it is possible to correctly guess the authorship, studies from other fields suggest the principal investigator's identity would remain unknown 60% to 75% of the time. So, while a system that provided perfectly unbreakable anonymity would be ideal, our goal is simply to obscure identity, to discourage guessing, and to reduce unconscious bias, and not make authorship a focus of the discussion and evaluation of a proposal.
How will you know if the experiment was successful or not?
It is not correct to consider the move to a double-anonymous process an experiment. It is one in a progression of changes that have been enacted over the years to improve the equity and integrity of the proposal review process. We will continue to evaluate the review process, with attention to fairness and balance over several factors, some of which are programmatic (e.g., are there more disproportionately more extragalactic programs than galactic?), and others demographic. We then make changes accordingly to address these issues.
Is there scientific evidence that anonymizing proposals results in a reduction of bias?
Yes. While this process may be new to astronomy, there is an abundance of literature on this topic from many different fields. The working group on anonymizing proposal reviews has reviewed the literature on this topic, and the results are clear: the removal of names and affiliations results in reduced bias in the review process. We have collected some relevant journal articles on this, which can be read here.
How difficult will the changes be on proposers?
The double anonymous proposal review process will require some changes in the way proposers write their proposals. We have written some Proposer Guidelines in Anonymous Reviews to describe these changes and aid in the proposal preparation. As one will note, the changes are mostly in the style, structure, and grammar used in describing the work done in the field, and the preparedness of the proposers to do the work. While not a lot of work, it will not be as simple as resubmitting previous versions of the same manuscript. We will also be asking each proposal team to submit a "Team Expertise and Background" document, providing a brief outline of the team members (non-anonymized) and their relevant expertise. This document will be shown to the review panels after the ranked list of proposals has been made. This document, which need be no longer than half a page, should be emailed to firstname.lastname@example.org after the submission of a proposal (a few days after the proposal deadline is fine). More on this final stage of review can be found below.
What changes will be made to the review panel process?
For most of the review process, the panels will not know the identity of the proposers. Panels must focus on the scientific merit of the proposal and the technical justification describing the use of the telescope to address the science proposed. This culminates in the panel making their ranked list of recommended programs. In addition to the panel chairs and the panel support staff, each panel will be assigned a "Leveler" for the duration of the review meeting. Levelers, who could be either STScI staff or external persons, will have received appropriate training and will attend all panel discussions as observers, and are responsible for ensuring that the discussion focuses solely on the aforementioned criteria, and does not deviate into speculation of team identity or other inappropriate topics. Once the ranked list is set, the identities of the proposers will be shown to the panel, in the form of the "Team Expertise and Background" document that the proposers submitted to STScI. At this final stage, the panel may choose to flag proposals for disqualification if the reason for doing so is well-justified and agreed to by a majority of the panel members. Reasons for disqualification include, but are not limited to: significant ethical misconduct in misrepresenting the scientific expertise of the team, misleading the panel as to the availability of supporting resources, or any other attempts to deliberately and unethically subvert the process. Should the panel choose to flag a proposal, they must articulate the reasons why in an open discussion with the panelists, chair, Leveler, and other appropriate members of the Science Mission Office at STScI. This discussion will be documented, as will the panel's initial ranking of the proposal. This disqualification flag, which would result in a non-selection of the proposal, is only a recommendation from the panel; the final decision rests with the Director.
How do the reviewers assess the proposers' responsible use of the telescope, or likelihood of scientific return?
As with all prior HST reviews, the panelists and the Telescope Allocation Committee (TAC) must use their expert judgement to determine whether each proposal would result in the proper use of the telescope and a scientific return on the project. Each proposal will still have a Description of the Observations section, in addition to the Scientific Justification. We encourage proposers to take extra care to sufficiently justify the technical requirements of the program in the Description of the Observations, such that the review panels can appropriately judge them.
How can we be sure that accepted HST proposals are actually feasible if the TAC can't assess the team's past experience?
It is STScI's responsibility to ensure that the community has equal opportunity for the use of HST, regardless of past experience. All accepted HST proposals are assigned a Program Coordinator (PC) who works with the PI and team to finalize the Phase II submission. The PCs are highly experienced and will flag particularly challenging proposals for further technical review. In addition, all proposals with PIs who are new to HST are assigned a Contact Scientist (CS) from the appropriate Instrument Division support team; the CS will provide both technical and scientific advice as necessary. If a program proves to be infeasible, it will not be executed.
As a reviewer, how can I be sure that the proposers are being ethical when discussing their expertise and/or access to other facilities? What if we allocate time to the wrong people?
First, it is a misconception that the review panels and the TAC allocate the time awarded for HST observations. The TAC makes recommendations to the STScI Director on which programs they feel are most worthy of observing time. The Director, as the selecting official, makes the final determination in coordination with appropriate members of STScI staff, including the Science Mission Office (SMO), HST Mission Office, ESA, and operations/scheduling staff. These recommended programs are also reviewed, with full identities exposed, during both the Phase II and Budget submissions by STScI staff and the Financial Review Committee (FRC). Secondly, at the final stage of the panel review, when the "Team Expertise and Background" section is shown to the reviewers (with names included), any egregious ethical violations or misrepresentations can be flagged. These will be brought to the attention of the Director, as well as the host institutions of the proposal team, and, if necessary, the broader community. Any such proposals are subject to disqualification.
How do the reviewers assess the proposers' responsible use of funds that are allocated with each science program?
The job of the reviewers is to evaluate the scientific merit of the proposals, and issue a recommendation to the Director on whether or not the proposed program is a worthwhile use of our most finite resource: observing time on HST. The funding of accepted proposals will remain, as it has always been, a separate part of the process, completely independent of the review panels and the TAC. Budget proposals will be reviewed by the FRC, and will not be anonymous.
How will this affect Archival, Theory, Treasury, and Large Proposals?
All proposals will be subject to this change and should be anonymized. For archival and theory proposals, the analysis plan, describing the tools and techniques that will be used to complete the proposed analysis, should remain in the PDF attachment, but should be anonymized in accordance with the proposer guidelines. The management plan, which describes the personnel responsible for the proposed study, should be removed from the proposal. The "Team Expertise and Background" section should describe the qualifications of the team members to carry out the proposed work. For accepted proposals, the management plan should be included in the budget proposal for review by the FRC.
When will this change be implemented?
The process will be implemented beginning in HST Cycle 26.
What will happen to proposals that are not sufficiently anonymized?
Compliance with this policy is mandatory, and proposers must follow the guidelines laid out in the Cycle 26 Call for Proposals. Flagrant violators will be subject to disqualification of their proposal before the review panel stage. Less serious violations (e.g., forgetting to change a reference from first person to third person) will be allowed to remain in contention, but will be flagged for review by SMO and the Director for a final decision. Feedback will be provided to the proposers regarding any violations.
I've followed the guidelines, but my work is so niche (or my analysis methods so unique) that I fear panelists may be able to determine my identity. Will I be flagged as non-compliant?
So long as the guidelines for proposers are followed, the answer is NO, such a proposal will not be considered to be in violation. It is not necessary to "water down" or obscure your science, your methods, or your tools, it is simply your responsibility to write about them in the third-person, in a way that does not intentionally identify yourself.
How can I be sure that someone won't plagiarize my proposal from a previous cycle?
STScI has strict plagiarism software that monitors all submissions. Suspicious cases will be flagged and reviewed by SMO staff, and plagiarized proposals may be disqualified. However, a similarity of ideas and proposals that can compete for the best use of HST to test these ideas is crucial to the scientific process.
How will the process deal with conflicts?
In some respects, the reviewer conflicts with a given proposal are a bit simpler. If the reviewers do not know who the proposers are, how can they be conflicted? However, there will still be checks within our system for reviewer conflicts, largely institutional or collaborative, and those with identified conflicts will be excused from the review of the proposal. As always, reviewers can (and should) identify issues not identified by our system, personal conflicts or possible competing proposals.
Won't this change make it harder to be awarded HST time?
To first order, no. Mathematically speaking, we're awarding the same amount of time as we have in previous cycles, meaning the same number of proposals will be accepted. If the number of submitted proposals were to rise, this would affect the oversubscription rate, but this has always been the case. This process is not about making it harder for some people or easier for others to get time, it's about ensuring that the best proposals are selected. Your best chance of being awarded observing time on HST is the same it has always been: think of a great idea, and write a great proposal.