You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 30 Next »

Site Contents

The HST peer review process will move to a double-anonymous review process, in which authors’ identities are concealed from both review panel members and TAC members. Provided here are responses to common issues (presented as a FAQ) to provide general information to the community on these changes. See the Anonymizing Proposal Reviews Working Group Home site for information on the working group that compiled these guidelines.  The Proposer Guidelines in Anonymous Reviews for help in preparing proposals, and a similar page Reviewer Guidelines in Anonymous Reviews gives general guidelines to reviewers. Please send questions and comments to hstanon@stsci.edu

Why are you moving to double-anonymous reviews?

STScI places a high value on the equity and integrity of the proposal review process. The goal is to give each reviewer an unbiased look at the proposal. Several studies have shown that a reviewer's attitude toward a submission may be affected, even unconsciously, by the identity of the lead author or principal investigator (see reference papers).  We have noted that over the last 15 cycles, HST proposals led by women have had systematically and successively lower success rates than those lead by men (Reid 2014). While the exact cause is unknown, independent studies of our reviews suggest a double-anonymous process would resolve this inequity, and may balance out other areas of potential bias including affiliation and country of origin. Such a process may also level the playing field between new and established researchers.

Are truly anonymous submissions even possible?

Even in our relatively small community, it's not as likely as one might believe that one would correctly guess the authorship of a proposal. While it is possible to correctly guess the authorship, studies from other fields suggest the principle investigator's identity would remain unknown 60% to 75% of the time. So, while a system that provided perfectly unbreakable anonymity would be ideal, our goal is simply to obscure identity, to discourage guessing, and to reduce unconscious bias, and not make authorship a focus of the discussion and evaluation of a proposal.

How will you know if the experiment was successful or not? 

It is not correct to consider the move to a double-anonymous process an experiment. It is one in a progression of changes that have been enacted over the years to improve the equity and integrity of the proposal review process. We continuously evaluate the review process, with attention to fairness and balance over several factors, some of which are programmatic (e.g., are there more disproportionately more extragalactic programs than galactic?), and others demographic. We then make changes accordingly to address these issues.

How difficult will the changes be on proposers?

The double anonymous proposal review process will require some changes in the way proposers write their proposals. We have written some Proposer Guidelines in Anonymous Reviews to describe these changes and aid in the proposal preparation. As one will note, the changes are mostly in the style, structure, and grammar used in describing the work done in the field, and the preparedness of the proposers to do the work. While not a lot of work, it will not be as simple as resubmitting previous versions of the same manuscript.

How do the reviewers assess the proposers' responsible use of the telescope, or likelihood of scientific return?

As with all prior HST reviews, the panelists and the Telescope Allocation Committee (TAC) must use their expert judgement to determine whether each proposal would result in the proper use of the telescope and a scientific return on the project. Each proposal will still have a Description of the Observations section, in addition to the Scientific Justification. We encourage proposers to take extra care to sufficiently justify the technical requirements of the program in the Description of the Observations, such that the review panels can appropriately judge them. 

How can we be sure that accepted HST proposals are actually feasible if the TAC can't assess the team's past experience?

It is STScI's responsibility to ensure that the community has equal access to HST, regardless of past experience. All accepted HST proposals are assigned a Program Coordinator (PC) who works with the PI and team to finalize the Phase II submission. The PCs are highly experienced and will flag particularly challenging proposals for further technical review. In addition, all proposals with PIs who are new to HST are assigned a Contact Scientist (CS) from the appropriate Instrument Division support team; the CS will provide both technical and scientific advice as necessary. If a program proves to be infeasible, it will not be executed.

How do the reviewers assess the proposers' responsible use of funds that are allocated with each science program?

The job of the reviewers is to evaluate the scientific merit of the proposals, and issue a recommendation to the Director on whether or not the proposed program is a worthwhile use of our most finite resource: observing time on HST. The funding of accepted proposals will remain, as it has always been, a separate part of the process, completely independent of the review panels and the TAC. Budget proposals will be reviewed by the Financial Review Committee, and will not be anonymous.

How will this affect Archival, Theory, Treasury, and Large Proposals?

All proposals will be subject to this change and should be anonymized. For archival and theory proposals, the analysis plan, describing the tools and techniques that will be used to complete the proposed analysis, should remain in the PDF attachment, but should be anonymized in accordance with the proposer guidelines. The management plan, which describes the personnel responsible for the proposed study, should be removed from the proposal. For accepted proposals, the management plan should be included in the budget proposal.

What will the implementation process be?

The implementation process is still being developed, but it is likely that this will be implemented in Cycle 26.

What will happen to proposals that are not sufficiently anonymized?

Panelists will flag proposals that contain identifying information as non-compliant for review by staff in the Science Mission Office (SMO). Non-compliance may affect the outcome of the proposal. Proposals may be downgraded or rejected. STScI reserves the right to disqualify any offending proposals. Feedback will be provided to the authors of the proposal when the reviewers comments are returned after the TAC has concluded and the Director has made the selections.

How can I be sure that someone won't plagiarize my proposal from a previous cycle?

STScI has strict plagiarism software that monitors all submissions. Suspicious cases will be flagged and reviewed by SMO staff, and plagiarized proposals may be disqualified. However, a similarity of ideas and proposals that can compete for the best use of HST to test these ideas is crucial to the scientific process.

How will the process deal with conflicts?

In some respects, the reviewer conflicts with a given proposal are a bit simpler. If the reviewers do not know who the proposers are, how can they be conflicted? However, there will still be checks within our system for reviewer conflicts, largely institutional or collaborative, and those with identified conflicts will be excused from the review of the proposal. As always, reviewers can (and should) identify issues not identified by our system, personal conflicts or possible competing proposals. 

  • No labels