Game Changer Academies

We are Improving Panel Discussions Mitigating Cognitive Bias Exploring “High Risk|High Reward”

Advancing Research Innovation

Game Changer Academies for Advancing Research Innovation is a partnership between Kardia Group and the CMMI division of the Engineering directorate within the National Science Foundation (NSF). CMMI’s Game Changer Academies (CGCA) builds on NSF’s robust agenda for rewarding innovation and transformative science by focusing on improving panel discussions, ameliorating a variety of forms of cognitive bias, and examining the meaning of “high risk/high reward” in an engineering context.

Game Changer Academies is an investment in NSF, innovative scholarship, and the peer review process. Panel Fellows support cutting edge scholarship; help align panels with the goals of NSF; increase consistency, efficiency, effectiveness, and fairness in the review process; and create better reviewer experiences for the scores of scholars who contribute to the NSF peer review process.

Game Changer Academies contributes to the advancement of research innovation by looking at how four topics impact the funding deliberation process: group behaviors, cognitive bias, social identity, and conflict. Watch Game Changer Academies host Diana Kardia explain how these four topics connect to research innovation in the video below.

The inaugural cohort of 144 CMMI Panel Fellows completed the program in October of 2021. A second cohort of 181 Panel Fellows completed the program in May of 2022. The 3rd cohort begins in December 2022 and runs through May 2023.

Over time, this investment in skilled discussion participants will elevate both the process and the outcomes of peer review in CMMI and will pave the way for NSF reviews more broadly – which already serve as a gold standard in the advancement of scientific knowledge.

You can read more about the program in this NSF news article:
NSF invests in reviewer development to advance innovation

NFS’s Peer Review Process

How Does GCA Advance Research Innovation?

What do participants say
about the program?

Testimonials

“The 2021 CGCA program fundamentally changed how I engage in group discussions.”

“The panel fellows’ program has helped me tremendously in becoming a good reviewer.”

Read more testimonials

More Program Feedback

See more participant feedback

2023 Program Dates

Virtual Main Sessions
(9-10:30am PT/noon-1:30 ET)

Wed Dec 7, 2022
Wed Jan 18, 2023
Wed Feb 15, 2023
Wed Mar 15, 2023
Wed Apr 19, 2023
Wed May 17, 2023

Virtual Discussion Sessions
(specific timing tbd at registration)

week of Jan 9-13
week of Feb 6-10
week of Mar 6-10
week of Apr 10-14
week of May 8-12

Join our mailing list

Game Changer Academies is also a mechanism for professional development specifically tailored to scholars. Panel Fellows gain a better understanding of panel dynamics and funding decision processes, gain insight about proposal writing and peer reviews, and improve their own capacity to be funded through a deeper understanding of scholarly funding processes. Insights gained through Game Changer Academies may also be applied to faculty meetings, lab dynamics, and other scholarly interactions in academia.

Academy Leadership

Changing The Game

Game Changer Academies is designed and led by Diana Kardia (L) and Kelly Mack (R), in collaboration with a team of faculty, organizational change experts, and leaders at NSF.

Please email us with any questions at inquiries@gamechangeracademies.com

Game Changer Academies FAQs

What will I learn in the Game Changers Academies?

Game Changer Academies focuses on intra- and inter-personal dynamics in the context of peer review. Our content is informed by research and best practice literature surrounding four key concepts: cognitive biases, group dynamics, social identity, and conflict. Additionally, we draw significantly from research on peer review and gatekeeping in academia. Our primary emphasis is on the panel review discussion itself: after individuals have drafted their individual reviews and before a panel summary and subsequent funding decision is made.

This program is not a “how to” format, nor does it give simple answers. Game Changer Academies is designed with faculty in mind: people who are already highly trained to synthesize data and information and translate ideas into action through complex, adaptive, and continually evolving discussions with peers.

Does Game Changer Academies define what NSF considers to be an acceptable risk of failure?

While NSF is fundamentally responsible for the funding decisions and processes it oversees, acceptable risk associated with the advancement of human knowledge and capacity cannot be defined by an institution. Across academia, the balance of risk, achievement and failure is negotiated daily through deep discussion between faculty, researchers, academic administrators, and scholarly experts. The target moves, evolves, and is understood differently according to a significant array of factors. Game Changer Academies exists because achieving this understanding requires highly skilled interactions and group processes. So, we don’t define an acceptable risk of failure in precise terms, but we do contribute to the capacity of the scholarly community to engage this important question.

Will I learn to resolve conflicts with opinionated overbearing senior panel members?

We hope so! This type of conflict is the result of untended processes, missed opportunities, and constrained interactions. Game Changer Academies examines these variables and provides tools to produce better outcomes.

Will Panel Fellows reduce implicit bias in the review process?

Panel Fellows will work with other panelists and Program Directors to mitigate cognitive bias in the review process. Because cognitive biases of any type, including implicit bias related to gender, race, and other social identities, are hardwired into the way we function as humans, awareness alone cannot eliminate bias. Instead we examine how and why cognitive biases operate so that Panel Fellows can actively take steps to mitigate the effects of these biases in the review process to produce more appropriate and equitable decisions.

This functions similarly to the role of a weather forecast: awareness does not change what the weather is, but it can significantly impact our ability to prepare for and navigate the effects of a particular weather pattern.

Will I gain abilities to distinguish well-constructed high-risk, high-reward proposals from poorly constructed ones?

We assume that all Panel Fellows already bring some facility to the tasks associated with proposal review. Game Changer Academies addresses how the review process functions after individuals have read and initially reviewed a proposal.

Will it help me understand what to look at in a proposal and what "intellectual merit" and "broader impacts" really mean?

The Game Changer Academies curriculum focuses on panel interactions and discussion; it is not designed to examine proposal components or review criteria. That said, high quality discussions hinge on common definitions and shared processes. (Indeed, a common discussion pitfall is working at cross-purposes as individuals to solve different problems.) Since intellectual merit and broader impacts exemplify a common cause of any NSF panel review, they do deserve (and receive) attention in the Game Changer Academies program. Watch this 20 minute video for additional information about NSF’s definition of these review criteria.

Will you tell us how NSF review processes are formed, executed and utilized for decision making? Will we find out about changes NSF is contemplating or planning?

GCA focuses on the interactions that occur within a panel review discussion; GCA program leaders and discussion leaders will bring this as their primary expertise. CMMI leadership and Program Directors are the best source of information about NSF’s review and decision-making process as well as any changes in the division. There may be opportunities during the course of CMMI’s Game Changer Academies to hear directly from CMMI leadership. Additionally, conversations with other Panel Fellows will be a great way exchange information about how the review process works.

Will I learn how to review a proposal for which I don’t have the exact expertise needed?

Academia is a conglomeration of ideas, questions, theories, frameworks, and projects that each represent a set of literatures, methodologies and disciplinary worldviews. As such, this skill set is cultivated across the full trajectory of a scholar’s career and is not easily generalized across disciplines or reduced to a procedure. What Game Changer Academies does address is how each individual brings vital and unique insights to a panel review discussion and how the interaction among panel members recognizes, utilizes, and benefits from these individual contributions so that a collective and relevant expertise is applied.

It seems to me that different panels reviewing the same set of proposals produce different results. Will Game Changer Academies fix that?

Mostly no – for two key reasons. First, the advancement of knowledge is a highly social and interpersonal process, not just a cognitive one. Science requires the human capacities to imagine, to challenge ourselves and each other, and to risk the unknown – and to support others as they take that risk. We rely on the variability, peculiarity, and genius of individuals to inform our understanding of quality, excellence, and science. In other words, the variable outcomes of panel discussions serve a larger mission.

To allow for this, NSF peer review is designed with many checks and balances. Panel reviews need not be so regimented that they produce the same outcome no matter the players. The process accounts for the variability.

At the same time, even functional peer review processes include avoidance of or poorly executed conflict; unconscious bias that penalizes, obscures, and excludes; and group dynamics hindered by academic egos and hierarchies. That is something we can and should change. Game Changer Academies’ Panel Fellows are at the cutting edge of this change.

What career stage makes for a good Panel Fellow: junior, senior, or mid-career?

As with panels themselves, faculty at all ranks are eligible to be Panel Fellows. Panel Fellows will also discuss how rank may impact group dynamics during the course of a given panel discussion.

Is this just a CMMI initiative or are other NSF divisions implementing this initiative?

CMMI is taking the lead with bringing Game Changer Academies to NSF. Other divisions and directorates are asking similar questions. CMMI’s success may pave the way for innovation in other programs.

Will this initiative lead to a small set of researchers having disproportionate power in determining research?

Game Changer Academies is an investment in the quality of panel discussions broadly. As skilled discussion participants, the presence of Panel Fellows on a panel is intended to elevate the participation of all panelists and increase the capacity of the panel to benefit from a diversity of expertise and experience.

Isn’t it the Program Director’s role to lead panels?

Yes. Program Directors will continue to lead the panel review process. The focus of Game Changer Academies is to increase panelists’ skill as participants in robust panel discussions.

Will future CMMI panels be limited to Panel Fellows?

There are no plans at this time to require Panel Fellow status of all CMMI panelists.

After my CGCA experience, I’d like to do more. Is there any kind of advanced training?

Yes. We are planning an in-person CGCA Advanced Panel Fellow Training in Alexandria, VA, on October 19-20, 2022. When the application process is open (in June 2022), we will invite all Panel Fellows from cohorts 1 and 2 to apply.

Privacy Statement

Invitee and participant contact information (“contacts”) will be shared among Kardia Group, LLC, GPRA Strategic Management, Inc., and the CMMI Directorate at NSF (“we”) as needed to administer the Game Changer Academies program. We may use contacts to notify individuals of future Game Changer Academies activities. We will not sell or share contacts with any other entities.

We are Improving Panel Discussions Mitigating Cognitive Bias Exploring “High Risk|High Reward”

Game Changer Academies

How Does GCA Advance Research Innovation?