Coordinator

• Dr. Dorothe Poggel (HWK)

Speaker

• Prof. Dr. Andrea Hildebrandt, Psychological Methods and Statistics, Department of Psychology, Faculty VI. – School of Medicine and Health Sciences, Carl von Ossietzky Universität Oldenburg (UOL)

Members

• Prof. Dr. Stefan Debener, Neuropsychology, Department of Psychology, Carl von Ossietzky Universität Oldenburg
• Dr. Carsten Gießing, Biological Psychology, Department of Psychology, Carl von Ossietzky Universität Oldenburg
• MSc. Nadine Jacobsen, Neuropsychology, Department of Psychology, Carl von Ossietzky Universität Oldenburg and Open Science Speaker of the University 
• Dr. Daniel Kristanto, Psychological Methods and Statistics & Biological Psychology, Department of Psychology, Carl von Ossietzky Universität Oldenburg
• Dr. Cassie Short, Psychological Methods and Statistics, Department of Psychology, Carl von Ossietzky Universität Oldenburg
• Prof. Dr. Christiane Thiel, Biological Psychology, Department of Psychology, Carl von Ossietzky Universität Oldenburg

Duration

March 2023 – February 2026

Problem Statement

Meta-Science, the emerging field of research dedicated to study the scientific process itself, is in its momentum. Many of us believe that it has become a permanent and important discipline in itself, along with the open science movement it promotes. Meanwhile, researchers and funding agencies spanning all scientific disciplines are increasingly emphasizing the role of Meta-Science in diagnosing and curing the replication crisis in science. Procedures such as study preregistration, open materials and open data, and registered reports are meanwhile widely practiced. They are also promoted and anticipated by many high-ranked journals and academic institutions. However, certain procedures proposed by the open science movement are still far from being used regularly. Methodological solutions that aim to add transparency to the black-box created by researchers’ degrees of freedom are only implemented by small communities characterized by extensive expertise in programming and data science. These communities feel urged to create user-friendly software solutions allowing to explore and address the multitude of choices in the scientific process, exclusive of a formal training in scientific programming and big data analyses. Furthermore, today meta-scientists engage in implementing systematic expertise crowdsourcing strategies to make domain specific, scientifically sound (defensible) choices in the study process explicit and easily accessible by entire scientist communities. 

To describe the multitude of researchers’ methodological choices, Gelman and Loken (2013) coined the term “garden of forking paths”. In every step of the study planning, data processing and analysis workflow, multiple defensible decisions and potential operations are available as choices. The established approach in science to date is to select one specific workflow from which results are reported and conclusions of the phenomenon studied are made. However, a multiple comparison problem is implicit to this approach, even if only one constructed dataset is used for statistical inferences. The reason is that theoretically a large variety of workflows are defensible, yet researchers do not correct their hypothesis tests to account for all theoretically possible comparisons that they did not explicitly carry out.   

More recently, the term multiverse has been proposed for the above elaborated problem (e.g., Steegen et al., 2016) to underline the high dimensionality of the garden of forking paths. Today, meta-scientists differentiate three domains of multiverse in the scientific process. Design multiverse refers to the study design specification, data multiverse to data processing and parametrization choices, and model multiverse to the statistical analyses options available to address specific research questions and/or hypotheses. The resulting multiverse is immense, domain specific and highly dynamic. This is because, for example in the neuroimaging field, methodological solutions are improved upon and advanced with huge pace. For small teams of scientists, such as single laboratories, it is thus impossible to gain a comprehensive map of the multiverse of defensible choices at a given moment in time. We need large collaborating communities (Wacker, 2017) and dynamic expertise crowdsourcing platforms to continuously map the multiverse in specific fields. 

Our study group aims to map and visualize the multiverse in two neuroimaging domains: (1) Stationary and mobile electroencephalography (EEG) applications in cognitive and personality neuroscience and (2) functional magnetic resonance imaging (fMRI) for network neuroscience approaches to individual differences in cognitive abilities. Furthermore, we aim to develop and promote the use of software solutions for a more easily applicable multiverse research in these domains. The focus group is tightly linked to the METEOR project embedded into the DFG Priority Program META-REP (“A Meta-scientific Program to Analyse and Optimise Replicability in the Behavioral, Social, and Cognitive Sciences”, SPP 2317) and to the entire program. We also collaborate with the DFG funded CoScience – The EEG Personality Project. Furthermore, our activities are open to the members of the DFG funded Research Training Group “Neuromodulation of Motor and Cognitive Function in Brain Health and disease” (RTG 2783). 

At the HWK, the Study Group contributes to the institute's focus on the quality of academic research.

Planned Activities

• We will carry out community workshops and launch interlinked crowdsourcing expertise platforms to map the multiverse in the domain of stationary and mobile EEG research of individual differences in cognition and personality, as well as network neuroscience fMRI approaches to individual differences in cognition.
• Aiming to make the outcome of the community discussions a collective good, we will summarize them along with the agreements and remaining controversies regarding the defensible choices in the multiverse in the form of opinion papers and guidelines for the above fields of interest.
• As supporting tools to accompany such guidelines, we will create open source and extendable visualizing tools which will allow researchers to navigate through the community agreed map of the multiverse.
• Within the DFG project METEOR, we are about to create easily adaptable analyses pipelines driven by machine learning with the aim of computationally managing the multiverse. These pipelines will be promoted by the study group.  
• On a more general level, we will carry out a series of educational activities (launching interactive and open learning tools for multiverse analyses, hands-on workshops) in collaboration with the Open Science Interest Group (OSIG) at the Department of Psychology, UOL. Thus, we will promote knowledge on multiverse analyses in the broader cognitive neuroscience community, specifically targeting early career scientists of the region Oldenburg/Bremen, potentially extending to further Institutions in North Germany and Groningen. 

Work Plan for the First Year

We will design three interlinked expertise crowdsourcing platforms (stationary EEG; mobile EEG; fMRI) and invite the community to share their expertise with respect to the scientific soundness of a multitude of decision options in the domain of design, data and model multiverse. We will focus on decisions hitherto collected from the literature and the community) that are invariant across a series of research questions typically addressed in the above mentioned neuroimaging fields. We will then analyze and summarize the collected expertise, specifically with the goal of pointing to controversies within the community with respect to defensible design, data processing and analytic choices. These controversies will then be targeted and potentially solved in focus groups, organized as in person workshops (separately for the EEG and fMRI field) planned to be carried out early 2024.     

Work Plan for the Second Year

We will carry out a 3-day workshop (one day per neuroimaging domain, with potential cross-domain participation) at the HWK. The goal is to bring together a group of experienced and early career scientists for each neuroimaging domain to discuss and potentially solve the controversies with respect to which design, data processing and analyses choices in the respective domain remain defensible. The outcomes of the workshops will be summarized in opinion papers and community guidelines, along with supporting tools. Furthermore, the focus group will collaborate with OSIG and the RTG 2783 and create open educational resources and organize hands-on workshops to promote multiverse analyses skills of early career scientists. 

Work Plan for the Third Year

The activities in the first and second year specifically target the data and model multiverse. In the last year we will more closely address the problem of design multiverse. There is evidence that many neuroimaging studies suffer from low reliability and reproducibility. A simple, however very expensive and in many cases non-practicable solution is to drastically increase the sample size of neuroimaging studies. It is currently unknown whether specific forking paths regarding design specification, data processing, and statistical analysis would improve the robustness and reliability of neuroimaging studies. This still needs to be systematically explored. Is it possible to identify forking paths best suited to small-sample studies? We aim to organize a workshop to address this prominent and crucial challenge in cognitive neuroscience, which is the problem of small samples. We will invite experts in statistics who engage in developing statistical tools for the analyses of small data to give a 2-day hands-on workshop. Above the hands-on part of this workshop, the goal will be to discuss applications in the domain of neuroimaging.  

References

Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. 

Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multi-verse analysis. Perspectives on Psychological Science, 11, 702–712.

Wacker, J. (2017). Increasing the reproducibility of science through close cooperation and forking path analysis. Frontiers in Psychology, 8, e1332.