(a) Background

Most meta-analyses are published as printed articles only. Even if meta-analytic data is accessible, there is a lack of standardization to enable updating meta-analyses easily. Therefore, most meta-analyses remain static. Moreover, without proficient knowledge of meta-analytic methods, meta-analyses cannot be replicated, nor can the sensitivity of the results to subjective decisions be examined. Evidence therefore is not easily accessible for potentially relevant target groups as practitioners and decision-makers. Therefore, open meta-analytic data and tools should aim at making meta-analytic evidence usable, accessible and easy to update.

(b) Objectives and system requirements

We are developing a platform to improve the re-usability and augmentation (i.e., by adding more recent primary studies) of meta-analyses. Its main functionalities are the standardization and extensibility of the data, and the use of basic meta-analysis tools on a graphical user interface (GUI). Therefore, we can state two main objectives:

1. Improve the efficiency of evidence collection and accessibility in psychology

To this end, the research community is enabled to collect meta-analytic data, make it easily reproducible and add new evidence building on the data already collected by others. Therefore, open meta-analytic data has to follow certain standards (e.g. data preparation and naming conventions) to make the augmentation and analysis of different datasets as efficient as possible. The resulting interoperability of the different datasets makes it easier to use the same analysis scripts for different meta-analyses and to conduct meta-meta-analyses for meta-scientific research questions.

2. Improve the benefit of meta-analytic evidence in psychology

An easy-to-use GUI should help to make up-to-date evidence accessible and understandable for decision-makers and practitioners, who are not proficient in meta-analytic methods. As meta-analytic outputs and plots are not self-explanatory, context-sensitive interpretation aids within the GUI and Plain Language Summaries for each meta-analysis will be offered to guide the process of drawing practical conclusions from the existing evidence for decision-making.

(c) Approach

A solution for an infrastructure for continuous updating of meta-analytic evidence is the concept of CAMAs (Community-Augmented Meta-Analysis). A CAMA is an open repository for meta-analytic data, that provides a GUI for meta-analytic tools. Researchers have the possibility to contribute data to the repository and to use the analyses on the GUI.

At ZPID, a service called PsychOpen CAMA is developed to serve the field of psychology. Meta-analyses can be published as CAMAs by providing the data in a standardized format. Therefore, data templates are available that define the structure and the naming of the variables for the meta-analyses. According to the outcome data given, effect sizes are calculated automatically with the escalc() function of the metafor package.

The web application allows the user to choose a dataset to get an overview of the data and to explore the distributions of potential moderators. Basic meta-analytical tools, as forest and funnel plot, p-curves and meta-regression can be requested from the web application. The functions needed for these analyses are part of a self-maintained R package. As each meta-analysis may differ in terms of analysis levels and relevant moderators for the meta-regression model, the description of these dataset-specific characteristics is given as meta-information associated to the corresponding dataset. In conjunction with the naming conventions from the data templates, this ensures interoperability and makes it possible to conduct analyses on various datasets and with different moderator variables with the functions from the psychopenCAMA package in R. The analysis requests are sent to an OpenCPU server, processed there and the results of the operations are given back as output on the GUI.

(d) Demonstration of the system

Up to now, community-augmented meta-analyses in three different domains in psychology (personality, cognitive development, and survey methods) are published on the platform. With the first release in early 2021, these will be accessible and can be updated by the research community in case of further evidence.

Using a printed meta-analysis as an example, the demonstration will show

1. The standardization and implementation of the data into PsychOpen CAMA
2. The functionalities on the GUI allowing replication and variation of analysis requests by producing dynamic output

(e) Conclusions and implications (expected)

Despite the obvious benefit of a platform allowing re-usability and community-driven accumulation of evidence, there are challenges remaining for PsychOpen CAMA and similar projects.

1. Enlarge the scope

At the moment, PsychOpen CAMA comprises univariate meta-analyses with three different types of effect sizes: Standardized mean differences, correlations, and proportions. For example, meta-analysis of results from structural equation models (MASEM), cannot be implemented at the moment. Network meta-analyses are not provided in PsychOpenCAMA currently. And in case raw data from some of the studies included in a meta-analysis are available, there is no way to consider the participant level in the analyses. These are just some examples that call for a step-by-step extension and adaptation of datasets, template standards, and analytic functionalities.

2. Automatization of data management

Up to now, data standardization and implementation of the seed meta-analyses are done manually. With the first release of PsychOpen CAMA, data providers are expected to fill in their data in the data template format. However, whenever questions arise, a user support from a CAMA expert will be needed and suggestions for potential extensions have to undergo quality-checks. It remains a challenge to make data entry and validation easier and user support and quality appraisal of the data more efficient by automatizing at least some tasks. However, a curation and augmentation of meta-analytic data without human effort is not possible at the moment.