Blog

ISR Roundtable 2023: The future of preserving the integrity of the scholarly record together

Madhura Amdekar – 2024 February 06

In Research IntegrityTrustworthinessStrategy

Metadata about research objects and the relationships between them form the basis of the scholarly record: rich metadata has the potential to provide a richer context for scholarly output, and in particular, can provide trust signals to indicate integrity. Information on who authored a research work, who funded it, which other research works it cites, and whether it was updated, can act as signals of trustworthiness. Crossref provides foundational infrastructure to connect and preserve these records, but the creation of these records is an ongoing and complex community effort. Crossref has always shown a deep commitment to preserving the integrity of the scholarly record in an open and scalable manner.

Given the increasing concerns in the community about matters of research integrity and integrity of the scholarly record (ISR), we at Crossref have been engaging with community members to understand what developments are needed. In 2022, we organised a roundtable discussion to talk about our role and the applicability of Crossref’s services in preserving and assessing the integrity of the scholarly record. We’ve acted on much of that feedback since, and so in October 2023, we organised a follow-up event, once more gathering representatives of publishers, research integrity experts, policy-makers, academic institutions, funders, and researchers (the full list of participants can be found in the appendix). This post aims to offer insight into the discussions at this event and the next steps. The objective of this event was to take the conversation forward by:

• Sharing the progress made by Crossref on matters related to ISR since the last roundtable event.
• Sharing information about how metadata contributes to the Research Nexus, and can act as trust markers for research outputs.
• Apprising the community about the latest membership trends and examples of activities that we see, such as title transfer disputes, unregistered DOIs, requests for deleting records, and sneaked references .
• Building upon the ideas discussed during the 2022 roundtable event to progress the conversation about issues related to ISR.
• Learning from the participants about their experiences of pursuing research integrity initiatives.
• Last but no less importantly, hearing from the participants their perspectives on strategies for preserving the integrity of the scholarly record, and opportunities for collaborating to leverage metadata to assess the integrity of the scholarly record.

The event was kicked off by Ed Pentz, who spoke to the participants about how integrity is key to Crossref’s mission, and Crossref’s vision of the Research Nexus. Next, Amanda Bartell, the Head of Member Experience at Crossref, shared the recent developments and trends in community behaviour. She expanded upon the actions taken by Crossref as part of its ISR program since the last roundtable event, which include:

• Acquisition and opening of the Retraction Watch database, which makes it easier to access information on retractions and corrections.
• Increased participation in the Global Equitable Membership (GEM) program, enabling a wider section of the community to provide and access trust signals.
• Newer developments around metadata that act as trust signals: e.g. 120K grants or awards now have a Crossref DOI, and the planned transition of the Open Funder Registry into ROR.
• Recruitment of a Community Manager to focus on working with publishers and editors, including on ISR (that’s me!), and recruitment of a Technical Community Manager to enable greater use of our APIs.

Amanda highlighted that all Crossref members should be using ROR IDs to provide affiliations for authors (along with ORCID iDs) in their Crossref metadata. She also shared some latest examples of community behaviours that we have seen, such as requests from authors to delete records of works that were published without their permission, title ownership disputes between publishers, and the recent instance of sneaked references.

Ivan Oransky, co-founder of Retraction Watch, and Lena Stoll, Product Manager at Crossref, were next, and they spoke about the future of the Retraction Watch database, and about the Crossmark service. After this, some of the other roundtable participants shared initiatives that they have undertaken that support ISR:

• Jodi Schneider from the University of Illinois Urbana-Champaign spoke about NISO’s CREC Working Group that has created a Recommended Practice that should be followed by relevant stakeholders for communicating retracted research (Crossref’s Director of Product Rachael Lammey was the co-chair of that group).
• Kihong Kim from the Korean Council of Science Editors shared information about the workshops that the Council has organised for researchers on publishing in journals.
• Alberto Martín-Martín from Universidad de Granada presented his thoughts on how to reconcile the publishing system and the institutional view of tracking research outputs.
• Bianca Kramer from Sesame Science spoke about her analysis of and the implications of sneaked references, duplicate references, and missing references for citation integrity.
• Joris van Rossum from STM Solutions spoke about the STM Integrity Hub and the integrity tools that are being developed in collaboration with some publishers.

Some of the most valuable reflections stemmed from discussions in small groups on these three key questions:

1. What value do you see in the integrity and completeness of the scholarly record in the way you operate? How do you contribute to it? How can it support you to achieve your own goals?
2. Are you aware of Crossref services? What are the barriers to more uptake? What are the challenges and opportunities?
3. What information is essential and nice to have for you in the scholarly records to support trust signalling and ascertaining trustworthiness?

As groups shared their discussions, a few themes became apparent that I would like to elaborate on further.

What is “complete”?

Given the prompt to talk about the value of completeness of the scholarly record, an immediate reaction at most tables was: how much metadata qualifies as “complete” metadata? Can the scholarly record be considered complete if some publishers or journals do not use Crossref? What is the optimum level of metadata that should be deposited by members - should a minimum data standard be defined by disciplines, or should there be standard data requirements for all? The composition of metadata appears to change over time, too, as the processes change and our ability to record their facets increases. While there were spirited discussions about what constitutes a complete scholarly record, everyone agreed that “completeness” of metadata, as much as is possible, should be the aim. Unambiguous and consistent standards may help with this, for example, the Metadata 20/20 community creation of principles and best practices, and potentially also using a set of recognition standards and reproducibility badges.

Global participation is equally important for a truly “complete” scholarly record. In order to enable as many in the scholarly community as possible to participate in Crossref services and metadata, Crossref launched the Global Equitable Membership (GEM) program in 2023. Under this initiative, membership and content registration fees are waived off for members from the least economically advantaged countries. We are seeing first signs that this initiative meaningfully lowers the barriers to participation for organisations based in those countries, and allows the global community to contribute towards the building of a comprehensive research ecosystem.

At the end of the day, it is important to recognize that rich metadata is crucial because it can be used for all kinds of analysis, which in turn can drive decision-making. Even if some of the metadata components are sporadically missing, that could be acceptable, because every piece of data counts!

Corrections and Retractions

Similar to last year, retractions and corrections continued to be a topic of great interest in this year’s roundtable. This was not surprising given their relevance as trust indicators as well as the recent development with the acquisition of the Retraction Watch database by Crossref. Having heard from Ivan about the Retraction Watch taxonomy of reasons for retractions and the metadata included in the database, participants expressed the need to investigate this taxonomy as a community standard. While the Retraction Watch taxonomy is not widely known, we at Crossref are working to map the Crossmark taxonomy with the Retraction Watch taxonomy, which will enable complete integration of the Retraction Watch database with the Crossref database.

It would also be useful to add more information to retraction notices. Having more information about the reasons for retraction will not only destigmatize retractions, but certain additional information, such as submission dates for those outputs, might help with ethical investigations to determine whether manuscripts were being submitted to multiple publishers simultaneously.

On the topic of retractions, another aspect that came up in the room was about incentives for researchers to publish as much and as quickly as possible. If researchers indulge in unethical publishing practices due to this pressure to publish, that is hugely detrimental to the cause of research integrity and to the progress of scientific research in general. However, there is a distinction to be made between the integrity of the research and the integrity of the scholarly record - unethical research and publishing practices, including but not limited to data falsification, fabrication, and plagiarism, affect research integrity while integrity of the scholarly record is affected by unavailability of metadata, outdated metadata, incomplete metadata records, and incorrect metadata (e.g. as seen in the case of sneaked references).

There was a lot of discussion about Crossmark, a cross-platform service provided by Crossref that allows readers to discover whether an item has been updated, corrected, or retracted just by clicking a button that is standardised across publication platforms. While most participants acknowledged its importance, they also pointed out that its uptake has been limited and publishers do not use it as much, perhaps because it is difficult to implement and there’s a matter of providing more clarity about it to the readers. There were suggestions to add a notification system to Crossmark such that every time a published output is retracted, a notification goes out. This seemed of particular interest to funders, whose grievance was that they are usually the last to find out when research that they have funded is retracted. They would welcome notifications that would alert them to such events.

We already have plans to consult with the community more specifically about what changes they’d like to see to Crossmark that will enable them to implement it easily and use it more frequently. Take a look at this thread on our community forum and add your thoughts for our next steps on Crossmark.

The importance of education

There was an overwhelming sentiment that there was a need for collective arbitration of research integrity issues. However, everyone recognized that this is not a role for Crossref. We can act as a “trust broker” by bridging different metadata and identifiers that otherwise might not interact, creating a network of research outputs whose credibility can be verified by others. Many participants called for Crossref to increase its efforts in educating community members about the importance of metadata and how different pieces can be linked together to make meaningful connections.

Research practices vary between countries, and between institutions. Correspondingly, the metadata being provided by diverse Crossref members may also vary. There is an opportunity here for the global research community to work together to increase awareness about ethical standards, so that a lack of specific metadata or its variances (e.g. unusually formatted metadata, or non-standard metadata fields) may not be construed as “lower quality” metadata. Many felt that the greatest need for education about metadata is for the academic community – although individual researchers contribute a wealth of metadata associated with any published research output, they do not necessarily understand how metadata contributes to the completeness of the scholarly record. There is a further opportunity to talk to the academic community about how different metadata components link together to form a rich network, supporting visibility and confidence in their work. A greater awareness about these topics is likely to encourage researchers to provide more metadata and identifiers.

While most participants at the roundtable event agreed about the need for this conversation and the educational opportunities here, if Crossref were to lead these efforts, it would represent, in some eyes, a diversion from its mission. We do have several initiatives already to support our communities. As part of the Crossref Ambassadors program, volunteers from the international scholarly community who believe in Crossref’s mission liaise with our team to conduct training in their communities about using Crossref services and, generally, about the importance of metadata. In 2023, we also launched a new online public forum, the PLACE, in collaboration with the Committee on Publication Ethics (COPE), the Directory of Open Access Journals (DOAJ), and the Open Access Scholarly Publishers Association (OASPA). This forum is a place where new publishers can connect with these organisations and learn about best practices in scholarly publishing via discussion posts and by asking questions, as they get started. Another initiative that is designed to help new Crossref members is the “Managed Member Journey”: as members join and move through the various stages of membership, key information is shared with them during each of these stages in the form of triggered automated emails, web pages, and webinars.

While Crossref’s direct interactions with researchers are limited, we welcome the community’s recognition of the need to raise awareness about these matters. We have started engaging more closely with the reporters of metadata issues, in many cases investigators and ‘sleuths’ in the area of research integrity, and plan some closer collaborations with this group in 2024. We are open to supporting community efforts to inform other stakeholders about the importance and uses of metadata.

Incentives for the community

Another theme that was heard repeatedly was “incentives”: incentives for researchers to contribute to a “complete” scholarly record, incentives for publishers to improve metadata, and incentives for everyone to report on and register retractions.

As I mentioned before, a shared sentiment is that researchers may not be aware of the value of rich metadata. While more publications, increased citations, and greater grant funding are some examples of incentives that are part of the current academic settings, the right incentives probably do not exist for researchers to provide complete metadata. With the diverse set of participants present at this meeting, some groups also discussed how the current research assessment system can change to incorporate other metrics, perhaps those based on open science and open data.

What could be the incentives for publishers to improve the metadata collected and deposited by them? One suggestion was that clearly defined benefits of rich metadata can incentivise publishers. Being aware of what funders are mandating, can be another incentive. On the same note, funders will benefit from knowing what metadata is being provided by publishers. This metadata is available through our open API, and nine key checks on members’ activity are available through our public Participation Reports.

Retractions featured again in the discussion on the topic of incentives. As shared by Ivan, retractions are on the rise every year, with about 43k retractions currently in the Retraction Watch database. On the other hand, retractions registered in Crossmark at the time of the meeting numbered just 14k and have recently jumped up to 25k thanks to Hindawi/Wiley’s dedication to good open metadata. Besides the fact that the uptake of Crossmark by Crossref members is limited, another reason for the low number of retractions being registered is the associated stigma. Corrections and errata are usually conflated with retractions, and all these terms, which represent different kinds of updates that may happen to a published item, have a stigma associated with them in the academic community. There is a need to destigmatize retractions, and perhaps incentivize them by noting that these updates are essential to uphold the integrity of the scholarly record and to highlight the publishers that are showing leadership in addressing the issues openly through up-to-date Crossref metadata.

What metadata is nice to have in the scholarly record?

We asked everyone what information they think is essential as well as “nice to have” in the scholarly record to support trust signalling, and we heard a range of answers. Peer review information was recognized to be important. This would include data on who the peer reviewers were and standard peer review terminology that has been published by NISO. More generally, as much metadata as possible about the main actors of the peer review process was considered important - such as designating who the corresponding author is, and who the handling editor or the decision-making editor was.

As special issues led by guest editors in journals have been brought to the attention of late due to the uncovering of irregularities in some of them, one of the first suggestions in this context was more metadata about special issues. Participants thought that it would be useful to collect and distribute information on handling/guest editors of special issues, peer reviewers, as well as submission and acceptance dates. Recently, COPE has released guidance on “best practices for guest-edited collections” , highlighting that this topic looms at the forefront for the scholarly information industry.

Adding information on ethical approvals provided by institutional review boards would add more nuance to the research outputs. Metadata about clinical trials helps to add transparency to research in a field, where reproducibility is of primary importance. Conflicts of interest are another factor that could be a cause of concern if not reported accurately; these declarations were mentioned by the participants as important for signalling trust.

Recognizing that it is the relationships in the metadata that add context to research output, participants echoed that better interlinking between preprints and their published versions is required. To aid with all of this, it has been suggested that a complete list of all metadata that can be deposited with Crossref be made available in a simple format, so that members have more visibility about all the possibilities that exist for providing metadata.

Next steps

We asked all participants if the discussions prompted them to plan to take any actions in the near future. Several attendees reflected that the discussion encouraged them to go back and review the metadata that they are depositing with Crossref, and how they can make more use of the data openly available from Crossref. We also heard how some found training opportunities therein - discussion points from the event could be included in workshops for affiliated researchers, and in COPE guidance for members. As encouraged by members of the NISO’s CREC Working Group, some participants were looking to respond in the (then open) consultations of the draft Recommended Practice, NISO RP-45-202X, Communication of Retractions, Removals, and Expressions of Concern (CREC). One message resonated loud and clear: preserving the integrity of the scholarly record cannot be a lone endeavour and has to be a community effort. Attendees expressed their commitment to continue these conversations, with the next most opportune time being at the STM week. Everyone recognised that collaboration in this space is the need of the hour: facilitating information and data sharing across all the players in the ecosystem would be crucial to progressing this topic. As Bianca Kramer declared during her presentation, “I am committed to using only open data in my research, as access to data is important for the community to detect problems at scale”.

At our end, we are looking to act on suggestions that are specific to Crossref:

Consultation with the community about Crossmark

One of the first things that we are doing in early 2024 is to consult with our community about the developments needed in the Crossmark service. Our key aim with this exercise would be to understand how we can enable a more effective uptake of this service so that Crossref members can easily fulfil their obligation of keeping their records updated. We are keen to understand what we can do to help our members to send us metadata about updates to an output, and how we can help downstream services that use this data. Insights from this consultation will also help inform how the Retraction Watch data can be most effectively integrated into Crossmark and communicated to users. Please visit the discussion and add your thoughts here: https://community.crossref.org/t/communicating-post-publication-updates-inviting-feedback-on-the-next-steps-for-crossmark/.

Development of resources for using our API

As there is clearly no dearth of metadata components that the community thinks would be “nice to have” for signalling trust, it is equally important to equip users and downstream service providers to be able to access the rich metadata that is available with Crossref. This rich metadata opens up new avenues for the development of services and resources that can benefit the scholarly community. On account of this, we plan to prioritise development of resources for using Crossref APIs. These efforts would include making available workbooks with a variety of API use cases - ranging from how to use basic API queries, to how to use APIs for obtaining grant information or for obtaining citation data and so on, as well as retrieving corrections, retractions, and update information, especially when the Retraction Watch dataset merges in with the rest of the Crossref metadata.

Working group to facilitate community efforts for preserving ISR

We are looking to set up a working group that will facilitate the various stakeholders in the scholarly ecosystem to work together towards preserving the integrity of the scholarly record. One direction for the group could be to consider the role and impact of Crossref metadata in ISR. Another area of focus will be to enrich information about retractions, corrections, and expressions of concern. Raising industry-wide awareness about the current concerns in upholding the integrity of the scholarly record, and how comprehensive metadata can act as markers of trust about research output, would be another focal point.

Continued community outreach

We will continue our efforts to engage with the community on the very important issues surrounding ISR. We are particularly keen to redouble our efforts to include more funders and institutions in these conversations. Preserving the integrity of the scholarly record needs to be a truly inclusive effort and will benefit from diverse voices in the community. With that in mind, consulting with the community in Asia is next on our radar.

We look forward to working with the community further on this important topic - if you are keen to participate in these discussions and want to contribute towards preserving the integrity of the scholarly record, we would love to hear from you. Please write to us at feedback@crossref.org if you have any suggestions on this topic.

Appendix: Participant list