2018 on Crossref

LIVE18: Roaring attendees, incomplete zebras, and missing tablecloths

Rosa Morais Clark — Thu, 27 Dec 2018 00:00:00 +0000

Running a smooth event is always the goal, but not always the case! No matter how well managed an event is, there is always a chance that things will not go according to plan. And so it was with LIVE18.

For the first day we were without the tablecloths we had ordered, which actually gave the room quite a nice, but unintentional, ‘rustic’ look. When they finally did arrive the following day, we realized we preferred the rustic look! Some of the merchandise we had prepared ended up sitting in Canadian Customs for a day and a half, which meant they arrived to us halfway through the first day of the event. Luckily attendees were distracted by the very cool ‘I heart metadata’ bags and didn’t seem to notice.

Unfortunately a significant number of registrants also had problems with Canadian regulations: they were denied visas to enter the country. Despite always trying to choose countries with international airport hubs and a welcoming policy, this was an unforseen blow.

But from setting up to take down, LIVE18 was truly a team effort. Even though many Crossref staff had traveled far and wide to get there, they all rallied to help the night before—hauling boxes through the streets of Toronto, stuffing attendee bags, hanging signage, and moving furniture around until 11:30 pm.

Because of these efforts—and despite the glitches—Crossref LIVE18 was a great success.

How good is your metadata?

That was the framing question at Crossref LIVE 18 in Toronto which this year focused on all things metadata. Over the course of the two-day event, we heard from guest speakers on the importance of collaboration, the significance of metadata to metrics, and what good metadata looks like. In our usual lively way, Crossref staff introduced a variety of new services, initiatives, and collaborations.

Crossref LIVE is helping surface key issues in the cleanup of metadata mismatch, after decades of the industry working in silos. I applaud Crossref for doing this. It’s great that we’re considering how to change the way we work and collaborate as an industry to make sure that we don’t run into metadata issues in this way again.

- Keynote speaker, Kristen Ratan, Co-Founder of the Collaborative Knowledge Foundation (Coko)

In her keynote speech, ‘Publishing Infrastructure: The Good, The Bad, and The Expensive’, Coko’s Kristen Ratan challenged the industry to rethink its slow, inefficient, and expensive resignation to infrastructure; and instead consider how a collaborative approach to sharing expertise in developing community-owned infrastructure could be faster, more flexible, and less costly.

View Kristen’s talk, The Good, The Bad, and The Expensive

The collaborations

Collaboration was a running theme at LIVE18. Geoffrey Bilder provided an overview of Crossref’s selective collaborations; DataCite’s Patricia Cruse introduced ROR, the community project to develop an open, sustainable, usable and unique identifier for every research organisation in the world—and she got the crowd really engaged at the beginning of her talk by encouraging us all to ROAR out loud!; Clare Dean and Ravit David sketched out the evolution of Metadata 2020, and Shelley Stall from the AGU introduced the ways they are urging the scientific community to adopt FAIR data principles (using her first data collection as an 11-year-old as an example!)

View Geoffrey’s talk, How Crossref (selectively) collaborates with others
View Patricia’s talk, ROR: The Research Oragnization Registry (Roar!) 🦁
View Clare and Ravit’s talk, Metadata 2020: This talk is sooo meta
View Shelley’s talk, My first data collection: Was it FAIR?

The solutions

Patricia Feeney, in the newly-created role of Head of Metadata, used a zebra to illustrate that not all of a publisher’s metadata is deposited with Crossref.
View Patricia’s talk, I am the boss of your Metadata (this one has the zebras) and also her talk on New resource/record types in the works at Crossref.

New tools

Jennifer Lin introduced Event Data, the new API that Crossref and DataCite have built together, enabling organisations to capture what happens to a DOI, including all of the places it is mentioned and links from/to. She also talked about Participation Reports, the new open dashboard to help members evaluate the completeness of their own metadata deposited with Crossref.

View Jennifer’s talks on Event Data, and Simplifying our services

The community

We also heard from the community. Paul Dlug from the American Physical Society boldly gave his view on ‘Why Crossref sucks’, and, with a view to helping Crossref improve in key areas, surfaced issues that members struggle with. Ed Pentz, Executive Director, provided an overview of the direction that Crossref is headed towards. Ginny Hendricks, Director of Member & Community Outreach, updated everyone on the expanding Crossref community and all the outreach activities her team conducts to engage them. Isaac Farley, new Technical Support Manager in the community team, told of his vision for moving to a more public, open, support model. Lisa Hart, Director of Finance & Operations announcing the results of our members votes in this year’s board election.

View Paul’s talk, Crossref sucks and how to cope!
View Ed’s talk, Our strategic direction
View Ginny’s talk, Expanding our constituencies
View Isaac’s talk, Open Support: From 1:1 to everyone

The perspectives

Guest speakers provided a range of fascinating perspectives from across scholarly communications. Graham Nott, who works with eLife, outlined how they were making their JATS to Crossref schema conversion tool openly available to the community for use. Jodi Schneider, Assistant Professor of Information Sciences at the University of Illinois at Urbana-Champaign, gave us an in-depth look at problem citations, with a focus on retractions. Bianca Kramer from Utrecht University discussed Crossref metadata use in an open scholarly ecosystem. Stephanie Haustein from the University of Ottawa gave a researcher perspective on the problems with traditional journal metrics, and how they are dependent on metadata, which is essentially flawed. She outlined her efforts to increase metrics literacy, putting metrics in context with comprehensive metadata. Geoffrey Bilder talked about Dominika’s work to evaluate our reference matching, and finally closed the show discussing the role of metadata in creating a provenance infrastructure, providing trustworthiness which is essential to progress the scholarly research cycle.

View Graham’s talk, JATS at eLife
View Jodi’s talk, Trouble at The Academy: Problem Citations
View Bianca’s talk, DOIs for whom? Crossref metadata in an open scholarly ecosystem
View Stephanie’s talk, Good metadata + metrics literacy = better academia
View Geoffrey’s talks on Reference matching, and Metadata as a signal of trust.

As LIVE18 came to a close we took the opportunity to acknowledge and thank everyone once again for helping us reach the milestone of 100 million registered content items this September. Everyone took to the stage and waved their Crossref Bigger Ambitions flags.

Thank you to everyone who participated in the event. Please save the dates for LIVE19 in Europe on 13-14 November, 2019!

Reference matching: for real this time

Dominika Tkaczyk — Tue, 18 Dec 2018 00:00:00 +0000

In my previous blog post, Matchmaker, matchmaker, make me a match, I compared four approaches for reference matching. The comparison was done using a dataset composed of automatically-generated reference strings. Now it’s time for the matching algorithms to face the real enemy: the unstructured reference strings deposited with Crossref by some members. Are the matching algorithms ready for this challenge? Which algorithm will prove worthy of becoming the guardian of the mighty citation network? Buckle up and enjoy our second matching battle!

TL;DR

I evaluated and compared four reference matching approaches: the legacy approach based on reference parsing, and three variants of search-based matching.
The dataset comprises 2,000 unstructured reference strings from the Crossref metadata.
The metrics are precision and recall calculated over the citation links. I also use F1 as a standard single-number metric that combines precision and recall, weighing them equally.
The best variant of search-based matching outperforms the legacy approach in F1 (96.3% vs. 92.5%), with the precision worse by only 0.9% (98.09% vs. 98.95%), and the recall better by 8.9% (94.56% vs. 86.85%).
Common causes of SBMV’s errors are: incomplete/erroneous metadata of the target documents, and noise in the reference strings.
The results reported here generalize to the subset of references in Crossref that are deposited without the target DOI and are present in the form of unstructured strings.

Introduction

In reference matching, we try to find the DOI of the document referenced by a given input reference. The input reference can have a structured form (a collection of metadata fields) and/or an unstructured form (a string formatted in a certain citation style).

In my previous blog post, I used reference strings generated automatically to compare four matching algorithms: Crossref’s legacy approach based on reference parsing and three variations of search-based matching. The best algorithm turned out to be Search-Based Matching with Validation (SBMV). SBMV uses our REST API’s bibliographic search function to select the candidate target documents, and a separate validation-scoring procedure to choose the final target document. The legacy approach and SBMV achieved very similar average precision, and SBMV was much better in average recall.

This comparison had important limitations, which affect the interpretation of these results.

First of all, the reference strings in the dataset might be too perfect. Since they were generated automatically from the Crossref metadata records, any piece of information present in the string, such as the title or the name of the author, will exactly match the information in Crossref’s metadata. In such a case, a matcher comparing the string against the record can simply apply exact matching and everything should be fine.

In real life, however, we should expect all sorts of errors and noise in the reference strings. For example, a string might have been manually typed by a human, so it can have typos. The string might have been scraped from the PDF file, in which case it could have unusual unicode characters, ligatures or missing and extra spaces. A string can also have typical OCR errors, if it was extracted from a scan.

These problems are typical for messy real-life data, and our matching algorithms should be robust enough to handle them. However, when we evaluate and compare approaches using the perfect reference strings, the results won’t tell us how well the algorithms handle harder, noisy cases. After all, even if you repeatedly win chess games against your father, it doesn’t mean you will likely defeat Garry Kasparov (unless, of course, you are Garry Kasparov’s child, in which case, please pass on our regards to your dad!).

Even though I attempted to make the data more similar to the noisy real-life data by simulating some of the possible errors (typos, missing/extra spaces) in two styles, this might not be enough. We simply don’t know the typical distribution of the errors, or even what all the possible errors are, so our data was probably still far from the real, noisy reference strings.

The differences in the distributions are a second major issue with the previous experiment. To build the dataset, I used a random sample from Crossref metadata, so the distribution of the cited item types (journal paper, conference proceeding, book chapter, etc.) reflects the overall distribution in our collection. However, the distribution in real life might be different if, for example, journal papers are on average cited more often than conference proceedings.

Similarly, the distribution of the citation styles is most likely different. To generate the reference strings, I used 11 styles distributed uniformly, while the real distribution most likely contains more styles and is skewed.

All these issues can be summarized as: the data used in my previous experiment is different from the data our matching algorithms have to deal with in the production system. Why is this important? Because in such a case, the evaluation results do not reflect the real performance in our system, just like the child’s score on the math exam says nothing about their score on the history test. We can hope my previous results accurately showed the strengths and weaknesses of each algorithm, but the estimations could be far off.

So, can we do better? Sure!

This time, instead of automatically-generated reference strings, I will use real reference strings found in the Crossref metadata. This will give us a much better picture of the matching algorithms and their real-life performance.

Evaluation

This time the evaluation dataset is composed of 2,000 unstructured reference strings from the Crossref metadata, along with the target true DOIs. The dataset was prepared mostly manually:

First, I drew a random sample of 100,000 metadata records from the system.
Second, I iterated over all sampled items, and extracted those unstructured reference strings, that do not have the DOI provided by the member.
Next, I randomly sampled 2,000 reference strings.
Finally, I assigned a target DOI (or null) to each reference string. This was done by verifying DOIs returned by the algorithms and/or manual searching.

The metrics this time are based on the citation links. A citation link points from the reference (or the document containing the reference) to the referenced (target) document.

When we apply a matching algorithm to a set of reference strings in our collection, we get a set of citation links between our documents. I will call those citation links returned links.

On the other hand, in our collection we have real, true links between the documents. In the best-case scenario, the set of true links and the set of returned links are identical. But we don’t live in a perfect world and our matching algorithms make mistakes.

To measure how close the returned links are to the true links, I used precision, recall and F1. This time they are calculated over all citation links in the dataset. More specifically:

Precision is the fraction of the returned links that are correct. Precision answers the question: if I see a citation link A->B in the output of a matcher, how certain can I be that paper A actually cites paper B?
Recall is the percentage of true links that were returned by the algorithm. Recall answers the question: if paper A cites paper B and B is in the collection, how certain can I be that the matcher’s output contains the citation link A->B?
F1 is the harmonic mean of precision and recall.

In the previous experiment, I also used precision, recall and F1, but they were calculated for each target document and then averaged. This time precision, recall and F1 are not averaged but simply calculated over all citation links. This is a more natural approach, since now the dataset comprises isolated reference strings rather than target documents, and in practice each target document has at most one incoming reference.

I tested the same four approaches as before:

the legacy approach, based on reference parsing
SBM with a simple threshold, which searches for the reference string in the search engine and returns the first hit, if its relevance score exceeds the predefined threshold
SBM with a normalized threshold, which searches for the reference string in the search engine and returns the first hit, if its relevance score divided by the string length exceeds the predefined threshold
SBMV, which first applies SBM with a normalized threshold to select a number of candidate items, and a separate validation procedure is used to select the final target item

All the thresholds are parameters which have to be set prior to the matching. The thresholds used in the experiments were chosen using a separate dataset, as the values maximizing the F1 of each algorithm.

Results

The plot shows the overall results of all tested approaches:

The exact values are also given in the table (the best result for each metric is bolded):

	precision	recall	F1
legacy approach	0.9895	0.8685	0.9251
SBM (simple threshold)	0.8686	0.8191	0.8431
SBM (normalized threshold)	0.7712	0.9121	0.8358
SBMV	0.9809	0.9456	0.9629

As we can see, the legacy approach is the best in precision, slightly outperforming SBMV. In recall, SBMV is clearly the best, which also decided about its victory over the legacy approach in F1.

How do these results compare to the results from my previous blog post? The overall trends (the legacy approach slightly outperforms SBMV in precision, and SBMV outperforms the legacy approach in recall and F1) are the same. The most important differences are: 1) on the real dataset SBM without validation is worse than the legacy approach, and 2) this time the algorithms achieved much higher recall. These differences are most likely related to the difference in data distributions explained before.

SBMV’s strengths and weaknesses

Let’s look at a few example cases where SBMV successfully returned the correct DOI, while the legacy approach failed.

Lundqvist D, Flykt A, Ohman A: The Karolinska Directed Emotional Faces - KDEF, CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet. 1998

matched to https://doi.org/10.1037/t27732-000

The target item is a dataset, which means unusual metadata fields and an unusual reference string.

Schminck, A. , ‘The Beginnings and Origins of the “Macedonian” Dynasty’ in J. Burke and R. Scott , eds., Byzantine Macedonia: Identity, Image and History (Melbourne, 2000), 61–8.

matched to https://doi.org/10.1163/9789004344730_006

This is an example of a book chapter. The reference string contains special quotes and dash characters.

R. Schneider,On the Aleksandrov-Fenchel inequality, inDiscrete Geometry and Convexity (J. E. Goodman, E. Lutwak, J. Malkevitch and R. Pollack, eds.), Annals of the New York Academy of Sciences440 (1985), 132–141.

matched to https://doi.org/10.1111/j.1749-6632.1985.tb14547.x

In this case, spaces are missing in the reference string, which might be problematic for the parsing.

R. B. Husar andE. M. Sparrow, Int. J. Heat Mass Transfer11, 1206 (1968).

matched to https://doi.org/10.1016/0017-9310(68)90036-7

This is another example of a reference string with missing spaces.

F. Cappello, A. Geist, W. Gropp, S. Kale, B. Kramer, and M. Snir. Toward exascale resilience: 2014 update. Supercomputing frontiers and innovations, 1(1), 2014.

matched to https://doi.org/10.14529/jsfi140101

In this case authors are missing in the Crossref metadata.

Li KZ, Shen XT, Li HJ, Zhang SY, Feng T, Zhang LL. Ablation of the Carbon/carbon Composite Nozzle-throats in a Small Solid Rocket Motor[J]. Carbon, 2011, 49: 1 208–1 215

matched to https://doi.org/10.1016/j.carbon.2010.11.037

Here we have unexpected spaces inside page numbers.

N. Kaloper, A. Lawrence and L. Sorbo, An Ignoble Approach to Large Field Inflation, JCAP 03 (2011) 023 [ arXiv:1101.0026 ] [ INSPIRE ].

matched to https://doi.org/10.1088/1475-7516/2011/03/023

In this case we have an acronym of the journal name and additional arXiv id.

KrönerE. ?Stress space and strain space continuum mechanics?, Phys. Stat. Sol. (b), 144 (1987) 39?44.

matched to https://doi.org/10.1002/pssb.2221440104

This reference string has a missing space, a missing word in the title, and incorrectly encoded special characters.

Suyemoto K. L., (1998) The functions of self-mutilationClinical Psychology Review 18(5): 531–554

matched to https://doi.org/10.1016/s0272-7358(97)00105-0

In this case the space is missing between the title and the journal name.

Ono , N. 2011 Stable and fast update rules for independent vector analysis based on auxiliary function technique Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics 189 192

matched to https://doi.org/10.1109/aspaa.2011.6082320

The parsing can also have problems with missing punctuation, like in this case.

Hybertsen M.S., Witzigmann B., Alam M.A., Smith R.K. (2002) 1 113

matched to https://doi.org/10.1023/a:1020732215449

In this case both title and journal name are missing from the reference string.

We can see from these examples that SBMV is fairly robust and able to deal with a small amount of noise in the metadata and reference strings.

What about the errors SBMV made? From the perspective of citation links, we have two types of errors:

False positives: incorrect links returned by the algorithm.
False negatives: links that should have been returned but weren’t.

When we apply SBMV instead of the legacy approach, the fraction of false positives within the returned links increases from 1.05% to 1.91%, and the fraction of false negatives within the true links decreases from 13.15% to 5.44%. This means with SBMV:

1.91% of the links in the algorithm’s output are incorrect
5.44% of the true links are not returned by the algorithm

We can also classify all the references in the dataset into several categories, based on the values of true and returned DOIs:

We have the following categories:

References matched to correct DOIs (1129 cases, returned and true blue)
References correctly not matched to anything (791 cases, returned and true white)
References not matched to anything, when they should be (58 cases, returned white, true grey)
References matched to wrong DOIs (7 cases, returned red, true yellow)
References matched to something, when they shouldn’t be matched to anything (15 cases, returned black, true white)

Note that in terms of these categories, precision is equal to:

And recall is equal to:

What are the most common causes of SBMV’s errors?

Incomplete or incorrect Crossref metadata. Even a perfect reference string formatted in the most popular citation style will not be matched, if the target record in the Crossref collection has many missing or incorrect fields.
Similarly, missing or incorrect information in the reference string is very problematic for the matchers.
Errors/noise in the reference string, such as:
- HTML/XML markup not stripped from the string
- multiple references mixed in one string
- spacing issues and typos
In a few cases a document related to the real target was matched, such as the book instead of its chapter, or the conference proceedings paper instead of the thesis.

Limitations

The most important limitation is the size of the dataset. Every item had to be verified manually, which significantly limited the possibility of creating a large set and also using a lot of independent sets.

Finally, the numbers reported here still don’t reflect the overall precision and recall of the current links in the Crossref metadata. This is because:

we still use the legacy approach for matching,
some references are deposited along with the target DOIs and are not matched by Crossref, these links are not analyzed here, and
in Crossref we have both unstructured and structured references, and in this experiment only the unstructured ones were tested.

What’s next?

The next experiment will be related to the structured references. Similarly as here, I will try to estimate the performance of the search-based matching approach and compare it to the performance of the legacy approach.

The evaluation framework, evaluation data and experiments related to the reference matching are available in the repository https://github.com/CrossRef/reference-matching-evaluation. Future experiments will be added there as well.

https://github.com/CrossRef/reference-matching-evaluation also contains the Python implementation of the SBMV algorithm. The Java implementation of SBMV is available in the repository https://gitlab.com/crossref/search_based_reference_matcher.

Phew - its been quite a year

Ed Pentz — Thu, 13 Dec 2018 00:00:00 +0000

As the end of the year approaches it’s useful to look back and reflect on what we’ve achieved over the last 12 months—a lot! To be honest, there were some things we didn’t get done—or didn’t make as much progress with as we hoped—but that happens when you have an ambitious agenda. However, we also got some things done that we didn’t expect to or that weren’t even on our radar at the end of 2017—this is inevitable as the research and scholarly communications landscape is rapidly changing.

In my blog post from the beginning of the year, the key projects I highlighted were Metadata Plus, Event Data, organisation IDs, Grant IDs and Metadata 2020, and that richer metadata and more record types were key goals. We did make very good progress on all of these projects as reported below.

For 2018 we were operating in the framework of the four strategic themes, or areas of focus, developed by the board and staff. These are: 1) Simplifying and enriching our services; 2) Improving our metadata; 3) Expanding constituencies, and 4) Selectively collaborating and partnering. These themes will also be guiding us in 2019.

Simplifying and enriching our services

Upgrading our tools

Over the past year, we’ve been busy streamlining our processes, developing new tools and adding new services. A key new tool is Metadata Manager which supports the Content Registration service by offering a simpler, more user-friendly, non-technical way to register and update metadata. It provides lots of context-sensitive help, registers content immediately, in real time, and provides guidance on how to make corrections—thereby ensuring each deposit is successful. Metadata Manager currently supports journal deposits (we would have liked to add more in 2018) but we will be adding other record types in 2019.

Upgrading our services

Crossref metadata has always been open through a number of interfaces without restriction, but this year we introduced an option for extra support and functionality, through Metadata Plus. Metadata Plus provides guaranteed uptime, snapshots of the complete set of metadata and enhanced support for organisations (members or not) that want to use Crossref metadata in their own services and systems.

Improving the member experience: New membership terms

This year we began to redesign the member experience and have made a lot of improvements to the sign-up and onboarding process, the most significant of which is the new click-through membership terms, introduced in July for new members and coming into effect for existing members in March 2019, which is proving to be a huge time saver for both our members and our team.

Improving our metadata

Our objective this year was to better communicate what metadata best practice is, to equip our members with all the data and tools they need to meet this best practice, and to achieve closer cooperation from service providers.

Best practice tools: Participation Reports

Released in Beta in August this year, Participation Reports provides a dashboard that gives a clear picture of the metadata that each member provides. This is a useful visualization of metadata that has long been available via our public REST API. Members can see where the gaps in the metadata are and get information on how to fill those gaps.

Communicating metadata best practice: Data Citations

The importance of linking data with literature can’t be understated. Research integrity and reproducibility depend on it. We’re committed to exposing the links between the literature and the data or software that supports it, and earlier this year we partnered with DataCite to make this a reality. All the data citations coming in from Crossref and DataCite are being pulled into Event Data.

Equipping members with all the data: Event Data

Event Data reached technical readiness. Event Data captures and records “events” such as comments, links, shares, bookmarks, and references. It provides open, transparent, and traceable information about the provenance and context of every event.

Expand constituencies

Crossref currently has 15,000 members in 140 countries. With that comes the need to increasingly and proactively work with emerging markets as they start to share research outputs globally.

Ambassador program

The Crossref Ambassador program launched in January and now has a team of 16 trusted contacts who work within our communities (as librarians, researchers, publishers, and innovators) around the world. They share great enthusiasm and belief in our work. We provide them with training and support, and they help us improve education about global research infrastructure in general and the opportunities that are enabled through richer metadata.

Funders and grant identifiers

I’m very happy to report that the Crossref board approved grants as a new record/resource type to be rolled out in 2019 - we made faster progress on this than expected. The proposal for grant identifiers was developed by staff in collaboration with the Crossref Funder Advisory Group and the Membership and Fees Committee. This means that funders will be joining Crossref and registering a standard set of metadata and a persistence identifier - a DOI - for their grants.

Collaborate and partner

So that our alliances with others have the greatest impact, we have aligned our strategic plans for scholarly infrastructure with others. Some of these alliances are led or driven by Crossref and with others we are involved but not leading.

ROR

We are working with the California Digital Library, DataCite and Digital Science as the Steering group for ROR - the Research Organization Registry - which is a new, community-led project that is developing an open, sustainable, usable, and unique identifier for research organisations based on the work done by the organisation Identifier Working Group in 2017 and 2018.

Metadata 2020

Metadata 2020 is a collaboration that advocates richer, connected, and reusable, open metadata for all research outputs, which will advance scholarly pursuits for the benefit of society. Over 140 volunteers—including publishers, librarians, researchers, platforms/tools, and other stakeholders—from 86 organisations, are working in six project groups. The projects are very strategically focused, looking at key issues like researcher communications, incentives, and shared best practices.

I can’t close off the year without mentioning the incredible milestone we reached this September when the 100th million content item was registered with Crossref. This was made possible by our members’ and the wider community’s commitment and contribution, so once again, thank you.

Roll on 2019!

Newly approved membership terms will replace existing agreement

Ginny Hendricks — Wed, 05 Dec 2018 00:00:00 +0000

In its July 2018 meeting, the Crossref Board voted unanimously to approve and introduce a new set of membership terms. At the same meeting, the board also voted to change the description of membership eligibility in our Bylaws, officially broadening our remit beyond publishers, in line with current practice and positioning us for future growth.

Tl;dr

It’s a very good thing to have clearer terms; we want everyone to understand what Crossref is about and what you’re getting into. It’s a material change so we will be notifying members by direct email in December. Nobody needs to sign anything as the new terms are not signed, but are click-through acceptances on application, and that process is already in effect for new applicants. The new terms come into effect on 1st March 2019 for existing members and no action is needed.

If you’re a sponsored member you’ll have a slightly adapted message soon as we work with your sponsor.
If you’re an NGO or US State Actor you will receive a slightly adapted message.

This post is for background explanation and information. We will email existing members directly, but no acceptance or signature—nor any action—will be needed.

Why are we updating the terms?

Being almost 20 years old the old agreement is out-of-date with current practice and technology, and has become quite long and confusing, especially for applicants for whom English is not their first language. Specific reasons include:

1. To improve efficiency

Over the years we’ve had feedback that our application process is too long and involved. The membership agreement used to be signed manually by each new Crossref member, often days after they applied. We also now process around 180 new members each month which is too many for a wholly manual process managed by just one person.

2. To clarify the wording

People would tell us that the agreement is too long and confusing, especially when English is not their first language. There are often questions about the “legalese” style of language that takes up too much time in back-and-forth discussions to ensure everyone has understood. Also, the main structure of the agreement has been in place for over a decade and needs updating to avoid confusion and to align with up-to-date language, services, technologies, and current practices.

3. To emphasize the community aspect and our members’ obligations

It is quite a commitment to participate fully in Crossref, and we want people to understand up-front what their obligations are as part of the collective membership. And also to realize what value they are receiving as well as contributing to other members. We needed clearer terms so that every organisation can understand what they are getting into.

Additionally, moving from signing contracts to click-through acceptance of standard terms emphasizes that Crossref is not a service provider or vendor. We are a not-for-profit community organisation. We don’t have the resources to negotiate and keep track of individual custom agreements.

What’s changing, step-by-step

We consulted with former and current legal counsel, the Membership & Fees Committee, and also with the M&F organisations individually. We have also absorbed a lot of feedback from many other members of all kinds and sizes.

For new members

The manually-signed membership agreement has already—for new members— been turned into a set of click-through terms that organisations agree to as part of the initial application process. It is no longer a separate document that needs to be signed or countersigned. This will simplify the application process for both new applicants and our staff.

For existing members

The new membership terms will come into effect for existing members on March 1st, 2019. Because this is a material change to the terms, we will be emailing members with more information but it’s important to note that no action is necessary from existing members. The new terms will replace the old terms automatically.

The table below sets out clause-by-clause the precise changes. Here is the 2018 membership agreement and the new terms in full.

The nitty-gritty details

Topic	New section	Old section	Summary of change(s)
Overall			Eliminates legalese in favor of plain English. Updates defined terms to current usage. Shifts from execution by signature to acceptance by affirmative action.
Introduction	Background	1	Updates description of Crossref’s activities to be current. Provides for a new applicant’s acceptance of Terms upon acceptance of application by Crossref and payment of first annual fee.
Members’ rights	1	2(a)	Streamlines wording; eliminates reference to right to recommend working committee members.
Members’ obligations	2	2(b)	Significant revision. Old 2(b) mentioned only payment of fees and appointment of a contact person. New Sec. 2 aims to capture all of a Member’s operational obligations in one place.
Metadata deposits	2(a), (b)	3(a)(i)	Updates language regarding metadata deposits to current terminology and practice.
Rights to content	2(c)	15	Streamlines wording.
Registering identifiers	2(d)	3(a)ii)	Streamlines the language around registering identifiers.
Linking	2(e)	3(a)(iii)	States, in clearer language, the obligation to embed identifiers.
Reference linking	2(f)	3(a)(iv)	Eliminates outdated provision on Cross-Linking; replaces with a best efforts covenant to engage in Reference Linking.
Display identifiers	2(g)	N/A	Adds an obligation to comply with Crossref’s display guidelines and ensure each identifier is hyperlinked to be citable.
Maintaining and updating metadata	2(h)	3(b)	Streamlines language. Adds obligation to maintain the URL and the accuracy of identifier data. Adds common examples of failure to maintain and update metadata.
Archiving	2(i)	3(d)	Adds link to examples of third-party archive providers. Adds option for Crossref to point to a “defunct DOI” page. Inserts best efforts obligation to contract with a third-party archive.
Content-specific obligations	2(j)	N/A	Adds reference to Crossref’s record type rules and obligation to comply.
Fees	3	2(b)	Old agreement referred generally to “all membership dues and any charges or fees as established by the Board from time to time and set forth on the PILA Site.” New Section 3 aims to summarize the categories of fees associated with membership, including a reference to service fees for optional services if and when elected by the Member. Adds Member obligation to cover wire transfer fees/other payment costs.
General license	4(a)	4	Clarifies that the license grant covers only metadata and identifiers “corresponding to such Member’s Content.”
Metadata rights & limitations	4(b)	5	Significantly streamlines wording.
Crossref’s IP	4(c)	6	Significantly streamlines wording.
Distribution of metadata	5	9(b)	Updates language regarding Crossref’s rights to distribute Metadata. Adds an explicit carveout for a Member’s reference distribution preference.
	N/A	7, 8, 9(a)	Deletes extensive provision relating to obsolete “Clean-Up” and “Reverse Look-Up” services. Deletes provisions relating to obsolete “caching and transfer” activities, and local hosting.
Use of marks	6	10	Substantially rewritten, including to reflect Crossref’s more permissive approach to use of its logo.
Maintenance of the Crossref Infrastructure	7	[No analog.]	Adds covenant of Crossref to maintain the Crossref Infrastructure.
Term	8	11	Eliminates the concept of automatically renewing 12-month terms. Replaces with a perpetual term that continues until superseded by an amended version.
Termination of membership	9(a)	11	Provides for termination by the member upon written notice, rather than 90 days’ written notice, to align with the Bylaws. Adds a for-cause termination right by the Member, and corresponding right to receive a refund of fees. Sets out certain bases for termination of membership by Crossref, consistent with the Bylaws.
Appeal rights	9(b)	13	No material change.
Effect of termination of membership	9(c)	12	Adds refund right for for-cause terminations.
Enforcement	10	13	Replaces “Crossref has the right but not the obligation to enforce the terms of this Agreement …” with “Crossref shall take reasonable steps to enforce these Terms … .”
Governing law; venue	11	14(a)	Keeps New York as choice of law, but moves forum to Boston, nearer to Crossref’s US location.
Disputes	12	14(b)	No material change (but note venue provision moved to 11(a)).
	N/A	15	Eliminates mutual “warranty” provision; addresses rights to content and anti-infringement under other provisions.
Indemnification	13	16	Removes concept that Member is indemnifying other Crossref Members. Streamlines and cleans up the indemnity language.
Limitation of Liability	14	17	Adds explicit reference to the Crossref Infrastructure.
Assignment	16(c)	22	Removed language providing that Crossref’s consent to assignment of the Terms shall not be unreasonably delayed or conditioned.
Amendment	18	2(c)	Old: “The Board shall have the power to modify the terms of this Agreement by publishing amended versions that will automatically supersede prior versions … . PILA will use its reasonable discretion in deciding if a modification is material, and if so will provide written notice” to the Member of the material changes. New: “These Terms may be amended by Crossref, via updated Terms posted on the Website and emailed to each Member not less than sixty (60) days prior to effectiveness. By using the Crossref Infrastructure after the effective date of any such amendment hereto, the Member accepts the amended Terms.”
Data privacy	19	N/A	Adds a GDPR-compliant privacy provision; adds a linked reference to Crossref’s new Privacy Policy.
Compliance	20	N/A	Adds a mutual compliance covenant and an OFAC/sanctions representation.
Various legal “boilerplate” terms (taxes, waiver, independent contractor	15-17	18-28	Streamlined; replaced with more contemporary formulations; eliminated some excess verbiage.

Thanks for reading this far!

Please contact our member experience team with any questions.

Updates to our by-laws

Lisa Hart Martin — Thu, 29 Nov 2018 00:00:00 +0000

Good governance is important and something that Crossref thinks about regularly so the board frequently discusses the topic, and this year even more so. At the November 2017 meeting there was a motion passed to create an ad-hoc Governance Committee to develop a set of governance-related questions/recommendations. The Committee has met regularly this year and the following questions are under deliberation regarding term limits, role of the Nominating Committee, implications of contested elections, and more.

The full motion to create the committee is:

The ad hoc Governance Committee should discuss and make specific recommendations (including where necessary proposing appropriate by-law amendments) about (i) the timing of the annual election of members and whether the newly elected Board can take office a fixed period after the election results are finalized; (ii) the role and responsibilities of the Nominating Committee and its relationship to the Board; (iii) the implications of having contested Board elections; (iv) the election of officers, Executive Committee members, and committee chairs, and v) options and required changes for board members to represent specific constituencies (e.g. based on membership types).

The Governance Committee members are:

Paul Peters (Hindawi and Board Chair)
Scott Delman (ACM and Board Treasurer)
Chris Shillum (Elsevier and Executive Committee member)
Mark Patterson (eLife)
Ed Pentz (Crossref Executive Director)
Lisa Hart (Crossref Finance & Operations Director)
Emily Cooke (Pierce Atwood, legal counsel).

The committee’s goal was to try to maintain and increase transparency; consider practicality and impact of any changes and ensure continuity and balance.

At the March meeting the committee provided an overview of the issues they had discussed. There was consensus to accept the committee’s recommendation to address all of the governance matters comprehensively at the July 2018 meeting.

Discussions resulted in two changes to our by-laws:

1. Membership eligibility

To provide clarity around membership qualification, we resolved to amend Article I Section 1 by replacing the text in its entirety with the following text:

Membership in Crossref shall be open to any organisation that publishes professional and scholarly materials and content and otherwise meets the terms and conditions of membership established from time to time by the Board of Directors, and to such other entities as the Board of Directors shall determine from time to time.

2. Start date of board terms

We also resolved to amend Article V Section 4 to replace the phrase “on the day after” with the phrase “during the next calendar quarter immediately following”. This allows the Board to meet directly ahead of Crossref’s Annual Meeting and Board election (from 2019) instead of directly after.

The first change captures the fact that we have a very broad community beyond what is seen as traditional publishers, who themselves do not solely identify as publishers anymore. It reflects how our membership has evolved, and also includes organisations that publish that aren’t publishers (universities, government agencies, etc.)

The second change was a practical one. As Crossref had its first contested election in 2017, and in 2018 as well, it seemed unreasonable to have a brand new Board meet the day after the election, especially when there is potential for officers to not be re-elected. The old by-laws were very specific about holding the Board meeting the day after the election. With this change, starting with the March meeting, the new Board will have a full calendar year of meetings, which seems more practical, and we will establish a process for the election of officers.

During that meeting we deliberated the following questions/recommendations raised by the committee:

Development of a policy on canvassing/campaigning by candidates in Board elections;
Development of policies on nominations to the positions of Chair, Treasurer, Executive Committee members, the Nominating Committee Chair, and the Audit Committee Chair;
Analysis of how best to achieve balance and representation on the Board going forward (designated seats and/or a binding Board remit to the Nominating Committee);
Analysis as to whether to impose term limits on board members;
Analysis as to how best to handle independent nominations to the Board (eliminate the option, or improve the process); and
Review of our governing documents’ provisions on vacancies, to confirm that the Board follows the required steps on the filling of vacancies.

At the November 2018 Board meeting—following Crossref LIVE18—there were two more amendments to the bylaws:

3. Removal of independant nominations

To remove Art. V II Sec. 3 on independent nominations. This change reflects the consensus that there is no need for independent nominations with the introduction of contested elections.

4. Membership start date of record

To amend Art. I Sec. 2 to amend language dealing with record date of membership. This is a practical change following the July 2018 introduction of new membership terms which are click-through online terms and don’t need counter-signatures.

The new Board will resume the discussion on designated seats at our March 2019 meeting.

Data Citation: what and how for publishers

Rachael Lammey — Fri, 23 Nov 2018 00:00:00 +0000

We’ve mentioned why data citation is important to the research community. Now it’s time to roll up our sleeves and get into the ‘how’. This part is important, as citing data in a standard way helps those citations be recognised, tracked, and used in a host of different services.

This week A Data Citation Roadmap for Scientific Publishers was published in Scientific Data. This roadmap is the outcome of a collaboration between different publishers that worked on identifying all steps you need to take as a publisher to implement data citation. If you want to know more about establishing a data policy, capturing data citations at the point of submission, or tagging data citations in your XML, we recommend you take a look at this article!

In this blog post, we’ll discuss the steps you need to take after you’ve implemented this roadmap. The steps in the roadmap describe how you can track & tag data citation yourself. Here we describe how Crossref can help you make these available to the rest of the community.

The ‘what’

Here’s the recap! From the Crossref perspective, there are two ways to add data citation links into the metadata that you register:

1. Metadata deposits using the references section of the schema

This is where ‘citations’ are normally recorded. Publishers include the data citation into the deposit of bibliographic references for each publication.

Publishers can deposit the full data or software citation as a unstructured reference. For guidance here, we recommend that authors cite the dataset or software based on community best practice (Joint Declaration of Data Citation Principles, FORCE11 citation placement, FORCE11 Software Citation Principles).

<citation key="ref=3">
<unstructured_citation>Morinha F, Dávila JA, Estela B, Cabral JA, Frías Ó, González JL, Travassos P, Carvalho D, Milá B, Blanco G (2017) Data from: Extreme genetic structure in a social bird species despite high dispersal capacity. Dryad Digital Repository. http://dx.doi.org/10.5061/dryad.684v0</unstructured_citation\>
</citation>
</citation_list>

Or they can employ any number of reference tags currently accepted by Crossref.

<citation key="ref2">
<doi>10.5061/dryad.684v0</doi>
<cYear>2017</cYear>
<author>Morinha F, Dávila JA, Estela B, Cabral JA, Frías Ó, González JL, Travassos P, Carvalho D, Milá B, Blanco G</author>
</citation>

We are exploring JATS4R recommendations to expand the current collection and better support these citations - more on this soon. We also encourage additional suggestions from the community.

2. Metadata deposits using the relations section of the schema

This is where other relationships can be recorded. Publishers assert the data link in the relationship section of the metadata deposit. Here, publishers can identify data which are direct outputs of the research results if this is known. This level of specificity is optional, but we’d recommend it as it can support scientific validation and research funding management.

Data and software citations via relation type enables precise tagging of the dataset and its specific relationship to the research results published. To tag the data & software citation in the metadata deposit, we ask for the description of the dataset & software (optional), dataset & software identifier and identifier type (DOI, PMID, PMCID, PURL, ARK, Handle, UUID, ECLI, and URI), and relationship type.

<program xmlns="http://www.crossref.org/relations.xsd">
<related_item>
<description>Data from: Extreme genetic structure in a social bird species despite high dispersal capacity</description>
<inter_work_relation relationship-type="references" identifier-type="doi">10.5061/dryad.684v0</inter_work_relation>
</related_item>
</program>
</doi_relations>

In general, use the relation type references for data and software resources.

Publishers who wish to specify that the data or software resource was generated as part of the research results can use the isSupplementedBy relation type.

The ‘how’

I create my own XML and register it with Crossref

Add links to datasets into your reference lists, including their DOIs if available as shown above and deposit them with Crossref. We’ll do the rest. If you want to add references to existing metadata records, you don’t need to redeposit the full article metadata, you can send us a resource-only deposit that just contains the reference metadata to append that to the existing metadata for the article. You can also use this method if you prefer to deposit references in a separate workflow to registering your content (we know some members prefer to work this way).

I’ve started using Metadata Manager for journal article deposits

Article<->Data relationships in Crossref

You can deposit data citations using either method using our new Metadata Manager tool. When entering journal article metadata, you can use the ‘Related Items’ section to enter the DOI (or other identifier) for the dataset, the type of identifier, a description of the relation type e.g. ‘Data from: Extreme genetic structure in a social bird species despite high dispersal capacity’, and the relation type - ‘references’ or ‘is supplemented by’ depending on the relationship between the data and the article as described above. When you make the deposit, this relationship information will be registered in Crossref along with the rest of the article metadata.

Metadata Manager also has a section where you can enter and match your references, and then deposit these with Crossref. If you choose this method, enter any data citations into the references section before depositing the article metadata with Crossref.

If you want to add this information to deposits you have already made using Metadata Manager, you can search for the journals and articles in the interface, bring up the existing metadata and add in the additional information before redepositing.

I use “simple text query” to search for and deposit references

Make sure you include any citations to data in the references you add into Simple Text Query. When you use simple text query to deposit these references, they will then be added into the article metadata in the Crossref database.

If you use OJS, they’re working on functionality (due for release soon) that will make it easier to deposit reference metadata with Crossref, so you can include citations to data in that.

All of this metadata—registered with Crossref—make it possible to build up pictures of data citations, linking, and relationships. Whether the citations come from the authors in the reference list or they are extracted by the publisher and then deposited, Crossref collects them across publishers. We then make the aggregate set freely available via Crossref’s APIs in multiple interfaces (REST, OAI-PMH, OpenURL) and formats (XML and JSON). DataCite does the same for data repositories and so this provides an easy way for publishers and data repositories to exchange information about data citations. As mentioned previously, this all feeds in Event Data. Data is made openly available to a wide host of parties across the extended research ecosystem including funders, research organisations, technology and service providers, indexers, research data frameworks such as Scholix, etc.

Do you have questions about how to add these links to your Crossref or DataCite metadata? We’ll be running a series of webinars in early 2019 to give you a chance to join us live and ask any questions you have. Eager to get started in the meantime? Let us know and we’ll start to coordinate.

Matchmaker, matchmaker, make me a match

Dominika Tkaczyk — Mon, 12 Nov 2018 00:00:00 +0000

Matching (or resolving) bibliographic references to target records in the collection is a crucial algorithm in the Crossref ecosystem. Automatic reference matching lets us discover citation relations in large document collections, calculate citation counts, H-indexes, impact factors, etc. At Crossref, we currently use a matching approach based on reference string parsing. Some time ago we realized there is a much simpler approach. And now it is finally battle time: which of the two approaches is better?

TL;DR

I evaluated and compared four approaches to reference matching: the legacy approach based on reference parsing, and three variants of the new idea called search-based matching.
A large automatically generated dataset was used for the experiments. It is composed of 7,374 metadata records from the Crossref collection, each of which was formatted automatically into reference strings using 11 citation styles.
The main metrics used for the evaluation are precision and recall. I also use F1 as a standard metric that combines precision and recall into a single number, weighing them equally. All values are calculated for each metadata record separately and averaged over the dataset.
In general, search-based matching is better than the legacy approach in F1 and recall, but worse in precision.
The best variant of search-based matching outperforms the legacy approach in average F1 (84.5% vs. 52.9%), with the average precision worse by only 0.1% (99.2% vs 99.3%), and the average recall better by 88% (79.0% vs. 42.0%).
The best variant of search-based matching also outperforms the legacy approach in average F1 for each one of the 11 styles.
A weak spot of the parsing-based approach is degraded/noisy reference strings, which do not appear to use any of the known citation styles.
A weak spot of search-based approach is short reference strings, and in particular citation styles that do not include the title in the reference string.

Introduction

In reference matching, on the input we have a bibliographic reference. It can have the form of an unstructured string, such as:

(1) Adamo, S. H.; Cain, M. S.; Mitroff, S. R. Psychological Science 2013, 24, 2569–2574.

The input can also have the form of a structured reference, such as (BibTex format):

 @article{adamo2013,
 author = {Stephen H. Adamo and Matthew S. Cain and Stephen R. Mitroff},
 title = {Self-Induced Attentional Blink: A Cause of Errors in Multiple-Target Search},
 journal = {Psychological Science},
 volume = {24},
 number = {12},
 pages = {2569-2574},
 year = {2013}
}

The goal of matching is to find the document, which the input reference points to.

Matching algorithms

Matching references is not a trivial task even for a human, not to mention the machines, which are still a bit less intelligent than us (or so they want us to believe…). A typical meta-approach to reference matching might be to score the similarity between the input reference and the candidate target documents. The document most similar to the input is then returned as the target.

Of course, still a lot can go wrong here. We can have more than one potential target record with the same score (which one do we choose?). We can have only documents with low to medium scores (is the actual target even present in our collection?). We can also have errors in the input string (are the similarity scores robust enough?). Life’s tough!

The main difference between various matching algorithms is in fact how the similarity is calculated. For example, one idea might be to compare the records field by field (how similar is the title/author/journal in the input reference to the title/author/journal of our candidate target record?). This is roughly how the matching works currently at Crossref.

The main problem with this approach is that it requires a structured reference, and in practise, often all we have is a plain reference string. In such a case we need to extract the metadata fields from the reference string (this is called parsing). Parsing introduces errors, since no parser is omniscient. The errors propagate further and affect the scoring… you get the picture.

Luckily, as we have known for some time now, this is not the only approach. Instead of comparing structured objects, we could calculate the similarity between them using their unstructured textual form. This effectively eliminates the need for parsing, since the unstructured form is either already available on the input or can be easily generated from the structured form.

What about the similarity scores? We already know a powerful method for scoring the similarities between texts. Those are (you guessed it!) scoring algorithms used by search engines. Most of them, including Crossref’s, do not need a structured representation of the object, they are perfectly happy with just a plain text query.

So all we need to do is to pass the original reference string (or some concatenation of the reference fields, if only a structured reference is available) to the search engine and let it score the similarity for us. It will also conveniently sort the results so that it is easy to find the top hit.

Evaluation

So far so good. But which strategy is better? Is it better to develop an accurate parser, or just rely on the search engine? I don’t feel like guessing. Let’s try to answer this using (data) science. But first, we need to decompose our question into smaller pieces.

Question 1. How can I measure the quality of a reference matcher?

Generally speaking, this can be done by checking the resulting citation links. Simply put, the better the links, the better the matching approach must have been.

A few standard metrics can be applied here, including accuracy, precision, recall and F1. We decided to calculate precision, recall and F1 separately for each document in the dataset, and then average those numbers over the entire dataset.

When I say “documents”, I really mean “target documents”:

precision for a document X tells us, what percentage of links to X in the system are correct
recall for a document X tells us, what percentage of true links to X are present in the system
F1 is the harmonic mean of precision and recall

F1 is a single-number metric combining precision and recall. In F1 precision and recall are weighted equally. It is also possible to combine precision and recall using different weights, to place more emphasis on one of those metrics.

We decided to look at links from the target document’s perspective, because this is what the academic world cares about (i.e. how accurate the citation counts of academic papers are).

Calculating separate numbers for individual documents and averaging them within a dataset is the best way to have reliable confidence intervals (which makes the whole analysis look much smarter!).

Question 2. Which approaches should be compared?

In total we tested four reference matching approaches.

The first approach, called the legacy approach, is the approach currently used in Crossref ecosystem. It uses a parser and matches the extracted metadata fields against the records in the collection.

The second approach is the search-based matching (SBM) with a simple threshold. It queries the search engine using the reference string and returns the top hit from the results, if its relevance score exceeds the threshold.

The third approach is the search-based matching (SBM) with a normalized threshold. Similarly as in the simplest SBM, in this approach we query the search engine using the reference string. In this case the first hit is returned if its normalized score (the score divided by the reference length) exceeds the threshold.

Finally, the fourth approach is a variation of the search based matching, called search-based matching with validation (SBMV). In this algorithm we use additional validation procedure on top of SBM. First, SBM with a normalized threshold is applied and the search results with the scores exceeding the normalized threshold are selected as candidate target documents. Second, we calculate validation similarity between the input string and each of the candidates. This validation similarity is based on the presence of the candidate record’s metadata fields (year, volume, issue, pages, the last name of the first author, etc.) in the input reference string, as well as the relevance score returned by the search engine. Finally, the most similar candidate is returned as the final target document, if its validation similarity exceeds the validation threshold.

By adding the validation stage to the search-based matching we make sure that the same bibliographic numbers (year, volume, etc.) are present in both the input reference and the returned document. We also don’t simply take the first result, but rather use this validation similarity to choose from results scored similarly by the search engine.

All the thresholds are parameters which have to be set prior to the matching. The thresholds used in these experiments were chosen using a separate dataset, as the values maximizing the F1 of each algorithm.

Question 3. How to create the dataset?

Results

We could try to calculate our metrics for every single document in the system. Since we currently have over 100M of them, this would take a while, and we already felt impatient…

A faster strategy was to use sampling with all the tools statistics was so generous to provide. And this is exactly what we did. We used a random sample of 2500 items from our system, which is big enough to give reliable results and, as we will see later, produces quite narrow confidence intervals.

Apart from the sample, we needed some input reference strings. We generated those automatically by formatting the metadata of the chosen items using various citation styles. (Similarly to what happens when you automatically format the bibliography section for your article. Or at least we hope you don’t produce those reference strings manually…)

For each record in our sample, we generated 11 citation strings, using the following styles:

Well known citation styles from various disciplines:
- american-chemical-society (acs)
- american-institute-of-physics (aip)
- elsevier-without-titles (ewt)
- apa
- chicago-author-date
- modern-language-association (mla)
Known styles + random noise. To simulate not-so-clean data, we randomly added noise (additional spaces, deleted spaces, typos) to the generated strings of the following styles:
- american-institute-of-physics
- apa
Custom degraded “styles”:
- degraded: a simple concatenation of authors’ names, title, container title, year, volume, issue and pages
- one author: a simple concatenation of the first author’s name, title, container title, year, volume, issue and pages
- title scrambled: same as degraded, but with title words randomly shuffled

Some styles include the DOI in the reference string. In such cases we stripped the DOI from the string, to make the matching problem non-trivial.

An ideal matching algorithm will match every generated string to the record it was generated from. In practise, some of the expected matches will be missing, which will lower the recall of the tested matching approach. On the other hand, it is very probable that we will get the precision of 100%. To have the precision lower than 100%, we would have to have some unexpected matches to our sampled documents, which is unlikely. This is obviously not great, because we are missing a very important piece of information.

What can we do to “encourage” such mismatches to our sampled documents? We could generate additional reference strings of documents that are not in our sample, but are similar to the documents in our sample. Hopefully, we will see some incorrect links from those similar strings to our sampled documents.

For each sampled document I added up to 2 similar documents (I used, surprise surprise, our search engine to find the most similar documents). I ended up with 7,374 items in total (2,500 originally sampled and 4,874 similar items). For each item, 11 different reference strings were generated. Each reference string was then matched using the tested approaches and I could finally look at some results.

Results

First, let’s compare the overall results averaged over the entire dataset:

The small vertical black lines at the top of the boxes show the confidence intervals at the confidence level 95%. The table gives the exact values and the same confidence intervals. The best result for each metric is bolded.

	average precision	average recall	average F1
legacy approach	0.9933 (0.9910 - 0.9956)	0.4203 (0.4095 - 0.4312)	0.5289 (0.5164 - 0.5413)
SBM (simple threshold)	0.9890 (0.9863 - 0.9917)	0.7127 (0.7021 - 0.7233)	0.7866 (0.7763 - 0.7968)
SBM (normalized threshold)	0.9872 (0.9844 - 0.9901)	0.7905 (0.7796 - 0.8015)	0.8354 (0.8249 - 0.8458)
SBMV	0.9923 (0.9902 - 0.9945)	0.7902 (0.7802 - 0.8002)	0.8448 (0.8352 - 0.8544)

The confidence intervals given in the table are the ranges, in which it is 95% likely to have the real average precision, recall and F1. For example, we are 95% sure that the real F1 for SBMV in our entire collection is within the range 0.8352 - 0.8544.

As we can see, each metric has a different winner.

The legacy approach is the best in precision. This suggests the legacy approach is quite conservative and outputs a match only if it is very sure about it. This might also result in missing a number of true matches (false negatives).

According to the paired Student’s t-test, the difference between the average precision of the legacy approach and the average precision of the second best SBMV is not statistically significant. This means we cannot rule out that this difference is simply the effect of the randomness in sampling, and not the sign of the true difference.

SBM with a normalized threshold is the best in recall. This suggests that it is fairly tolerant and returns a lot of matches, which might also result in returning more incorrect matches (false positives). Also in this case the difference between the winner and the second best (SBMV) is not statistically significant.

SBMV is the best in F1. This shows that this approach balances precision and recall the best, despite being only the second best in both of those metrics. According to the paired Student’s t-test, the difference between SBMV and the second best approach (SBM with a normalized threshold) is statistically significant.

All variants of the search-based matching outperform the parsing-based approach in terms of F1, with statistically significant differences. This shows that in search based-matching it is possible to keep precision almost as good as in the legacy approach, and still include many more true positives.

Let’s also look at the same results split by the citation style:

For all styles the precision values are very high, and the legacy approach is slightly better than all variations of the search-based approach.

In terms of recall and F1 SBM with a simple threshold is better than the legacy approach in 8 out of 11 styles. The three styles for which the legacy approach outperforms SBM with a simple threshold are styles that do not include the title in the reference strings (acs, aip and ewt). The reason for this is that the simple threshold cannot be well calibrated for shorter and longer reference strings at the same time.

SBM with a normalized threshold and SBMV is better than the legacy approach in recall and F1 for all 11 styles.

The weak spot of the legacy approach is degraded and noisy reference strings, which do not appear to use any of the known citation styles.

The weak spot of the search-based matching is short reference strings, and in particular citation styles that do not include the title in the string.

Limitations

The limitations are related mostly to the method of building the dataset.

All the numbers reported here are estimates, since they were calculated on a sample.
The numbers show strengths and weaknesses of each approach, but they do not reflect the real precision and recall in the system:
- Since we included only 2 similar documents for each document in the sample, precision is most likely lower in the real data.
- We used a number of styles distributed uniformly. Of course in the real system the styles and their distribution might be different, which affects all the calculated numbers.

What does the sample say?

Dominika Tkaczyk — Fri, 09 Nov 2018 00:00:00 +0000

At Crossref Labs, we often come across interesting research questions and try to answer them by analyzing our data. Depending on the nature of the experiment, processing over 100M records might be time-consuming or even impossible. In those dark moments we turn to sampling and statistical tools. But what can we infer from only a sample of the data?

Imagine you are cooking soup. You just put some salt in it and now you are wondering if it is salty enough. What do you do next?

Option #1: Since you carefully measured 1/7 of a teaspoon of salt per 0.13 litres of soup (as always), you already know the soup is fine. Everyone else better stop asking silly questions and eat their soup.
Option #2: You stir everything carefully and taste a tablespoon. If it is not salty enough, you put more salt in the soup and repeat the tasting procedure.
Option #3: You eat a tablespoon of soup and it tastes fine. But wait, there’s more soup in the pot, what if the sip you’ve just tasted was somehow different than the rest? You decide it’s better to eat another spoon of soup (which tastes fine). Still, a lot of soup left, who knows what that tastes like? It might be safer to eat an entire bowl of soup. Hmm, still not sure, you’ve eaten such a small fraction of the soup, who can guarantee the rest tastes the same? You have no choice but to eat another bowl, and then some more… Ooops, now you have eaten the entire pot of soup! At least you can be 100% sure now that the soup was indeed salty enough. The problem is, there is no soup left, and also, you don’t feel so good. But people are getting hungry, so you start cooking a new batch…

If your answer was option #3, read on. Your life is going to get easier!

TL;DR

Sampling and confidence intervals can be used to estimate the mean of a certain feature, or the proportion of items passing a certain test, by calculating it only for a random sample of items, instead of the entire large set of items. Note that estimating =/= guessing.
Confidence intervals are a way of controlling the amount of uncertainty related to randomness in sampling.
The confidence interval has a form (estimated value - something, estimated value + something). Confidence interval at the confidence level 95% is interpreted as follows: we are 95% sure that the real value that we are estimating is within our calculated confidence interval.
The higher the confidence level (i.e. the more certain we want to be about the interval), the wider the interval has to be.
The larger the sample, the narrower the confidence interval.
We are never 100% sure that the value we are estimating is actually within our calculated confidence interval. By setting the confidence level high, we only make sure this is a very likely event.

The problem

Sampling and estimating drew my attention while I was working on the evaluation of the reference matching algorithms. In Crossref’s case, reference matching is the task of finding the target document DOI for the given input reference string, such as:

(1) Adamo, S. H.; Cain, M. S.; Mitroff, S. R. Psychological Science 2013, 24, 2569–2574.

Accurate reference matching is very important for the scientific community. Thanks to automatic reference matching we are able to find citing relations in large document sets, calculate citation counts, H-indexes, impact factors, etc.

For several weeks now I have been investigating a simple reference matching algorithm based on the search engine. In this algorithm, we use the input reference string as the query in the search engine, and we return the first item from the results as the target document. Luckily, at Crossref we already have a good search engine in place, so all the pieces are there.

I was interested in how well this simple algorithm works, i.e. how often the correct target document is found. For example, let’s say we have a reference string in APA citation style generated for a specific record in Crossref system. How certain can I be that it will be correctly matched to the record’s DOI?

I could calculate this directly by generating the APA reference string for every record in the system and trying to match those strings to DOIs. Since we already have over 100M records, this would take a while and I was getting impatient. So instead of eating the whole pot of soup, I decided to stir and taste just a little bit of it, or, academically speaking, use sampling and confidence intervals.

These statistical tools are useful in situations, where we have a large set of items, and we want to know the average of a certain feature of an item in our set, or the proportion of items passing a certain test, but calculating it directly is impossible or difficult. For example, we might want to know the average height of all women living in USA, the average salary of a Java programmer in London, or the proportion of book records in the Crossref collection. The entire set we are interested in is called a population and the value we are interested in is called a population average or a population proportion. Sampling and confidence intervals let us estimate the population average or proportion using only a sample of items, in a reliable and controlled way.

Experiments

In general I wanted to see, how well I can estimate the population proportion of records passing a certain test, using only a sample.

In the following experiments, the population is 1 million metadata records from the Crossref collection. I didn’t use the entire collection as the population, because I wanted to be able to calculate the real proportion and compare it to the estimates.

The test for a single record is: whether the APA reference string generated from said record is correctly matched to the record’s original DOI. In other words: if I generate the APA reference string from my record and use it as the query in Crossref’s search, will the record be the first element in the result list? Note that this proportion can also be interpreted as the probability that the APA reference string will be correctly matched to the target DOI.

Estimating from a sample

I took a random sample of size 100 from my population and calculated the proportion of the records correctly matched - this is called a sample proportion. In my case, the sample proportion is 0.92. This means that in my sample 92 reference strings were successfully matched to the right DOIs. Not too bad.

I could now treat this number as the estimate and assume that 0.92 is close to the population proportion. On the other hand, this is only a sample, and a rather small one, which raises doubts. What if our 92 correct matches happen to be the only correct matches in the entire 1M population? In such a case, our estimate of 0.92 would be very far from the population proportion. This uncertainty related to sampling randomness can be captured by the confidence interval.

Confidence interval

The confidence interval for my 100-point sample, at the confidence level 95%, is 0.8668-0.9732. This is interpreted as follows: we are 95% sure that the real population proportion is within the range 0.8668-0.9732. Note that the sample average (0.92) is exactly in the middle of this range.

100 items is not a big sample. Let’s calculate the confidence interval for a sample 10 times larger. From a sample of size 1000 I got the estimate 0.932, and the confidence interval 0.9164-0.9476. Based on this, we can be 95% sure that the real population proportion is within the range 0.9164-0.9476.

It seems the our interval got smaller when we increased the sample size. Let’s plot the intervals for a variety of sample sizes:

The blue line represents the estimated proportion for samples of different sizes, and the grey vertical lines are confidence intervals. The estimated proportion varies, because for each size a different sample was drawn.

We can see that increasing the sample size decreases the interval. This should make intuitive sense: if we have more data to estimate from, we can expect our estimate to be more reliable (i.e. closer to the population proportion).

What about the confidence level? By setting the confidence level we specify, how certain we want to be about our confidence interval. So far I used 95%. What happens if I calculate the confidence intervals for my original sample of 100 records, but with varying confidence level?

In this case the average is always the same, because only one sample was used.

As we can see, increasing the confidence level widens the interval. In other words, the more certain we want to be about the interval containing the real population average, the wider the interval has to be.

Sampling distribution

So far so good, but where does this magic confidence interval actually come from? It is calculated by the theoretical analysis of the sampling distribution (not to be confused with sample distribution):

Sample distribution is when we collect one sample of size k and calculate a certain feature for every element in the sample. It is a distribution of k values of the feature in one sample.
Sampling distribution is when we independently collect n samples, each of size k, and calculate the sample proportion for each sample. It is the distribution of n sample proportions.

Imagine I collect all samples of size 100 from my population and I calculate the sample proportion for each sample. This is the sampling distribution. Now I randomly choose one number from this sampling distribution. Note that this is equivalent to what I did before: choosing one random sample of size 100 and calculating its sample proportion.

According to Central Limit Theorem, sampling distribution is approximately normal with the mean equal to the population proportion. Here is the visualisation of the sampling distribution:

The black vertical line shows the mean of the sampling distribution. This is also the real population proportion. The grey area covers the middle 95% of the distribution mass (within 2 standard deviations from the mean).

When we choose one sample and calculate the sample proportion, there are two possibilities:

With 95% probability, we were lucky and the sample proportion is within the grey area. In that case, the real population proportion is not further than 2 standard deviations from our estimate.
With 5% probability, we were unlucky and the sample proportion is outside the grey area. In that case, the real population proportion is further than 2 standard deviations from our estimate.

So with the confidence of 95% we can say that the real population proportion is within 2 standard deviations from our sample proportion. We can see now that these 2 standard deviations of the sampling distribution define our confidence interval at the confidence level of 95%.

Smaller confidence level would make the grey area narrower, and the confidence interval would shrink as well. Larger confidence level makes the grey area, and the confidence interval, larger.

To look more closely at the sampling distribution, I generated sampling distributions for all combinations of “n samples of size k”, where n and k are the elements of the set {25, 50, 100, 200, 400, 800, 1600, 3200}. This is only an approximation, since the real sampling distributions would contain many more samples.

Here is the heatmap showing the mean of each sampling distribution (this should be approximately the same as the real population proportion):

We can see that there is some variability in the top left part of the heatmap, which corresponds to small sample sizes and small numbers of samples. The bottom right part of the heatmap shows much less variability. As we increase the sample size and number of samples, the mean of the sampling distribution approaches numbers around 0.933.

Here is the heatmap showing the standard deviation for each sampling distribution:

We can clearly see how the standard deviation decreases when we increase the sample size. This is consistent with the previous observation, that the confidence interval decreases when the sample size is increased.

Let’s also see the histograms of all the sampling distributions:

Here we can see the following patterns:

All histograms indeed seem to be centered around approximately the same number.
The more samples we include, the more normal the sampling distribution appears. This happens because with more samples the real sampling distribution is better approximated.
The larger the sample size, the narrower the sampling distribution (i.e. smaller standard deviation).

The estimation vs. the real value

Let’s go back to my original question. What is the proportion of reference strings in APA style, that are successfully matched to the original DOIs of the records they were generated from? So far we observed the following:

A small sample of 100 gave the estimate 0.92 (confidence interval 0.8668-0.9732)
A larger samples of 1000 gave the estimate 0.932 (confidence interval 0.9164-0.9476)
The means of sampling distributions seem to slowly approach 0.933

So what is the real population proportion in my case? It is 0.933005. As we can see, the estimations were fairly close, and the intervals indeed contain the real value.

Now I can also calculate the confidence interval for each sample in my sampling distributions, and then the fraction of the intervals that contain the real population proportion (I expect these numbers to be close to the confidence level 95%). Here is the heatmap:

We can see that for larger sample sizes indeed the fractions are high. The fraction is not always above 95%, as we would expect, especially for smaller sample sizes. One of the reasons is that when we calculate the confidence interval, we approximate the standard deviation of the population with the standard deviation of the sample. This is not always a reliable estimate, especially for small samples. This suggests that sample sizes of at least 1000-2000 should be used.

Be careful

Some important things to remember:

Aggregate functions. As mentioned before, apart from estimating the proportion, a similar procedure can be applied for estimating the average of a certain numeric feature.
(Lack of) certainty. Remember that the confidence level < 1. This means that we are never sure that our confidence interval contains the true population proportion. If for any reason you need to be 100% sure, just process the entire dataset.
Randomness, a.k.a. “stirring before tasting”. The sample has to be chosen randomly. Beware of assuming that the dataset is shuffled and taking the first 1000 rows!
Sample size. We know already that the larger the sample, the better. As a rule of thumb, using sample sizes < 30 makes the estimates, including the interval, rather unreliable.
Skewness. In general, the more skewed the original feature distribution, the larger sample we need. In case of the proportion, the sample should contain at least 5 data points of each value of the feature (passes/doesn’t pass the test).
Generalization. The sample average/proportion can be used as an estimate for the population average/proportion, but only the population it was drawn from. This means that if we applied any filters before sampling (which is equivalent to sampling from a subset passing the filter), we can reason only about the filtered subset of the data.
Reproducibility. This is more of an engineering concern. In short, all the analyses we do should be reproducible. In the context of sampling it means, at the very least, that we should record the samples we use.

Why Data Citation matters to publishers and data repositories

Helena Cousijn — Thu, 08 Nov 2018 00:00:00 +0000

A couple of weeks ago we shared with you that data citation is here, and that you can start doing data citation today. But why would you want to? There are always so many priorities, why should this be at the top of the list?

I’m sure you heard this before—data sharing and data citation are important for scientific progress. The three key reasons for this are:

1) Transparency and reproducibility

Most scientific results that are shared today are just a summary of what researchers did and found. The underlying data are not available, making it difficult to verify and replicate results. If data would always be made available with publications, transparency of research would be greatly improved.

2) Reuse

The availability of raw data allows other researchers to reuse the data. Not just for replication purposes, but to answer new research questions.

3) Credit

When researchers cite the data they used, this forms the basis for a data credit system. Right now researchers are not really incentivized to share their data, because nobody is looking at data metrics and measuring their impact. Data citation is a first step towards changing that.

The benefits described above are all quite long-term, so why, as a publisher or data repository, should you put your resources towards implementing data citation workflows now? During our pre-conference workshop at FORCE2018 we asked repositories and publishers this question. Below you’ll find some of the answers.

Data repositories

For data repositories, data citation leads to increased visibility of both the repository and the datasets. The workshop revealed that many repositories do a lot of work to establish links between articles and datasets, thereby significantly contributing to transparency in research. Some of the repositories explained that they hire curators that text mine articles to find associations and manually curate datasets to ensure information about links is part of the metadata. This is reflected in Event Data, where 99% of links between articles and datasets comes from data repository metadata. This downstream enrichment of metadata is useful, but it would be more effective if all stakeholders strive to establish these links at a much earlier stage in the research communication process.

ICPSR, the Inter-university Consortium for Political and Social Research, shared:

ICPSR views data citation as vital. As a large social science data archive, ICPSR curates, preserves, and distributes data for the research community to re-use over time. Data citation makes data visible to the research community. Without it, data cannot be accessed for re-use or reproduced for transparency. Its use cannot be tracked and counted to reveal its impact and potential for new uses by investigators in new fields or in combination with new types of data. Data creators cannot receive adequate credit for their intellectual output. And the original investment by funders and scientists to create those data stops producing dividends. Therefore, data citation plays an essential role in the data sharing lifecycle.

Proper data citation, with a unique identifier, makes it much easier to measure impact. When data use is not cited or cited obliquely, it is rendered virtually invisible. Hence, much data use is still not easily detected. The ICPSR Bibliography of Data-related Literature represents ICPSR’s efforts to identify publications that analyze data distributed at ICPSR and link them directly to the data in the ICPSR catalog. As of 2018, ICPSR has a searchable database that contains nearly 80,000 citations of published and unpublished works resulting from analyses of data held in the archive. ICPSR also makes the case for data citation in its brief new video, “ICPSR 101: Why Should I Cite Data?”

GBIF, the Global Biodiversity Information Facility, explained:

The work required to collect, clean, compile and publish biodiversity datasets is significant and deserves recognition. Researchers publish studies based on data made available through GBIF.org at a rate of about 2 papers every single day. It is crucial for GBIF to link these scientific uses to the underlying data as one measure of demonstrating the value and impact of sharing free and open biodiversity data. At the moment, however, only about 10 percent of authors cite or acknowledge the datasets used in research papers properly. As a result, data publishers efforts often risk going unnoticed, and the true impact of sharing data remains invisible. GBIF will continue to work with publishers and researchers to provide guidance and input for how to best cite the use of GBIF-mediated data in scientific journals to ensure proper attribution and reproducible research and to demonstrate the true value of free and open access to biodiversity data.

Publishers

By ensuring data is cited in a consistent way, publishers help provide transparency and context for the content they publish. Depositing that information as part of the Crossref metadata helps that work go further by uncovering how data is being used across multiple publications and publishers This means patterns can be explored and researchers can gain more comprehensive recognition and credit for the work they have done.

Melissa Harrison, Head of Production Operations at eLife says:

eLife is committed to ensuring researchers get credit for all their outputs, and data is a major component of this. We’re working with Crossref and JATS4R to enable publishers to tag their JATS data content consistently and thus create an easy crosswalk to their Crossref deposits. The JATS4R guidance on Data Availability Statements, linked to and incorporating data citations, will be updated soon, please watch that space!

It will be really interesting to see how much re-use of previously published data is happening, look for patterns in re-use, and see links and hopefully building up of data by different research groups. Ultimately, this will incentivize researchers and publishers to ensure it is correctly accredited at source and in publications, improving the cycle further.’

Anita de Waard, VP of Research Collaborations at Elsevier, says:

One of the key recommendations of the Force11 Manifesto was to “3.3 Add data, software, and workflows into the publication as first-class research objects”, which will allow greater reproducibility and rigor to experimental research, and allow the reuse of all digital artefacts in the scholarly lifecycle. By following the data citation principles, we achieve two things: the author presents a richer representation of their work, and the data producer receives credit for the hard work of curating and publishing citable datasets.

Mendeley Data and Elsevier are active contributors to the Scholix framework that as a collaborative and open standard, allows the open mining of relationships between articles and datasets. We are also active participants in the new Enabling FAIR Data Project, and next to supporting the TOP Guidelines in all domains, require all authors in the earth and space sciences to deposit their data before publication.

Next week at Crossref LIVE18, Patricia Cruse from DataCite will talk about Data Citations and why they matter. If you’re in Toronto next week, do not hesitate to ask her or anyone from Crossref anything you want to know about data citation!

Ten more days 'til Toronto

Ginny Hendricks — Fri, 02 Nov 2018 00:00:00 +0000

Our LIVE Annual Meeting is back in North America for the first time since 2015, and with just 10 days to go, there’s a lot going on in preparation. As you’d expect with a How good is your metadata? theme—the two-days will be entirely devoted to the subject of metadata—because it touches everything we do, and everything that publishers, hosting platforms, funders, researchers, and librarians do. Oh, and it’s actually super awesome too—and occasionally fun.

Metadata is what is used to describe the story of research: its origin, its contributors, its attention, and its relationships with other objects. The more machines start to do what humans cannot—parse millions of files through multiple views—the more we see what connections are missing, and the more we start to understand the opportunities that better metadata could offer.

We love metadata so much that we’re producing an 8-foot-high depiction of the ‘perfect’ record, in both XML and JSON, for people to gape at and annotate in person. Sneak preview:

The perfect metadata record is eight feet tall. SchemaSchemer

Both days feature plenary-style talks, insights from ourselves and guests who will regale us with tales of metadata woes and wonders.

Lisa will be there at the end of Day 1 to update everyone on some recent and potential governance changes, and—the reason we started these gatherings—to reveal the results of our 2018 board election, the second contested election we’ve held, and already with twice the voters from 2017.

Our amazing guest speakers are too brilliant and too experienced to highlight in just one blog. But check out the LIVE18 schedule to see what they’ll be talking about:

Patricia Cruse, DataCite
Ravit David, University of Toronto
Clare Dean, Metadata 2020
Paul Dlug, American Physical Society
Kristen Fisher Ratan, CoKo Foundation

Stefanie Haustein, University of Ottawa
Bianca Kramer, Utrecht University
Graham Nott, Freelance developer (eLife/JATS)
Jodi Schneider, University of Urbana-Champaign
Shelley Stall, American Geophysical Union

We’ll be taking over the entire second floor of the Toronto Reference Library, whose three rooms will house a bunch of conversational sessions as well as some more formal talks:

Rally is the main room where we’ll have the plenary-style talks, a corner for Unscheduled Maintenance offering live support for your questions about billing or tech for Ryan, Shayn, Isaac, Jason, Chuck, & Mike. Running down the whole left side of this room is also the You-are-Crossref wall where the community will showcase their work with metadata through posters - feel free to bring one along and find Patricia to get the sticky tack.
The LIVE Lounge is where you can eat, drink, rest, and chat and where you’ll likely find Rosa as she laises between the caterers, the venue, AV, and all of us. The Lounge is also where we’ll gather for much-needed post-election refreshments at the end of Tuesday.
The Bigger Ambitions Room is where a lot of the Unplugged sessions will take place. This room will feature three separate stations:
- R&D & Product where you can chat with Geoffrey, Esha, Jennifer L, Patrick, and Christine about your big ideas for us, and what we’re working on already.
- Metadata discussions and annotations of the perfect record (previewed above) with Patricia, together with space to ideate around metadata principles.
- Uses and users of metadata where Jennifer K will help us understand just how far Crossref metadata can reach, and who and what people are doing with it.

We cannot wait to show you what else we have planned :-)

For those of you not able to attend, recordings of the presentations will be made available on the event page directly soon after.

Otherwise - see you there!

Crossref LIVE Brazil evoked vibrant Q&A session

Susan Collins — Wed, 31 Oct 2018 00:00:00 +0000

There has been a steady increase in the growth of our membership in Latin America—and in Brazil in particular—over the past few years. We currently have more than 800 Brazil-based members; some as individual members, but most are sponsored by another organisation. As part of our LIVE Local program Chuck Koscher and I traveled to meet some of these members in Goiânia and Fortaleza, where we co-hosted events with Associação Brasileira de Editores Científicos do Brasil (ABEC Brasil)—one of our largest Sponsors.

These events always provide a great opportunity for us to update our members on new and upcoming Crossref developments. They are also an important way for us to discover more about the varied needs of our members’ communities and learn how we can work together better.

The LIVE Brazil events were attended by more than two hundred members and were held at the Universidade Federal de Goiás and the Universidade de Fortaleza respectively. Chuck and I enthusiastically demonstrated two new tools from Crossref— Participation Reports and Metadata Manager, we discussed our newest record types—preprints and peer review reports, and continually highlighted the importance (and the uses) of quality metadata.

We were joined by some fantastic guest speakers; Milton Shintaku from ABEC explained how to register content using the Crossref/OJS deposit plugin and Crossref ambassador, Edilson Damasio, spoke about Similarity Check and gave a demonstration of how to use the iThenticate interface when checking papers for originality.

The vibrant Q&A sessions reflected the varying needs of the audience. We talked generally about the different Crossref services and went more in-depth with discussions around submitting relationship metadata for peer review and preprints. Crossmark and its implementation was also a hot topic, as was how to benefit from Similarity Check—and in particular how to address cases of duplication in submitted manuscripts, and the setting up of plagiarism policies for each journal. There was also a lot of discussion around OJS integrations, and we were able to share that PKP/OJS is currently in the process of enhancing the Crossref/OJS integration, including the ability for publishers to deposit references.

We were also pleased to see so much interest in supplementing Crossref metadata with references, Similarity Check URLs, license information, etc. To address this we’re running a webinar in Brazilian Portuguese entitled: “Registering content and adding to your Crossref metadata in Portuguese” on 26th November. You can sign up here if you’d like to attend.

I’d like to thank Universidade Federal de Goiás and the Universidade de Fortaleza for hosting the events, providing the venues and the translation team, and of course, thanks to everyone who came!

A special mention of ABEC for their help in organizing and promoting the events. As a Sponsor, they relieve our team of an intense amount of technical support, billing, and other administrative burdens, saving us time and expense, while offering a localized service to Brazilian publishers.

Crossref staff with co-hosts ABEC and representatives from UFG who helped with the event - thank you!

It’s not about the money, money, money.

Amy Bosworth — Thu, 18 Oct 2018 00:00:00 +0000

But actually, sometimes it is about the money. As a not-for-profit membership organisation that is obsessed with persistence, we have a duty to remain sustainable and manage our finances in a responsible way. Our annual audit is incredibly thorough, and our outside auditors and Board-based Audit committee consistently report that we’re in good shape.

Our Membership & Fees committee regularly reviews both membership fees and Content Registration fees for a growing range of research outputs. Together with our staff, the Board regularly reviews financial projections that inform our budgeting process and approve our budget each year.

Financial sustainability means the persistence of our infrastructure and services

We run a tight ship here at Crossref. We have to. So it’s not ideal when we have to chase members and users for late payments, but it’s an important part of keeping the organisation afloat, and keeping our dedicated service to scholarly communications running. And that’s my job at Crossref.

Working here for over six years now, I’ve seen a lot of development in our finance department. We strive as a team to always improve our communication with members and users to deliver the best ‘customer’ experience. To do this, we are always tweaking our processes to improve efficiency and accuracy, and welcome all feedback.

How the invoice schedule works

Our annual membership invoices are sent out each January, and our Content Registration invoices are generated four times a year, each quarter. All invoices are emailed to the billing contact for your organisation (please be sure to update us with any contact changes!) and have a due date of net 45 days. Our invoices now have a “pay now” link in the body of the email. This offers a faster and more convenient way for you to pay, simply by clicking on the link to our payment portal. You can also view invoices as PDFs in the payment portal. An important part of our accounting process is the automated invoice reminder schedule. There are three billing reminders we send by email:

The day immediately after the invoice due date;
21 days past the invoice due date; and
45 days past the invoice due date.

We don’t want to see you go!

We understand there are many factors that can make prompt payment a challenge for some people: international transfer delays or fees; funding for your publishing operations may end; change of contacts; problems receiving our emails.

When an account is 90 days past due, a further email notifies you that your service is at risk of suspension. If an account is then suspended for non-payment it becomes at risk of being ‘terminated’. Once an account has been terminated, you will need to contact our membership specialist to rejoin Crossref. Please note that we send numerous notifications/reminders before suspension or termination takes place (we don’t want to see you go!). We can always be reached at billing@crossref.org for any invoice inquiries you may have.

Tips that work for other users

There are some things you can do to speed-up or simplify payments:

Pay with a credit card, using our online payment portal. This is fast, convenient, and lower in fees
Always reference an invoice number on the payment to ensure that it’s applied to your account efficiently
Be sure to make billing@crossref.org a ‘safe’ email address, so that you receive our invoices and reminders
Always keep us up-to-date with any contact changes at your organisation, to ensure that we have accurate information for invoicing and other communication
We recommend giving us a generic email address for your accounts payable team, such as accounts@publisher.com so that if somebody leaves that job, invoices can still get through.

Thanks for working with us! Please let me know in the comments below if you have any feedback or additional tips for your fellow Crossref community members.

Good, better, best. Never let it rest.

Jennifer Kemp — Tue, 16 Oct 2018 00:00:00 +0000

Best practices seem to be having a moment. In the ten years since the Books Advisory Group first created a best practice guide for books, the community beyond Crossref has developed or updated at least 17 best practice resources, as collected here by the Metadata 2020 initiative. (Full disclosure: I co-chair its Best Practices group.)

Books have been one of the fastest growing resource/record types at Crossref for some time, and best practices are just one of the Book Advisory Group’s efforts. Over the past ten years, the members of the books group have updated and added to the guide, and it’s now time for it to get some visibility, so we have added it to our website for easy reference.

These best practices are not documented for the sake of it. They have real value and can help guide internal conversations to evaluate current practices, for example. They can also play a role in making or changing policies, training staff and providing instructions to authors on citation formatting.

Here are a few recent changes I’d like to highlight:

A new section has been added that addresses books hosted on multiple platforms
The section on versions, (including books in multiple formats) has been expanded and clarified
A section on the use of DOIs in citations has been added

It is neither final nor comprehensive, and never will be. Best practices by their very nature must evolve over time—and those with such a broad scope as books will inevitably lack some detail—but that’s all the more reason for the community to stay engaged. Looking ahead to future work from the group, chapter-level metadata is likely to get more attention.

Over the past few years the Books Advisory Group, chaired with aplomb by Emily Ayubi of the American Psychological Association (APA), has spent a lot of time on Crossref initiatives, like Multiple Resolution and DOI display changes but also on broader industry topics like ORCID iDs for book authors, and the Books Citation Index.

As Emily’s term as chair comes to an end this year, we welcome Charles Watkinson of the University of Michigan as chair starting in 2019. The group meets next on 12 December when we will hear from Coko about Editoria and have a discussion about developing our new Metadata Manager Content Registration tool for books, and more.

If you want to share your thoughts on best practices or if you have other topics you’d like us to consider, please get in touch.

Metadata Manager: Members, represent!

Jennifer Lin — Mon, 15 Oct 2018 00:00:00 +0000

Over 100 Million unique scholarly works are distributed into systems across the research enterprise 24/7 via our APIs at a rate of around 633 Million queries a month. Crossref is broadcasting descriptions of these works (metadata) to all corners of the digital universe.

Whether you’re a publisher, institution, governmental agency, data repository, standards body, etc.: when you register and update your metadata with Crossref, you’re relaying it to the entire research enterprise. So make sure your publications are fully and accurately represented.

Metadata Manager is here to help

This year, we’ve released a new tool aimed to make this easier and give you, members, full control over your metadata. Presenting: Metadata Manager. It helps to:

Simplify and streamline the Content Registration service, with a user-friendly interface
Give you greater flexibility and control of metadata deposits
Support users who are less familiar with XML
Boost metadata quality, encourage cleaner and more complete metadata records

Metadata Manager is available to all our members and the service providers they work with, providing assistance with a wide range of metadata-related tasks:

Regular Content Registration conducted by journal staff, editors and service providers
Registering corrections, retractions, or other editorial expressions of concern
Matching references to their DOIs and registering them with the publication
Adding metadata to existing records such as license and funding information, abstracts, or data citations
Late-arriving editorial updates/corrections after initial publication
Unexpected corrections to production hiccups
Emergency editorial changes that affect publication record
Accelerated registration for special pieces published outside of regular workflow
Securely and efficiently transfer titles to another publisher as the authorized owner

Issues arise all the time in the dynamic and challenging work of scholarly communications. Metadata Manager provides a fast and easy way to meet these head-on when broadcasting new content or updating existing content. Submissions through this tool are processed immediately upon submission (i.e., no queues!).

This new tool empowers our members to “represent” in the exhilarating thrum of data reaching our API users. At this moment in time, it only supports journals, but our development team is currently working hard to include the remaining record types.

Features

Here’s a smattering of highlights from the Metadata Manager feature list:

All metadata: easily adds any and all metadata, allowing publishers to add richness and depth to their records.
Prevents rejected submissions: it ensures you have satisfied all the basic Content Registration requirements and points out any input errors.
Expedited deposit: the Content Registration system processes each submission immediately, bypassing the deposit queue.
Historic log: easy to read archive of all previous submissions.
Effortless review: provides a clean, condensed view of metadata (invariably complicated and lengthy) to support human review of the content before submission.
Aids members to follow best practices: checks for completeness and reminds users of the full breadth of metadata available for the article, volume/issue, and the journal itself.
Full control over title transfers: no need to make these requests through our support channels. Complete the transfer at your convenience, directly through the system.

For those of you that have looked at your own metadata contribution with the use of our new Participation Reports, you’ll find using Metadata Manager a quick and useful way to help you level-up your records.

Members, represent!

We invite you to register and update your publications with Metadata Manager, relay the metadata fully and accurately to the entire research enterprise. Check out the comprehensive help documentation to find out how to set up your workspace and get started right away with your usual Content Registration login details.

As mentioned, we are continuing development, adding support for all remaining record types as well as enhancing existing features. The webDeposit form will remain available throughout this time. For journal publishers, give us a whirl and let us know if you see something missing or there’s a function that would improve your Content Registration experience!

Using the Crossref REST API. Part 12 (with Europe PMC)

Christine Cormack Wood — Wed, 10 Oct 2018 00:00:00 +0000

As part of our blog series highlighting some of the tools and services that use our API, we asked Michael Parkin—Data Scientist at the European Bioinformatics Institute—a few questions about how Europe PMC uses our metadata where preprints are concerned.

Tell us a bit about Europe PMC

Europe PMC is a knowledgebase for life science research literature and a platform for innovation based on the content, such as text mining. It contains 34.6 million abstracts and 5 million full-text articles. At Europe PMC we support the research community by developing tools for knowledge discovery, linking publications with underlying research data, and building infrastructure to support text and data mining. Our goal is to create a supportive environment around open access content and data, to maximise its reuse.

What problem is your service trying to solve?

Recent years have seen a dramatic increase in the popularity of preprints within life sciences literature. Preprints have been supported by Crossref since November 2016. In response to the rise in popularity, we have started indexing preprints alongside traditional journal publishing within Europe PMC. We expect this will:

provide another means to access and discover this emergent form of scholarly content
help explore more transparently the role of preprints in the publishing ecosystem
support their inclusion in processes such as grant reporting and credit attribution systems

How do you use Crossref metadata?

Europe PMC operates an open citation network that uses reference lists from our full-text content, supplemented with metadata supplied by the Crossref OAI-PMH API. The number of citations we retrieve from Crossref increased significantly in 2017 thanks to the efforts of the Initiative for Open Citations (I4OC) in improving awareness about sharing citation data.

Our work to ingest preprints into Europe PMC, however, represents our first use of the Crossref REST API. We make a series of queries for each preprint provider, making use of the “posted-content”, “prefix” and (optionally) “has-abstract” filters. We intend to migrate to using the REST API for the majority of retrievals of Crossref content in due course.

What metadata values do you make use of?

Currently we make use of the following fields:

posted as a publication date
abstract
DOI
author for author given names and surnames
title as the preprint title
is-preprint-of to establish preprint –> article links

How often do you extract/query metadata?

We query the REST API daily making use of the from-index-date filter and cursor pagination to insert new or modify existing records. This means that preprints will be available in Europe PMC within 24 hours of the metadata being sent to Crossref. We store the full REST response in MongoDB, a document-based database. Here are some examples of Crossref API queries used to preprint provider PeerJ Preprints:

calling `https://api.crossref.org/works?filter=type:posted-content,has-abstract:true,from-index-date:2018-07-29,prefix:10.7287&sort=updated&rows=1000&cursor=*`
calling `https://api.crossref.org/works?filter=type:posted-content,has-abstract:true,from-index-date:2018-07-29,prefix:10.7287&sort=updated&rows=1000&cursor=AoN4ldf88uQCe6e1g%2FPkAj8SaHR0cDovL2R4LmRvaS5vcmcvMTAuNzI4Ny9wZWVyai5wcmVwcmludHMuMjcwNjJ2MQ%3D%3D`
Done importing PeerJ Preprints
modified: 2
inserted: 10

What do you do with the metadata?

From the database we parse out the relevant fields and pass them to our main relational database prior to indexing. This avails the preprint abstracts to all of the value-added services we offer for peer-reviewed abstracts, such as citations, grants, ORCID claiming, text mining, etc. We assign a unique persistent identifier comprising “PPR” followed by a number (1) to each preprint record.

This is displayed on the Europe PMC site as an abstract record, analogous to PubMed records, but with an obvious banner (2) indicating to readers the preprint designation; a tooltip provides further explanation of what a preprint is in comparison to a peer-reviewed article.

Once available on the Europe PMC platform, we then apply downstream processes including:

providing an Unpaywall link directly to the full-text (3);
adding a hyperlink to the final published version (if there is one that we can detect) (4);
incorporating the preprint into our citation network (5);
adding useful links to e.g. alternative metrics, scientific comments and peer reviews, underlying research data in life science databases (6);
providing text mined annotations via SciLite (7);
including funding information (8);
displaying ORCID claims in the author list (9).

What are the future plans for Europe PMC and preprints?

The inclusion of preprints within Europe PMC is of immediate benefit to researchers who want to explore the very latest research. Moreover we see this as an opportunity for both ourselves and the community to explore how preprints fit into the wider publishing ecosystem; for example to answer questions such as: How often will they be cited? How will they be linked to grant funding and other credit systems? How will they be reused?

What else would you like our API to do?

The REST API and rich metadata model provided by Crossref around preprints are both excellent, but the population of the metadata fields by preprint providers can be limited and/or heterogeneous. The key challenge we see is in encouraging providers to populate the Crossref metadata fields more fully and in a uniform manner.

Thanks to Michael.

If you’d like to share how you use our Metadata APIs please contact the Community team.

A wrap up of the Crossref blog series for SciELO

Christine Cormack Wood — Fri, 05 Oct 2018 00:00:00 +0000

Crossref member SciELO (Scientific Electronic Library Online), based in Brazil, celebrated two decades of operation last week with a three-day event The SciELO 20 Years Conference.

The celebration constituted an important landmark in SciELO’s evolution, and an exceptional moment for them to promote the advancement of an inclusive, global approach to scholarly communication and to the open access movement.

As part of the anniversary activities SciELO asked us to write a series of five blogs that would help the organisations of Brazil to better understand the following:

Why all articles should have a DOI
The critical role of the DOI
The basics of record types, translations, preprints, Crossmark, and more
The basics of Crossref sponsorship, and
How to make the most of your Crossref membership

Below you’ll find an abstract of each of these blog posts as well as a link to the published posts in Brazilian Portuguese, Spanish and English.

Why all articles should have a DOI
In today’s world, an author’s work needs a Digital Object Identifier (DOI) for it to become discoverable, citable, and linkable. This unique alphanumeric string identifies the content of a research work, and remains associated with it irrespective of changes to its web location. Discover the origins of the DOI, how Crossref was founded, and why they continue to exist and persist.

Read the full blog in Brazilian Portuguese, Spanish, or English

The critical role of the DOI
Find out why URL links to research articles are fragile, and how DOIs are essential in building stable, persistent links between research objects. This is achieved through the metadata that members deposit with Crossref, as part of their obligations. Learn how we can all contribute to creating a global, robust research record.

Read the full blog in Spanish or English

The basics of record types: Preprints, Crossmark, translations, and more
What’s the difference between preprints and ahead of print? When should you use each; and, what are the DOI requirements? This article answers those questions and provides a basic overview of how to connect the metadata records of related record types, like translations.

Read the full blog in Brazilian Portuguese, Spanish, or English

The basics of Crossref sponsorship
There are many organisations that want to register content and benefit from the services Crossref provides, but may not be able to do so alone. These organisations use sponsors. Sponsors are organisations who publish on behalf of groups of smaller organisations. Nearly 650 of our 800 Brazilian members are represented by such a sponsor.

Read the full blog in Brazilian Portuguese, Spanish, or English

How to make the most of your Crossref membership
Since Crossref was founded in 2000, its member organisations have registered metadata and persistent identifiers (DOIs) for over 100 million content items. This information is used extensively by the research community—individuals and organisations—who need to find, cite, link and assess research outputs. As a SciELO member, the metadata you provide to Crossref when you register content is key to the discoverability of your journal content.

Read the full blog in Brazilian Portuguese, Spanish, or English

Data citation: let’s do this

Rachael Lammey — Thu, 04 Oct 2018 00:00:00 +0000

Data citation is seen as one of the most important ways to establish data as a first-class scientific output. At Crossref and DataCite we are seeing growth in journal articles and other record types citing data, and datasets making the link the other way. Our organisations are committed to working together to help realize the data citation community’s ambition, so we’re embarking on a dedicated effort to get things moving.

Efforts regarding data citation are not a new thing. One of the first large-scale initiatives to establish data citation as a standard academic practice was the FORCE11 Joint Declaration of Data Citation Principles (JDDCP) in 2014. This declaration was endorsed by over 100 organisations in the scholarly community as well as many individuals.

Following this agreement on how data citation should be done, many projects followed. Within FORCE11, the Data Citation Implementation Pilot brought together publishers and repositories to put data citation into practice and work on the implementation of the JDDCP. Within the context of the Research Data Alliance, a data-literature linking group started under the name of Scholix to establish a framework for exchanging information about the relationships between articles and datasets. The infrastructure building blocks now feed into projects such as Make Data Count and Enabling FAIR Data.

Projects aside, if datasets are cited consistently and in a standard way, it will make it much easier for the research community to see links between different research outputs and work with these outputs. It also makes it much easier to count these citations, so that researchers can get credit for their data and the sharing of that data.

The underlying work has been done to create an infrastructure that will effectively support and disseminate information on data citation. Data citation is here today!

Different organisations know how to handle data citations, and are starting to count these and make that information available in turn. This means that the only thing that’s needed is for people to actually cite data, and this information be captured and passed on. Some Crossref and DataCite members have already made great progress on this already (see Melissa Harrison’s blog on what eLife is doing).

The goals of all the data citation projects can only be realized if you start doing data citation, and we know you’ll have questions about it…

In the coming months, we’ll be posting several blogs and organizing sessions to tell you how you can start doing data citation - if you’re attending FORCE2018 you can catch our joint workshop there. So stay tuned and please get in touch if you can’t wait, we’d love to help you get started!

100,000,000 records - thank you!

Ed Pentz — Wed, 26 Sep 2018 00:00:00 +0000

100,000,000. Yes, it’s a really big number—and you helped make it happen. We’d like to say thank you to all our members, without your commitment and contribution we would not be celebrating this significant milestone. It really is no small feat.

To help put this number into context; the National Museum of China has just over 1 million artifacts, the British Library has around 25 million books, Napster has 40 million tracks, and Wikidata currently contains 50 million+ items.

Digging into the 100 Million

Within these 100 Million registered content records there are many different record types.

And within these record types, more than 69 million records have full-text links, 31 million+ have license information and 3 million+ contain some kind of funding information. An overview of these and other Crossref vital statistics is available on our dashboard.

100 Million—what does your contribution look like?

Our recently-launched participation reports allow anyone to see the metadata Crossref has. It’s a valuable education tool for publishers, institutions and other service providers looking to understand the availability of the metadata they have registered with us.

Through an itemized dashboard Participation Reports allows you to monitor the metadata you are registering, even if this work is done by a third party or another department. You can see for yourself where your gaps are, and what you could improve upon. Next to each metadata element, there’s a short definition, letting you know more about it, and—crucially—what practical steps you can take to improve the score.

The dashboard provides the percentage counts across ten key metadata elements: References, ORCID iDs, Funder Registry IDs, Funding award numbers, Crossmark metadata, License URLs, Text-mining links, Similarity Check URLs, and Abstracts.

And not only can you see your own metadata—the dashboard enables you to view the registered metadata of all our 11,076 members.

How are these 100 Million content records being used?

Every service we provide is based on our metadata, and our APIs expose all of that metadata. Over the past year or so we have been collecting use cases from members that actively utilize the Metadata APIs and have turned these into a Metadata APIs blog series so that we can share these stories of how our metadata is used with the wider community.

A big number. Even bigger ambitions.

Gaps or errors in metadata are passed on to thousands of other services, which causes problems downstream and means we all suffer. So it makes sense for the metadata you deposit to be as accurate and complete as possible. The more elements there are to the metadata, the higher the chance of others finding and using the content. We aim to continually find effective ways to communicate this wider story around the importance of open infrastructure and metadata.

Over the years we’ve made great progress in connecting information about researchers, their affiliations, grants, and research outputs. Imagine how much more powerful this information would be if supplemented by more comprehensive, accurate, and up-to-date metadata.

Sources - all data as of Sept 26, 2018
National Museum of China has 1,050,000 artifacts
The British Library has around 25 million books, more than any other library
Wikidata currently contains 50,290,632 items
NAPSTER currently has 40 million tracks (Napster is known as Rhapsody in the US)

Join us in Toronto this November for LIVE18

Christine Cormack Wood — Tue, 25 Sep 2018 00:00:00 +0000

LIVE18, your Crossref annual meeting, is fast approaching! We’re looking forward to welcoming everyone in Toronto, November 13-14.

This year’s theme “How good is your metadata?” centers around the definition and benefits of metadata completeness, and each half day will cover some element of the theme:

Day one, AM Defining good metadata
Day one, PM Improving metadata quality and completeness
Day two, AM What does good metadata enable?
Day two, PM Who is using our metadata and what are they doing with it?

Both days will be packed with a mixture of plenary and interactive sessions. Speakers include:

Patricia Cruse, DataCite
Kristen Fisher Ratan, CoKo Foundation
Stefanie Haustein, University of Ottawa
Bianca Kramer, Utrecht University
Shelley Stall, American Geophysical Union
Ravit David, University of Toronto Libraries
Graham Nott, Freelance developer of an eLife JATS conversion tool
Paul Dlug, American Physical Society

A ‘meet and mingle’ drinks reception will be held directly after the election results on day one.

About the theme—how good is your metadata?

The reach and usefulness of research outputs are only as good as how well they are described. Metadata is what is used to describe the story of research: its origin, its contributors, its attention, and its relationship with other objects.

The more machines start to do what humans cannot—parse millions of files through multiple views—the more we see what connections are missing, the more we start to understand the opportunities that better metadata can offer.

LIVE18 will focus this year entirely on the subject of metadata. It touches everything we do, and everything that publishers, hosting platforms, funders, researchers, and libraries do.

Come and join the discussions

Register to join us this 13 and 14 November, at the Toronto Reference Library, 789 Yonge Street, Toronto, Canada—we look forward to seeing you there.

Using the Crossref REST API. Part 11 (with MDPI/Scilit)

Christine Cormack Wood — Tue, 18 Sep 2018 00:00:00 +0000

Continuing our blog series highlighting the uses of Crossref metadata, we talked to Martyn Rittman and Bastien Latard who tell us about themselves, MDPI and Scilit, and how they use Crossref metadata.

Can you give us a brief introduction yourselves, and to MDPI/Scilit

Martyn is Publishing Services Manager at MDPI. He joined five years ago as an editor and has worked on editorial, production, and software projects. Prior to joining MDPI, he completed a PhD and worked as a postdoc. His research covered physical chemistry, biochemistry and instrument development. Bastien Latard is the project leader of Scilit. He created Scilit as part of his Master’s degree in 2013. He is now completing a PhD on the subject of semantically linking research articles, using data from Scilit.

Scilit was developed in 2014 by open access (OA) publisher MDPI with the goal of having a backup of metadata for all OA articles. Soon, Scilit became more general and embraced all articles with a digital object identifier (DOI) from Crossref and those with a Pubmed ID (PMID). After seeing the potential of the database and how it could be used in a number of different contexts, we decided to make it public. Recently, other article types, including preprints have been integrated. Our main goal now is to provide useful services to the research and academic publishing communities.

What problem is your service trying to solve?

Other indexing databases offer paid access, are highly selective, or host documents apart from research articles. We want to offer a comprehensive database, but also one that clearly identifies open access material. The last part is still a work in progress, but we have made good progress recently.

To make the access as direct as possible, we have recently integrated several OA aggregators that pick up or host free versions of full-text articles, including CORE, Unpaywall, and PubMed Central.

Can you tell us how you are using the Crossref Metadata API at MDPI/Scilit?

Scilit queries Crossref’s API in order to index metadata for single articles. DOIs are a key part of the system; because they are standards, we can use them to merge new sources into Scilit while avoiding duplicates. We cross-check the data from Crossref against other sources and update it as necessary. Citation data is also really appreciated and opens doors to further developments.

As a publisher, MDPI makes daily deposits to Crossref, to register journal articles on mdpi.com, conference papers from sciforum.net, and preprints from Preprints.org. We also use the data collected at Scilit to find suitable reviewers and let authors know when their work has been cited.

What metadata values do you pull from the API?

As much as we can! Scilit crawls the latest indexed articles every few hours to ensure it is as up-to-date as possible. This is the most important function of our system because it provides metadata for the very latest published articles, including a link to the publisher version. Scilit parses Crossref metadata and saves them. They are then indexed into our solr search engine for fast, real-time usage.

Have you built your own interface to extract this data?

We wrote our own code to get the data, but the API interface made this very straightforward. Scilit has been developed completely in-house by MDPI and the lead developer, Bastien Latard, is currently completing a PhD looking at how to make the most of the data using semantic data extraction.

What are the future plans for MDPI/Scilit?

Scilit is and will be highly used in MDPI current and future projects. We have a few ideas about how to improve Scilit. We are, for example, implementing a scientific profile networking service, which will allow scholars to build their own (scientific) network with lots of functionalities. We think that it will be a really good place to search, comment, exchange around articles… maybe even more!

What else would you like to see the REST API offer?

Crossref is already doing a great job, especially with its integrated citation data. Maybe further analysis and mapping of data about organisations and institutions would be an improvement.

Thank you Martin and Bastien. If you’d like to share how you use the Crossref Metadata APIs please contact the Community team.

Where does publisher metadata go and how is it used?

Laura J Wilkinson — Mon, 17 Sep 2018 00:00:00 +0000

Earlier this week, colleagues from Crossref, ScienceOpen, and OPERAS/OpenEdition joined forces to run a webinar on “Where does publisher metadata go and how is it used?”.

Stephanie Dawson explained how ScienceOpen’s freely-accessible, interactive search and discovery platform works by connecting and exposing metadata from Crossref. Her case study showed that articles with additional metadata had much higher average views than those without - depositing richer metadata helps you get the best value from your DOIs!

Pierre Mounier of OPERAS/OpenEdition showed us how a variety of persistent identifiers (PIDs) including DOIs, ORCID iDs, and Funder Registry IDs have been used on OA book platforms to improve citations, author attribution, and tracking of funding. He described a forthcoming annotations project with Hypothes.is, and explained how Crossref metadata is being used in both usage and alternative metrics.

Five ways to register content with Crossref

My overview of Content Registration outlined the five ways to register content with Crossref:

Via the manual web deposit form
Through Crossref’s new Metadata Manager tool (beta)
With OJS’s Crossref plugin - more information here (see OJS downloads Version 3.1.0 and above is the best option for supporting the fullest Crossref metadata)
With a manual XML upload file
Or, using HTTPS to POST XML

I also emphasized the importance of depositing, adding, and updating your metadata, and spoke about:

Basic citation metadata: titles, author names, author affiliations, funding data, publication dates, issue numbers, page numbers, ISSNs, ISBNs…
Non-bibliographic metadata: reference lists, ORCID iDs, license data, clinical trial information, abstracts, relationships…
Crossmark: errata, retractions, updates, and more
How important it is to have accurate, clean, and complete metadata
The importance of registering your back-year records

How to see the metadata you have

Anna Tolwinksa, Crossref’s Member Experience Manager, gave us an overview of the new Participation Reports tool. She explained how Participation Reports allows anyone to see the metadata Crossref members have registered with us, and how you can see for yourself where the gaps in your metadata are, and—importantly—how you can improve your coverage.

What we learnt

There are 10 key metadata elements or checks in Participation Reports that aid in Crossref members’ content discoverability, reproducibility and research integrity:
- References
- ~~Open References~~ [EDIT 6th June 2022 - all references are now open by default].
- ORCID iDs
- Funder Registry IDs
- Funding award numbers
- Text mining URLs
- License URLs
- Similarity Check URLs
Every day, research organisations around the world rely on metadata from Crossref, and use it in a variety of systems. Here are a few examples. Many organisations that enable research depend on Crossref’s metadata; we received over 650 million queries just last month
Crossref members should check Participation Reports to see what percentage of their content includes rich metadata If the percentages are low, Crossref is happy to work with you to help understand and improve your coverage
Richer metadata helps research to be found, cited, linked to, assessed, and reused
To make sure your work can be found!

Catch up with the webinar recording, and slides from Laura, Stephanie, Pierre, and Anna’s presentations, and please contact us if you have any questions.

Event Data is production ready

Christine Buske — Wed, 12 Sep 2018 00:00:00 +0000

We’ve been working on Event Data for some time now, and in the spirit of openness, much of that story has already been shared with the community. In fact, when I recently joined as Crossref’s Product Manager for Event Data, I jumped onto an already fast moving train—headed for a bright horizon.

What’s on the horizon? Well, the reality is you never really reach the horizon. Good product development—in my opinion—is like that train. You keep aiming for the horizon and passing all the stations (milestones) along the way, but the horizon keeps moving as you add features, improve the service, and maybe even review where you are headed. However, for Event Data we are pleased to say we have now arrived at a rather important station.

Technical readiness

Thank you to all the beta testers who have journeyed with us this far—we’ve listened and learned, refined and rebuilt with the help of your feedback. We are now thrilled to say that we are service production ready. We’ve reached the station called ‘technical readiness’, and are eager to see more users board our train!

During this time of building and refining, Event Data has grown to include at least 66,7 million events from sources like (in order of magnitude): Wikipedia, Cambia Lens, Twitter, Datacite, F1000, Newfeeds, Reddit links, Wordpress.com, Crossref, Reddit, Hypothesis, and Stackexchange. Wikipedia alone accounts for 50 million events (and counting).

What does this mean?

Event Data is production ready.

Being production ready means we are not going to make any breaking changes to the code, and we are excited to see more people jump on board to explore where you can go with Event Data, and what product or service you might want to build with it.

Getting started

Having a look at Event Data, and using it, is easy. While the user guide outlines everything you need to know to get fully engrossed, you can get your feet wet with a few sample queries:

Above I mentioned Event Data has about 50 million Wikipedia events, you can check if that has grown by looking at a query that lists all distinct events by source (your browser will need a JSON viewer extension):

https://api.eventdata.crossref.org/v1/events/distinct?facet=source/:*&rows=0

You can also see a live stream of events going through Event Data.

For all events registered for a specific content item, you simply query http://api.eventdata.crossref.org/v1/events?obj-id=https://doi.org/XXX, where XXX is replaced with the DOI.

What next?

We are now focusing on the final stretch towards the official roll-out. Beyond this, we will continue to add sources and features and have a healthy roadmap to keep us on track. We value any feedback you have for us about your own journey with Event Data. Your feedback may help shape the direction we take in the future. Most of all, we are all excited to see what people build with it!

We look forward to continuing on our Event Data journey and we welcome you all aboard the train! Please contact me with your ideas.

Crossref at the Frankfurt Book Fair 2018

Amanda Bartell — Tue, 11 Sep 2018 00:00:00 +0000

How good is your metadata? Find out at the Frankfurt Book Fair…

At the Frankfurt Book Fair this year (Hall 4.2, Stand M82), the Crossref team will be on hand to give you a personal tour of our new Participation Reports tool. Or join us at The Education Stage to hear about how this new tool can help you view, evaluate and improve your metadata participation.

How good is your metadata? Join us Thursday 11th October at 15.30 at the Education Stage in Hall 4.2 to find out

Lots of reasons to visit our stand

We’ll be located in the same place as last year, Hall 4.2, Stand M82, and there are lots of reasons to visit us:

Get your metadata participation evaluated - Anna Tolwinska and Amanda Bartell will walk you through your own Participation Report and provide guidance on how to improve your results. Discover how complete your metadata is, where the gaps are, and how other publishers compare.

Discuss a technical issue that’s hindering your metadata participation (or any other technical issue) with Isaac Farley and Paul Davis from our Technical Support team.

Jennifer Kemp will also be around to answer all your metadata use and reuse questions. She’s looking forward to chatting with all kinds of service providers and toolmakers.

On the strategy side, Ginny Hendricks will be there on Wednesday 10th if you’d like to discuss any policy stuff, new ideas, or find out what Crossref is planning next.

Ask us anything

Not just Participation Reports—you can ask us about anything. Perhaps about our newer record types such as preprints, pending publications (i.e. DOIs on acceptance), or data citations. Or, ask us how you can:

Advance scholarly pursuits for the benefit of society, through Metadata 2020
Check papers for originality, with our service for editorial rigour, through Similarity Check
Discover where and how research is being discovered, through Event Data
Reveal who is citing your published papers and how platforms can display this information, with our Cited-by service
Provide evidence of trust in published outputs, revealing updates, corrections and retractions, through our Crossmark service

Let us know if you’d like to book in a meeting with one of us, or do just stop by the stand to say “Guten Tag”.

We look forward to seeing you there - bis dann!

Presenting PIDapalooza 2019

Christine Cormack Wood — Tue, 28 Aug 2018 00:00:00 +0000

PIDapalooza, the open festival of persistent identifiers is back and it’s better than ever. Mark your calendar for Dublin, Ireland, January 23-24, 2019 and send us your session ideas by September 21.

Yes, it’s back and – with your support – it’s going to be better than ever! The third annual PIDapalooza open festival of persistent identifiers will take place at the Griffith Conference Centre, Dublin, Ireland on January 23-24, 2019 - and we hope you’ll join us there!

Hosted, once again, by California Digital Library, Crossref, DataCite, and ORCID, PIDapalooza will follow the same format as past events – rapid-fire, interactive, 30-60 minute sessions (presentations, discussions, debates, brainstorms, etc.) presented on three stages – plus main stage attractions, which will be announced shortly. New for this year is an unconference track, as suggested by several attendees last time.

In the meantime, get those creative juices flowing and send us your session PIDeas! What would you like to talk about? Hear about? Learn about? What’s important for your organisation and your community and why? What’s working and what’s not? What’s needed and what’s missing? We want to hear from as many PID people as possible! Please use this form to send us your suggestions. The PIDapalooza Festival Committee will review all forms submitted by September 21, 2018 and decide on the lineup by mid-October.

As a reminder, the regular themes are:

PID myths: Are PIDs better in our minds than in reality? PID stands for Persistent IDentifier, but what does that mean and does such a thing exist?
PIDs forever - achieving persistence: So many factors affect persistence: mission, oversight, funding, succession, redundancy, governance. Is open infrastructure for scholarly communication the key to achieving persistence?
PIDs for emerging uses: Long-term identifiers are no longer just for digital objects. We have use cases for people, organisations, vocabulary terms, and more. What additional use cases are you working on?
Legacy PIDs: There are of thousands of venerable old identifier systems that people want to continue using and bring into the modern data citation ecosystem. How can we manage this effectively?
Bridging worlds: What would make heterogeneous PID systems ‘interoperate’ optimally? Would standardized metadata and APIs across PID types solve many of the problems, and if so, how would that be achieved? What about standardized link/relation types?
PIDagogy: It’s a challenge for those who provide PID services and tools to engage the wider community. How do you teach, learn, persuade, discuss, and improve adoption? What’s it mean to build a pedagogy for PIDs?
PID stories: Which strategies worked? Which strategies failed? Tell us your horror stories! Share your victories!
Kinds of persistence: What are the frontiers of ‘persistence’? We hear lots about fraud prevention with identifiers for scientific reproducibility, but what about data papers promoting PIDs for long-term access to reliably improving objects (software, pre-prints, datasets) or live data feeds?

We’ll be posting more information on the PIDapalooza website over the coming months, as well as keeping you updated on Twitter (@pidaplooza).

In the meantime, what are you waiting for!? Book your place now – and we also strongly recommend that you book your accommodation early as there are other big conferences in Dublin that week.

PIDapalooza, Dublin, Ireland, January 23-24, 2019 - it’s a date!

Leaving the house - where preprints go

Jennifer Lin — Tue, 21 Aug 2018 00:00:00 +0000

“Pre-prints” are sometimes neither Pre nor Print (c.f. https://doi.org/10.12688/f1000research.11408.1, but they do go on and get published in journals. While researchers may have different motivations for posting a preprint, such as establishing a record of priority or seeking rapid feedback, the primary motivation appears to be timely sharing of results prior to journal publication.

So where in fact do preprints get published?

Although this is a simple question, we have not had an easy way to answer how this varies across disciplines, preprint repositories and journals. Until now. Crossref metadata provides not only an open and easy way to do so, but up-to-date data to get the latest results.

rOpenSci makin’ it sweet & easy

Crossref asks preprint repositories to update their metadata once a preprint has been published by adding the article link into its record via the “is-preprint-of” relation. As the record is processed, we make the link available going both directions, while preserving the provenance of the statement in the metadata output (“asserted-by”: “subject” or “asserted-by”: “object”). This results in bidirectional assertions in the Crossref REST API where search engines, analytics providers, indexes, etc. can get from the preprint to the article (“is-preprint-of”) as well as vice versa (“has-preprint”), making it easier to find, cite, link, assess, and reuse.

Using rOpenSci’s R library for the Crossref REST API (rcrossref), we pulled all articles connected to a previous preprint (https://api.crossref.org/works?filter=relation.type:has-preprint&facet=publisher-name:&rows=0) and then aggregated them based on journal via their ISSNs (https://api.crossref.org/works?filter=relation.type:has-preprint&facet=issn:), tallying the results in a tidy table with the journal name (ex: PLOS Biology (https://api.crossref.org/journals/2167-8359)).

The big reveal

So without further delay, let’s look at the results of the 20 journals with the highest number of preprints associated with its articles (data from August 21, 2018):

Publisher	Journal	Count
PeerJ	PeerJ	1184
Springer Nature	Scientific Reports	394
eLife	eLife	375
PLOS	PLOS ONE	338
Proceedings of the National Academy of Sciences	PNAS	205
PLOS	PLOS Computational Biology	196
Springer Nature	Nature Communications	187
PLOS	PLOS Genetics	169
The Genetics Society of America	Genetics	168
Oxford University Press	Nucleic Acids Research	148
Oxford University Press	Bioinformatics	138
The Genetics Society of America	Genetics	120
The Genetics Society of America	G3: Genes, Genomes, Genetics	104
Cold Spring Harbor Laboratory	Genome Research	104
Oxford University Press	Molecular Biology and Evolution	100
MDPI AG	Energies	98
MDPI AG	Sensors	96
Springer Nature	BMC Genomics	92
MDPI AG	International Journal of Molecular Sciences	86
JMIR Publications	Journal of Medical Internet Research	83

This list has not been normalized or weighted based on the size of the journal. The following observations are informed speculations, as we can only infer so much from the raw data:

Disciplinary practice: This phenomenon where preprints are a part of disciplinary practice accounts for about half of the journals represented on the list. Certain communities such as genetics and computational fields have been early adopters of preprints. As such, we see higher rates of preprint-to-article publication in journals that publish their work.
Partnerships: Partnerships that facilitate submission from the preprint repository directly to a publisher or peer review service (ex: BioRxiv B2J program) make it easier for researchers to move from preprint-sharing seamlessly to submitting their journal article manuscript.
Tie-ins: A quarter of the journals on the list are run by publishers with a preprint service, and have been able to tie together both arms of publishing. This removes barriers to journal article submission in the same manner as integrations between repositories and publishers, but does so as a single party.
Publisher support and treatment: We also see that strong proponents and early partners of preprint repositories tend to have higher counts. Some publishers have been more outspoken in their welcome of preprints, such as PNAS. Sometimes this support also comes in the form of special treatment. In the process of crafting editorial policy on publishing results previously posted in a preprint, some journals have carved out particular affordances in their publication workflow and content delivery streams that may contribute to the higher counts of articles. For example, Nature Research displays the preprints of submitted articles under consideration: https://nature-research-under-consideration.nature.com/.
Mega-journals: Mega-journals such as Scientific Reports and PLOS ONE have not discouraged preprints. As such, and due to the size of their publication output, they have easily found a place among the higher counts on the list.

Taking a closer look

One major consideration in these results, concerns what’s missing in the data. These fall into two camps: incomplete member data, and incomplete membership coverage.

We have been working with our members to deposit preprints using the proper record type, and to provide links to published articles in their metadata. However, not all have yet done so (ex: SSRN), leading to holes in our research nexus graph, which subsequently detracts from the completeness of the data.

We celebrate the preprint repositories who are required to update their metadata when an article is published from a preprint, thereby populating the map with critical bridges between preprints and articles. Crossref participation benefits not only the content owner, but the membership at large and all the systems across the research ecosystem powered by Crossref metadata.

Lastly, this data is dependent on the coverage of preprint repositories who register content with us. We are thrilled that Center for Open Science, our newest preprints addition who represents 21 community repositories, has recently filled in swaths of the map. But there remain dead zones in the research graph from repositories who are not Crossref members (ex: ArXiv). Their disciplines, as a result, are under represented in these results.

Everyone dive in!

As to the question of “where do preprints get published?”, anyone in fact can answer this question based on the metadata Crossref collects and provides to the community as an open infrastructure provider. We encourage the community to explore and analyze the data further with other available datasets to glean more insights on how scholarly communications is changing with the increasing growth of preprints. For example, the effective results across all journals represented can be weighted based on the number of articles published by each journal.

Crossref data is open for all to examine and reuse through our REST API. Please dive in and share your findings with us!

2018 election slate

Lisa Hart Martin — Fri, 17 Aug 2018 00:00:00 +0000

With Crossref developing and extending its services for members and other constituents at a rapid pace, it’s an exciting time to be on our board. We recieved 26 expressions of interest this year, so it seems our members are also excited about what they could help us achieve.

From these 26, the Nominating Committee has put forward the following slate.

The 2018 slate: seven candidates for five available seats

African Journals OnLine (AJOL), Susan Murray, South Africa
American Psychological Association (APA), Jasper Simons, USA
Association for Computing Machinery (ACM), Scott Delman, USA
California Digital Library (CDL), Catherine Mitchell, USA
Hindawi, Paul Peters, UK
Sage, Richard Fidczuk, USA
Wiley, Duncan Campbell, USA

Read the candidates’ organisational and personal statements

Candidates were chosen based on the following criteria:

Follow the guidance from the Board to provide a slate or seven or fewer.
Maintain the current balance of the board with respect to size of organisations.
Improve balance in other areas, with respect to gender and geography.
Also consider types of organisations and sector, as well as engagement with Crossref and its services.

You can be part of this important process, by voting in the election

If your organisation is a member of Crossref on September 14, 2018 you are eligible to vote when voting opens on September 28, 2018 (affiliates, however, are not eligible to vote).

How can you vote?

On September 28, 2018, your organisation’s designated voting contact will receive an email with a link to the formal Notice of Meeting and Proxy Form with concise instructions on how to vote. An additional email will be sent with a username and password along with a link to our online voting platform. It is important to make sure your voting contact is up-to-date.

Want to add your voice?

We are accepting independent nominations until November 7, 2018. organisations interested in standing as an independent candidate should contact me by this date with a list of ten other Crossref members that endorse their candidacy.

The election itself will be held at LIVE18 Toronto, our annual meeting, on 13 November 2018 in Canada. We hope you’ll be there to hear the results.

Using the Crossref REST API. Part 10 (with Kudos)

Christine Cormack Wood — Mon, 13 Aug 2018 00:00:00 +0000

Continuing our blog series highlighting the uses of Crossref metadata, we talked to David Sommer, co-founder and Product Director at the research dissemination management service, Kudos. David tells us how Kudos is collaborating with Crossref, and how they use the REST API as part of our Metadata Plus service.

Introducing Kudos

At Kudos we know that effective dissemination is the starting point for impact. Kudos is a platform that allows researchers and research groups to plan, manage, measure, and report on dissemination activities to help maximize the visibility and impact of their work.

We launched the service in 2015 and now work with almost 100 publishers and institutions around the world, and have nearly 250,000 researchers using the platform.

We provide guidance to researchers on writing a plain language summary about their work so it can be found and understood by a broad range of audiences, and then we support researchers in disseminating across multiple channels and measuring which dissemination activities are most effective for them.

As part of this, we developed the Sharable-PDF to allow researchers to legitimately share publication profiles across a range of sites and networks, and track the impact of their work centrally. This also allows publishers to prevent copyright infringement, and reclaim lost usage from sharing of research articles on scholarly collaboration networks.

An example of a Kudos publication page showing the plain language summary

How is Crossref metadata used in Kudos?

Since our launch, Crossref has been our metadata foundation. When we receive notification from our publishing partners that an article, book or book chapter has been published, we query using the Crossref REST API to retrieve the metadata for that publication. That data allows us to populate the Kudos publication page.

We also integrate earlier in the researcher workflow, interfacing with all of the major Manuscript Submission Systems to support authors who want to build impact from the point of submission.

More recently, we started using the Crossref REST API to retrieve citation counts for a DOI. This enables us to include the number of times content is cited as part of the ‘basket of metrics’ we provide to our researchers. They can then understand the performance of their publications in context, and see the correlation between actions and results.

A Kudos metrics page, showing the basket of metrics and the correlation between actions and results

What are the future plans for Kudos?

We have exciting plans for the future! We are developing Kudos for Research Groups to support the planning, managing, measuring and reporting of dissemination activities for research groups, labs and departments. We are adding a range of new features and dissemination channels to support this, and to help researchers to better understand how their research is being used, and by whom.

What else would Kudos like to see in Crossref metadata?

We have always found Crossref to be very responsive and open to new ideas, so we look forward to continuing to work together. We are keen to see an industry standard article-level subject classification system developed, and it would seem that Crossref is the natural home for this.

We are also continuing to monitor Crossref Event Data which has the potential to provide a rich source of events that could be used to help demonstrate dissemination and impact.

Finally, we are pleased to see the work Crossref are doing to help improve the quality of the metadata and supporting publishers in auditing their data. If we could have anything we wanted, our dream would be to prevent “funny characters” in DOIs that cause us all kinds of escape character headaches!

Thank you David. If you would like to contribute a case study on the uses of Crossref Metadata APIs please contact the Community team.

Peer review publications

Jennifer Lin — Sun, 12 Aug 2018 00:00:00 +0000

Peer review publications—not peer-reviewed publications, but peer reviews as publications

Our newest dedicated record type—peer review—has received a warm welcome from our members since rollout last November. We are pleased to formally integrate them into the scholarly record, giving the scholars who participated credit for their work, ensuring readers and systems dependably get from the reviews to the article (and vice versa), and making sure that links to these works persist over time.

Many of our members make the peer review history available to researchers in different ways. Their extra effort to post review materials alongside the article will now go further once they are registered with us and linked to the journal article. They spoke of publishing peer reviews as a standard part of their publishing operation. The scholarly contributions of their editors and referees are validated, stewarded, and published in the manner of the articles: as per general practice. To fully realize this, they are ensuring that these publications are discoverable, citable, and part of the formal scholarly record—for all the thousands of systems which draw on Crossref metadata.

Article metadata + peer review metadata = a fuller picture of the evolution of knowledge

The growing collection

As of August 12, 2018 three publishers have registered 12446 peer reviews in the dedicated resource type schema we rolled out last November. PeerJ (10.7287) with 12015 at time of writing and Stichting SciPost (10.21468) with 297 works. ScienceOpen (10.14293) has registered 126 reviews of papers on their post-publication platform.

The peer review metadata collected is partly similar, though otherwise unique to other content. In the former, general metadata that we accept for the articles, as well as the reviews, include an ORCID iD to identify the reviewer, editor, and/or author 0; license 0. This metadata is quite distinct from the article and is important to collect, not only as a discrete publication in its own right, but also to provide richer context for the actual results shared in the associated article. They are authored by different people than the paper’s contributors (author response/rebuttal excepting). They need not have the same license.

Currently, none of this data has been registered. (From the publishers we’ve talked to, this is largely due to factors related to limitations in their technology systems.) And like other record types, we link up scholarly materials in the metadata and fill in the research nexus graph through relations.

There’s no better way to understand peer review metadata than to look at real examples from our members:

PeerJ review (https://doi.org/10.7287/peerj.2707v0.1/reviews/1) and its metadata (https://api.crossref.org/works/10.7287/peerj.2707v0.1/reviews/1)
ScienceOpen review (https://doi.org/10.14293/s2199-1006.1.sor-uncat.a5995373.v1.rhrmgu) and its metadata (https://api.crossref.org/works/10.14293/s2199-1006.1.sor-uncat.a5995373.v1.rhrmgu)
SciPost review (https://doi.org/10.21468/scipost.report.10) and its metadata (https://api.crossref.org/works/10.21468/scipost.report.10)

Review-specific metadata is also critical to capturing the shape of the scholarly discussion. These include:

Review date (required)
Scholarly work reviewed (required)
Recommendation
Revision stage
Review round
Contributor name

PeerJ, SciPost, and ScienceOpen have registered this whole set where applicable (review round not applicable to post-publication reviews), with the exception of the recommendation.

Scholarly contributions captured in time

Published peer reviews uniquely highlight the nature of research ideas evolving over time, spotlighting the nature of this as a collective effort involving multiple individuals. The more metadata, the bolder the story. We have created a set of reference metadata (fictitious) to illustrate this phenomenon. Josiah Carberry submits a manuscript to the Journal of Psychoceramics, entitled “Dog: A Methodology for the Development of Simulated Annealing.” It undergoes two rounds of review with two referees each round. The article https://doi.org/10.5555/12345681 is published and registered on May 6, 2012 along with the history of peer review materials on the same day:

First submission

Referee report 1 - https://doi.org/10.5555/12345681.9879
Referee report 2 - https://doi.org/10.5555/12345681.9880
Editor decision - https://doi.org/10.5555/12345681.9881

Revision round 1

Author rebuttal - https://doi.org/10.5555/12345681.9882
Referee report 1 - https://doi.org/10.5555/12345681.9883
Referee report 2 - https://doi.org/10.5555/12345681.9884
Editor decision - https://doi.org/10.5555/12345681.9885

Published reviews can show peer feedback in progress; the progress of scholarly discussion unfolding, as expert ideas build upon each other. Many of us have traditionally located the article’s publication as the climactic event, but the story in fact doesn’t end there. Pre-publication becomes post-publication. Throughout this time, research is validated and sprouts into new ideas.

Peer review platform Publons is working on getting reviews authored on its platform registered with us. Doing so will mean that PeerJ article, “Transformative optimisation of agricultural land use to meet future food demands” by Lian Pin Koh, Thomas Koellner, and Jaboury Ghazoul https://doi.org/10.7717/peerj.188 with three scholarly discussions published over the course of peer review, would also be accompanied by a fourth that occurred after publication from Gene A. Bunin https://publons.com/publon/3374/, not yet registered.

Research begets research

In my investigation of review publications registered, two examples cropped up, highlighting the richness of the research process not only as it shows a set of research results evolve through scholarly discussion, but as it is then folded into new research outputs.

A PeerJ article “Software citation principles” https://doi.org/10.7717/peerj-cs.86 has had a very rich life: https://api.crossref.org/works/10.7717/peerj-cs.86. It was originally submitted as a preprint and underwent multiple iterations of improvement (https://doi.org/10.7287/peerj.preprints.2169, https://doi.org/10.7287/peerj.preprints.2169v1, https://doi.org/10.7287/peerj.preprints.2169v2, etc.). It then was subjected to peer review. And three referee reports are published alongside the final publication:

We glimpse a view of time unfolding here:

NB: in the review metadata, all the dates provided reference September 19, 2016 when they were published with the accompanying research article. To really make the metadata useful, we recommend providing the date the review was received, rather than published (for publishers who are publishing pre-publication review materials).

The reviews were then cited in three versions of the F1000Research article, “A multi-disciplinary perspective on emergent and future innovations in peer review” (https://doi.org/10.12688/f1000research.12037.1, https://doi.org/10.12688/f1000research.12037.2, and https://doi.org/10.12688/f1000research.12037.3). These three all link up on the Crossref metadata map. The visualization below is only an entrypoint into this picture of research dissemination and the spread of ideas.

András Láng served as a reviewer for a paper by Danilo Garcia and Fernando R. González Moraga published as “The Dark Cube: dark character profiles and OCEAN” (https://doi.org/10.7717/peerj.3845). As of the blog release date, this paper has been cited by two sources:

Source: https://doi.org/10.7717/peerj.3845, CC-BY 4.0

What this view of the paper does not reveal is that Láng’s review (https://doi.org/10.7287/peerj.3845v0.1/reviews/2) provided such insight to the original researchers that the first author (Garcia) incorporates the discussion in his subquent work. This evidence is documented in the citation list of that new publication, “Encyclopedia of Personality and Individual Differences” https://doi.org/10.1007/978-3-319-28099-8_2302-1. What a wonderful illustration of the ways in which peer reviews can operate like other publications, and how far is it from being unique. But up to now, we have not yet programmatically captured them in a formal way as we do now with these materials registered properly as a review.

The evolution of Crossref’s piece

In the same spirit of ever evolving knowledge, we also continue to update our schemas based upon community feedback. Are references important? Tell us! What new metadata on peer reviews are important to answer your questions or help you do what you need? Members, if you are interested in registering your peer review content with us, please get in touch.

Org ID: a recap and a hint of things to come

John Chodacki — Thu, 02 Aug 2018 00:00:00 +0000

Cross-posted on the blogs of University of California (UC3), ORCID, and DataCite: https://doi.org/10.5438/67sj-4y05.

Over the past couple of years, a group of organisations with a shared purpose—California Digital Library, Crossref, DataCite, and ORCID—invested our time and energy into launching the Org ID initiative, with the goal of defining requirements for an open, community-led organisation identifier registry. The goal of our initiative has been to offer a transparent, accessible process that builds a better system for all of our communities. As the working group chair, I wanted to provide an update on this initiative and let you know where our efforts are headed.

Community-led effort

First, I would like to summarize all of the work that has gone into this project, a truly community-driven initiative, over the last two years:

A series of collaborative workshops were held at the Coalition for Networked Information (CNI) meeting in San Antonio TX (2016), the FORCE11 conference in Portland OR (2016), and at PIDapalooza in Reykjavik (2016).
Findings from these workshops were summarized in three documents, which we made openly available to the community for public comment:
organisation Identifier Project: A Way Forward (PDF)
organisation Identifier Provider Landscape (PDF)
Technical Considerations for an organisation Identifier Registry (PDF)
A Working Group worked throughout 2017 and voted to approve a set of recommendations and principles for ‘governance’ and ‘product’:
Governance Recommendations
Product Principles and Recommendations
We then put out a Request for Information that sought expressions of interest from organisations to be involved in implementing and running an organisation identifier registry.
There was a really good response to the RFI; reviewing the responses and thinking about next steps led to our most recent stakeholder meeting in Girona in January 2018, where ORCID, DataCite, and Crossref were tasked with drafting a proposal that meets the Working Group’s requirements for a community-led, organisational identifier registry.

Thank you

I want to take this opportunity to thank everyone who has contributed to this effort so far. We’ve been able to make good progress with the initiative because of the time and expertise many of you have volunteered. We have truly benefited from the support of the community, with representatives from Alfred P. Sloan Foundation; American Physical Society, California Digital Library, Cornell University, Crossref, DataCite, Digital Science, Editeur, Elsevier, Foundation for Earth Sciences, Hindawi, Jisc, ORCID, Ringgold, Springer Nature, The IP Registry, and U.S. Geological Survey involved throughout this initiative. And we couldn’t have done any of it without the help and guidance of our consultants, Helen Szigeti and Kristen Ratan.

The way forward

The recommendations from our initiative have been converted into a concrete plan for building a registry for research organisations. This plan will be posted in the coming weeks.

The initiative’s leadership group has already secured start-up resourcing and is getting ready to announce the launch plan—more details coming soon.

We hope that all stakeholders will continue to support the next phase of our work – look for announcements in the coming weeks about how to get involved.

As always, we welcome your feedback and involvement as this effort continues. Please contact me directly with any questions or comments at john.chodacki@ucop.edu. And thanks again for your help bringing an open organisation identifier registry to fruition!

References

Bilder, G., Brown, J., & Demeranville, T. (2016). Organisation identifiers: current provider survey. ORCID. https://doi.org/10.5438/4716

Cruse, P., Haak, L., & Pentz, E. (2016). organisation Identifier Project: A Way Forward. ORCID. https://doi.org/10.5438/2906

Fenner, M., Paglione, L., Demeranville, T., & Bilder, G. (2016). Technical Considerations for an organisation Identifier Registry. https://doi.org/10.5438/7885

Laurel, H., Bilder, G., Brown, C., Cruse, P., Devenport, T., Fenner, M., … Smith, A. (2017). ORG ID WG Product Principles and Recommendations. https://doi.org/10.23640/07243.5402047

Laurel, H., Pentz, E., Cruse, P., & Chodacki, J. (2017). organisation Identifier Project: Request for Information. https://doi.org/10.23640/07243.5458162

Pentz, E., Cruse, P., Laurel, H., & Warner, S. (2017). ORG ID WG Governance Principles and Recommendations. https://doi.org/10.23640/07243.5402002

3,2,1… it’s ‘lift-off’ for Participation Reports

Anna Tolwinska — Wed, 01 Aug 2018 00:00:00 +0000

Metadata is at the heart of all our services. With a growing range of members participating in our community—often compiling or depositing metadata on behalf of each other—the need to educate and express obligations and best practice has increased. In addition, we’ve seen more and more researchers and tools making use of our APIs to harvest, analyze and re-purpose the metadata our members register, so we’ve been very aware of the need to be more explicit about what this metadata enables, why, how, and for whom.

This week we take an important step towards this goal with a much-anticipated announcement: Participation reports are in beta release—so come along and take a look!

What does this mean?

Participation Reports gives—for the first time—a clear visualization of the metadata that Crossref has. Search for any member to find out what percentage of their content includes 10 key elements of information, above and beyond the basic bibliographic metadata that all members are obliged to provide. This includes metadata such as ORCID iDs for contributors, funding acknowledgements, reference lists, and abstracts—richer metadata that makes content more discoverable, and much more useful to the scholarly community as a whole, including among members themselves.

You can filter by content such as journal articles, book chapters, datasets, and preprints, and compare current content (past two calendar years and year-to-date) to back file content (older than that). And within the journal articles view, you can drill down to view the metadata completeness for each individual journal. We hear that editorial boards are keen to see that aspect!

We’re delighted that participation reports are now available in beta. That means that while we are confident that the data shown is accurate, there could be the odd glitch as we monitor use.

Thank you to everyone who has helped us to test the reports and provided so much valuable feedback. We plan to expand and improve participation reports to include additional metadata elements, metadata quality checks, and adherence to Crossref best practice such as DOI display. We’re still listening so do get in touch if you have questions or suggestions, or would like a more detailed walk through. There is also a feedback button right in-situ in the tool.

Crossref LIVE and local (to you)

Vanessa Fairhurst — Wed, 18 Jul 2018 00:00:00 +0000

The last few months have been busy for the Crossref community outreach team. We’ve been out and about from Cape Town to Ulyanovsk—and many places in between—talking at ‘LIVE locals’ to members about all things metadata. Our LIVE locals are one-day events, held around the world—but local to you—that provide both deeper insight into Crossref, and information on our services and how to benefit from them. These events are always free to attend, and whether you are a long-established member, totally new, or not even a member at all, we welcome you all to join us.

At our most recent events we collaborated with some fantastic organisations and welcomed attendees from a variety of backgrounds including editors, publishers, service providers, researchers and other metadata users.

South Africa

In April Chuck Koscher, Director of Technology, and I travelled to South Africa for two LIVE locals, one in Pretoria and the other in Cape Town—and both in collaboration with the Academy of Science of South Africa (ASSAf). ASSAf also provided two excellent speakers, Nadine Wubbeling (ASSAf) and Pierre de Villiers (AOSIS), who shared their experiences with Crossref and presented valuable insights into the work that they do.

Delivering events for a varied audience like this means there are often differing levels of knowledge and experience. So, to make sure everyone benefited from our sessions, we covered the different ways you can work with the Crossref deposit system as an XML pro, or an absolute beginner. This included a live demonstration of our new deposit tool Metadata Manager (currently in beta) which should help those less technically-minded people (like myself), and be a big improvement upon our current web deposit form.

|||

The day ended with a technical session, where attendees discussed specific issues they needed help with, which mainly focussed on retrieving metadata in the Crossref system, interpreting reports, and support with XML.

Images left to right: Dr. Pierre de Villiers talks about the Crossref Experience at AOSIS, and the stunning scenery of Table Mountain provided a nice backdrop to our Cape Town event.

Russia

Just back from a few days in Russia 🇷🇺. We ran a @CrossrefOrg LIVE local in Ulyanovsk for 60 editors, made plans to do more education and outreach in the region and caught a #FifaWorldCup2018 game... pic.twitter.com/GSdNEujJXa
— Rachael Lammey (@rachaellammey) June 22, 2018

The World Cup wasn’t the only big event in Russia last month. That’s right, we were there too—with our very first Russian LIVE local! On the 19th June, 60 attendees from a range of academic and publishing institutions joined us at The Ulyanovsk State Pedagogical University. Rachael Lammey and I introduced Crossref, the role of identifiers, and how to register different resource types with us. We also discussed the use and importance of providing accurate and comprehensive metadata, and shared some interesting use cases.

Guest speaker Professor Zinaida Kuznetsova talked about her experiences of working with Crossref and the benefits of being a member. This was complimented by a talk by fellow guest speaker Maxim Mitrofanov from Crossref sponsoring organisation, NEICON. Maxim explained how NEICON works with Crossref, and provide services for the smaller members they support. Maxim is also one of our Crossref Ambassadors - and he will be running more Russian webinars on our services in the near future, so look out for those listed on our webinar page!

We’d like to say a big thank you to the team at Ulyanovsk State Pedagogical University for their support and help with the event. Also thanks to our fantastic interpreters who helped us immensely by relaying the information to the audience in Russian, as well as helping to translate and answer questions.

Germany

Najko Jahn from Göttingen State and University Library talks about how he uses @CrossrefOrg metadata in his work #CRLIVEGermany pic.twitter.com/Y89ZkBMoSh
— Vanessa Fairhurst (@NessaFairhurst) June 27, 2018

One week later and we were in Hannover, Germany. Crossref’s Laura Wilkinson, Joe Wass and Jennifer Kemp joined me for this event, which was held in collaboration with the German National Library of Science and Technology (Technische Informationsbibliothek - TIB at their impressive venue in on the 27th June.

The day focused on all things metadata - how it can be used and why good metadata is important. This included taking a look at our new Participation Reports tool and a fascinating talk from guest speaker Najko Jahn from Göttingen State and University Library on the benefits of using Crossref metadata for libraries and scientists.

Datacite’s Britta Dreyer also spoke about how DataCite and Crossref support research data sharing, before Joe Wass and I presented updates to the collaborative Org ID project and Event Data service. The day concluded with us sharing more ways to participate in Crossref and other community initiatives.

Questions? Вопросов? Fragen?

Over the course of these events we were asked many questions—and here are some of the more interesting/common ones posed to the team:

Q. Do I have to join Crossref directly, or can I join as part of a group of smaller organisations?
A. You don’t have to be a direct member, you can join via a Sponsor. See our sponsors page for a list of Sponsors in your area, and for more information on becoming a Sponsor.

Q. Can I link translations of works together?
A. Yes, a journal article published in two languages can each be assigned its own DOI, and then linked in the metadata using the relationship type TranslationOf from our schema.

Q. Does the web deposit form support depositing abstracts and references?
A. No, it doesn’t. However, our new Metadata Manager tool does and if you are in interested in trying it out in beta, let us know.

Q. Can I share your new Participation Report tool with my colleagues?
A. Yes you can! It’s open and available for use, just come along and search for a member.

Q. Can I also register book chapters, dissertations and other record types under the same prefix?
A. Yes you can. You can register any of the different resource types we support under one prefix.

Q. Will you be doing more events in this region in future?
A. We hope so, and we are always happy to hear from those who wish to collaborate on future events, so just contact us to get involved.

Status, I am new

Isaac Farley — Mon, 02 Jul 2018 00:00:00 +0000

Hi, I’m Isaac. I’m new here. What better way to get to know me than through a blog post? Well, maybe a cocktail party, but this will have to do. In addition to giving you some details about myself in this post, I’ll be introducing our status page, too.

A little about me

In mid-April, I began as the new Support Manager. My goal is to fill the very large shoes left by Patricia Feeney moving into the Head of Metadata role. I know Patricia knows Crossref and the rich community of members (and metadata!) inside and out. I’ll get there too. For now, I have immersed myself in tackling as many of your support questions as possible, so I may have already met some of you on a support ticket. If so, thanks for your patience; you likely have already taught me a thing or two!

Isaac, on the lookout to provide you excellent support

I came to this position from one of our members – the Society of Exploration Geophysicists, where I served as the Digital Publications Manager for the last five years. Like many of you, I was always impressed, intrigued, and excited by the work underway at Crossref and wanted to be a part of the team. So, here I am, very much looking forward to the challenge ahead.

I work remotely from Tulsa, Oklahoma, where I live with my wife and two daughters. Tulsa doesn’t have as many members as D.C., London, or Jakarta, but I hope to meet some of you during outreach trips, LIVE events, online in a webinar, or in our support community.

One of the things that attracts me to being a part of this community are our truths. As a quick reminder, the truths are:

Come one, come all
One member, one vote
Smart alone, brilliant together
Love metadata, love technology
What you see, what you get
Here today, here tomorrow

I am drawn to forward-thinking, action-oriented communities that value collaboration and openness. These truths, and the ten weeks I have been at Crossref, have confirmed that this is one of those communities. As your new support manager, I want to emphasize our commitment to transparency: Ask me anything; I’ll tell you what I know. In that spirit, I have the privilege of introducing our new status page—a key piece in furthering our own transparency and openness.

status.crossref.org

Our new status page provides critical, real-time information about our services—it helps us tell our overall story. If you are looking for metrics on the performance of our APIs, websites, the deposit system, or new beta services, bookmark this page. The system metrics provide daily, weekly, and monthly overviews of each of our services’ response time (in milliseconds) and uptime, or percentage of time that service has been operational during your selected time span (daily, weekly, or monthly).

From this page, we’ll announce planned maintenance and keep you regularly updated when we have an incident. And, we’ll provide regular status updates for these incidents when in progress, updated, and completed.

Our new status page – status.crossref.org

I encourage you to subscribe to the updates from the top-right corner of the page. While we’ll update this page with any service-related outages, subscribing for notifications will allow you to stay current on the latest. We’ll describe maintenance and incidents clearly, simply, and timely when we have them. And, if we don’t, call us on it.

If you have questions about the performance of our services, the status page is a great starting place. If you still have questions, ask us, we’ll tell you what we know.

Using the Crossref REST API. Part 9 (with Dimensions)

Christine Cormack Wood — Wed, 27 Jun 2018 00:00:00 +0000

Continuing our blog series highlighting the uses of Crossref metadata, we talked to the team behind new search and discovery tool Dimensions: Daniel Hook, Digital Science CEO; Christian Herzog, ÜberResearch CEO; and Simon Porter, Director of Innovation. They talk about the work they’re doing, the collaborative approach, and how Dimensions uses the Crossref REST API as part of our Metadata Plus service, to augment other data and their workflow.

Introducing Dimensions

Dimensions is a next-generation approach to discovering, connecting with and contextualising research. Modern academics need data about the research ecosystem in which they exist as much as the administrators who develop institutional research strategies. All academics are now required to think long-range about their research projects, contextualise their research, and demonstrate the impact of their program. Additionally, they need to find funding, ensure that students go on to good positions, and hire talented colleagues whose skills fit well with ongoing projects. Dimensions gives the first fully-linked view of publications, grants, patents and clinical trials in an analytically-centred user experience.

How is Crossref data used within Dimensions?

For an article to appear in Dimensions it must have a Crossref DOI, so it would not be possible to create Dimensions’ Publication index without Crossref’s data. Dimensions is built on several principles that we’ve talked about before. Here the most relevant of those principles are:

unique identifiers should underlie everything that we do;
data should not be inclusive and the tool should allow the user to select what they want to see;
data should be more available to our community;
data should be presented with as much contextual information as possible;
the community should have enough data available to be able to create and experiment with their own metrics and indicators.

In the context of these principles, Crossref makes a perfect starting place to create a tool like Dimensions. We use the Crossref data to know about our possible “universe” of articles. We then enhance the Crossref core with data from several different places: open access publications in the DOAJ, PubMed, BioArXiv, and through relationships with publishers. In all, 60 million of the 95 million articles in the Dimensions index have a full text version that we can text and data mine for additional information.

In Dimensions’ enhancement stage we can extract address information (where not included in the original Crossref record) and map it to GRID funding information and the list of funders in Crossref’s Funder Registry as well as to our database of grants in Dimensions.

How have you incorporated citation data?

Access to citations has historically been a thorny issue for citations databases. However, I4OC celebrated its first anniversary in April this year and this project has been a key driver in helping us to build Dimensions with the level of citation coverage that we managed –– it is a fantastic enabling initiative and should be warmly welcomed by the sector. Crossref is not the only source we were able to use to gather citation data; some text mining was needed to get a full graph. Dimensions goes beyond inter-article citations and includes links between patents and publications, links between clinical trials and publications, and Altmetric mentions of publications.

Is Dimensions openly available?

Given that there is so much open data in Dimensions, it was always our intention to give a free version to the community. If you visit http://app.dimensions.ai then you’ll be able to play with the system and use it for your research. While only the publications index is fully open, when you see a link to a grant, patent or clinical trial in an article detail page, you’ll be able to navigate to that record so that you can see the full context of the data.

Beyond the ability to link the publications, Dimensions also displays the CV information which the researcher made visible publicly.

Most recently, we’ve integrated ORCID into Dimensions. This means that you can push data from Dimensions into ORCID if you connect your ORCID account to your Dimensions account.

What are the future plans for Dimensions?

Dimensions is still moving quickly and adding more functionality. Our aim is to release more data facets very soon. We plan to add a Policy Document archive and a Research Data archive. We’ve already found some fascinating insights from joining the existing data together and these two new archives should add even more interesting data.

What else would Dimensions like to see in Crossref metadata?

Open access information is something that we work with Unpaywall to source for Dimensions right now. It would be great if Crossref and Unpaywall could work together to make this data higher quality and more ubiquitous.

Thank you Daniel, Christian and Simon.

If you would like to contribute a case study on the uses of Crossref Metadata APIs please contact the Community team.

Meet the members, Part 3 (with INASP)

Christine Cormack Wood — Wed, 20 Jun 2018 00:00:00 +0000

Next in our Meet the members blog series is INASP, who isn’t a direct member, but acts as a Sponsor for hundreds of members. Sioux Cumming, Programme Specialist at INASP tells us a bit about the work they’re doing, how they use Crossref and what the future plans for INASP are.

Can you tell us a little bit about INASP?

INASP is an international development organisation working with a global network of partners in Africa, Latin America and Asia. We have a vision of research and knowledge at the heart of development, so are working to support individuals and institutions to produce, share and use research and knowledge, which can transform lives.

Our work includes strengthening research communication, which we do via AuthorAID (supporting researchers, especially early-career researchers, in getting their research published); improving information access (supporting library consortia with access to international journals and other online resources); supporting evidence use in policy making; working with higher-education institutions to improve critical thinking skills; improving gender equity in research systems; and my area, which I’ll talk more about below, supporting academic publishing in the Global South.

INASP’s approaches are based on the core pillars of capacity development, convening, influencing and working in partnership. INASP promotes equity by actively addressing the needs of both men and women across all our work and addressing issues of power within the research and knowledge system.

What’s your role within INASP?

I’m a Programme Specialist and since I started at INASP 15 years ago I’ve been responsible for our academic publishing work. This work supports increased visibility, accessibility and quality of peer-reviewed journals published in developing countries so that the research outputs that are produced in these countries can be found, shared and used more effectively.

We recognize two big challenges for Southern journals in playing their part in global research systems. The first is awareness of Southern journals, many of which were until recently only available in print. Supporting editors and national organisations to put their journals online on central platforms (the Journals Online platforms) has helped increase their visibility.

We have also provided support to the local management teams in communicating about the platforms, the journals and the research they publish, and we recently published a Handbook for Journal Editors - www.inasp.info/editorshandbook. This is intended to be a free resource for editors worldwide that can be used as a stand-alone handbook or as an accompaniment to the journal quality online course that we are currently developing.

The second challenge is supporting publishing quality and enabling Southern journals to demonstrate their quality so they will be regarded as credible. In the early days, this, for me, was largely about providing training and mentoring for journal editors and Journals Online platform managers about standard publishing practices.

More recently, as local handover progressed, our role shifted towards helping journals to demonstrate their credibility. Last September INASP and African Journals Online launched our Journal Publishing Practices and Standards (JPPS) framework for assessing the quality of Southern publishing processes. This has been really well received by the international publishing sector and by the journal editors we work with.

Tell us a bit about who you support, and how you support them

We support others communicating their research and finding out about the research of others.

The five Journals Online platforms that were handed over to local management at the end of March collectively host 397 journals from Bangladesh, El Salvador, Honduras, Mongolia, Nepal, Nicaragua and Sri Lanka.

These platforms help the research from these countries to become even more integrated in the global research community. Some fascinating and valuable research is published in the journals on these platforms. You can see some examples of this research in this article about a small piece of work we did with these platforms to commission and disseminate press releases of some of the research.

What’s your participation level with Crossref?

INASP has been a Crossref Sponsor for the Journals Online platforms since 2008 and all articles on the sites have DOIs assigned to them (approximately 50,000 articles). All the in-country training sessions for journal editors publishing via the JOL platforms have included sessions to explain how DOIs work and why they are important. We have also trained editors on how to find and include the DOIs for the references of their articles. More recently, in 2015, we provided access to the Crossref Similarity Check service to editors, which enabled them to improve the quality of their submissions by identifying instances of plagiarism before the articles were published.

What trends are you seeing in your part of the scholarly communications community?

Demonstrating credibility of journals is an important part of journal publishing today. There are so many journals worldwide and it is a tough challenge for authors and readers to navigate this sector – a challenge that we often see through the discussions in our AuthorAID network. But it is important that researchers don’t simply turn to the handful of well-known publishers in the Global North that have dominated scholarly discourse to date.

To really tackle global issues and increase equality in global research we need to work towards levelling the playing field and including all voices – and this challenge needs to be embraced across the global research and knowledge system. We have seen encouraging signs over the past couple of years of magazines, blogs, conference organizers and industry groups in the Global North approaching us to help bring in more global perspectives to scholarly discussions. However, there is plenty more to be done and we are particularly focusing on equity in our new areas of work.

Collaboration with Crossref over the past few years has been one of a number of ways that we have been able to connect small, scholar-led titles in the Global South with the latest global standards and approaches in scholarly publishing. This is important as it all helps to level the playing field.

Including DOIs in papers is one of the criteria for being awarded a JPPS star and thus the journals are incentivized to understand and use them more.

What are INASP’s plans for the future?

INASP has recently completed a major five-year programme of work with a significant focus on strengthening organisations in the countries we have been working in and handing over responsibility for managing things like the Journals Online platforms. We are now in a new phase of work, building on what has gone before but with a particular emphasis on improving equity both within and between research systems.

Many challenges remain – the global research system still tends to be biased towards the Global North. From an academic publishing perspective this is apparent both in terms of awareness of journals and also in terms of impressions of credibility. JPPS is intended to tackle the latter challenge but it is still early days – we only announced the first badges awarded a few months ago. Over the next few years we will be building on and strengthening this work and ensuring that it is an important part of the processes for journal editors and for authors and readers.

Thank you Sioux for your participation in our Meet the members series. If your organisation would like to feature in this series, please get in touch.

Preprints growth rate ten times higher than journal articles

Jennifer Lin — Thu, 31 May 2018 00:00:00 +0000

The Crossref graph of the research enterprise is growing at an impressive rate of 2.5 million records a month - scholarly communications of all stripes and sizes. Preprints are one of the fastest growing types of content. While preprints may not be new, the growth may well be: ~30% for the past 2 years (compared to article growth of 2-3% for the same period). We began supporting preprints in November 2016 at the behest of our members. When members register them, we ensure that: links to these publications persist over time; they are connected to the full history of the shared research results; and the citation record is clear and up-to-date.

Summary

As of May 24, 2018 we have 44,388 works (see API query https://api.crossref.org/types/posted-content/works with a json viewer) registered as posted content. Today that number is over 150k. Preprints are part of this record type category, which is meant to house scholarly outputs that have been posted online and intended for publication in the future.

For a more granular view, see the monthly stats captured by Jordan Anaya in PrePubMed. This data is based on a slightly different set of preprint repositories, though both show the same trends.

The figure below shows the preprints registered with Crossref, broken down by repository.

We eagerly await our newest preprints member, Center for Open Science, who will soon be registering the preprints from their 18 community archives with us (~9k preprints total to date).

Metadata coverage

We accept a range of metadata for the preprints registered with us, including:

Repository name & hosting platform
Contributor names & ORCID iDs
Title
Dates (posted, accepted)
License
Funding
Abstract
Relations
References

As with all resource/record types, certain metadata is required, though others are optional. We encourage full coverage of metadata in the record where applicable and possible. So what are publishers including in their posted content records? The summary view is as follows:

License: 9926 (json), 22% (PeerJ Preprints, ChemRxiv)
Funder: 0 (json), 0%
ORCID: 19309 (json), 44% (bioRxiv, PeerJ Preprints, Preprints.org, ChemRxiv)
Abstracts: 35874 (json), 81% (bioRxiv, PeerJ Preprints, ChemRxiv)
References: 1921 (json):, 4% (JMIR)

Compared to all the published content registered with us over time, preprints have above average coverage of ORCID iDs deposited and show well above average with abstract metadata. However, they are significantly lagging behind with depositing references, license, and funding metadata. (See a summary of the full corpus stats taken two months ago in the blog post, A Lustrum over the Weekend).

Preprint-article pairs

Members registering preprints have an obligation to update the metadata record when a journal article is subsequently published, to clearly identify this work. This pairing is passed on to our metadata users: indexing platforms; recommendations engines; platforms; tools, etc. which pull from our APIs. (The preprint landing page also must link to the article.) As such, the preprint-article pairings are amassing as each week passes. We currently have a total of 12983 (json) preprints connected to articles. The figure below provides the counts based on repository.

Citations

We can see from preprint Cited-by counts that researchers are indeed citing preprints in their articles. This practice is an extension of the common citation behavior to provide evidence for and credit to previous work, a natural consequence of work shared with their peers. The most highly cited preprint papers (json) as of May 24, 2018 are as follows. In some cases, a subsequent paper was published from the results shared in the preprint. These have also accrued citations in their own right and these are also indicated in the table below.

No.	Cited-by	Preprint DOI	Preprint title	Date	Subsequent journal article	Citations of journal article
1	Cited-by 72	https://doi.org/10.1101/005165	qqman: an R package for visualizing GWAS results using Q-Q and manhattan plots	May 14, 2014.	n/a	n/a
2	Cited-by 63	https://doi.org/10.1101/002824	HTSeq - A Python framework to work with high-throughput sequencing data	August 19, 2014	Bioinformatics, https://doi.org/10.1093/bioinformatics/btu638	2372
3	Cited-by 43	https://doi.org/10.1101/030338	Analysis of protein-coding genetic variation in 60,706 humans	May 10, 2016	Nature, https://doi.org/10.1038/nature19057	1598
4	Cited-by 38	https://doi.org/10.1101/002832	Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2	November 17, 2014	Genome Biology, https://doi.org/10.1186/s13059-014-0550-8	3284
5	Cited-by 32	https://doi.org/10.1101/021592	Salmon provides accurate, fast, and bias-aware transcript expression estimates using dual-phase inference	August 30, 2016	Nature Methods, https://doi.org/10.1038/nmeth.4197	112
6	Cited-by 22	https://doi.org/10.1101/012401	DensiTree 2: Seeing Trees Through the Forest	December 8, 2014	n/a	n/a
7	Cited-by 21	https://doi.org/10.1101/011650	FusionCatcher - a tool for finding somatic fusion genes in paired-end RNA-sequencing data	November 19, 2014	n/a	n/a
8	Cited-by 19	https://doi.org/10.1101/048991	Analysis of shared heritability in common disorders of the brain	September 6, 2017	n/a	n/a
9	Cited-by 18	https://doi.org/10.1101/006395	Error correction and assembly complexity of single molecule sequencing reads	June 18, 2014	n/a	n/a
10	Cited-by 18	https://doi.org/10.1101/032839	Spread of the pandemic Zika virus lineage is associated with NS1 codon usage adaptation in humans	November 25, 2015	n/a	n/a

The relationship between preprints and the proceeding publication is an interesting area that is not yet well understood. We invite the community to analyze the Crossref metadata using the REST API in concert with other datasets. For example, the citation lifecycle for these two research products has been one of speculation so far without a systematic investigation into patterns and timeframes of preprint citations and those of its succeeding article across the corpus. Here, submission dates would be critical data to this research question as publication windows vary significantly by publisher and by paper.

Linking references is different from registering references

Anna Tolwinska — Wed, 30 May 2018 00:00:00 +0000

From time to time we get questions from members asking what the difference is between reference linking and registering references as part the Content Registration process.

Here’s the distinction:

Linking out to other articles from your reference lists is a key part of being a Crossref members - it’s an obligation in the membership agreement and it levels the playing field when all members link their references to one another.

Registering references when you register your content is completely different. It’s enriching the metadata record that describes your content, and it allows Crossref and others—including non-members—to use them.

Reference Linking

A research article usually includes a reference list of citations to other works that helped inform it. The original function of Crossref was to provide a central service for publishers that enabled them to link to each others’ content from these reference lists—using a DOI as a persistent link. This meant that members of all sizes and in all disciplines could easily link to one another without having to sign hundreds of bilateral agreements.

We made Reference Linking obligatory for Crossref members because it’s fundamental to making content discoverable, and because when everyone links their references, research travels further and benefits everyone.

Registering references

Every single day hundreds of members register and update their metadata with us—and every single day hundreds of organisations search for, extract and use it. To make sure your content is discovered in this process, it’s important to make the metadata you register with us as rich as possible. Rich metadata includes information such as journal title, article author, publication date, page numbers, ISSN, abstracts, ORCID iDs, funding information, clinical trials numbers, license information, and of course—references.

Additionally, registering references is ~~a prerequisite~~ recommended for participating in our Cited-by service—which provides citation counts and lists, and ultimately makes your content more discoverable. [EDIT 7th February 2024 - it is no longer required but highly recommended.]

We know it’s not easy for smaller publishers to deposit references. Read more on how to here. Our upcoming Metadata Manager tool will allow you to register your references at the same time as the rest of your content. This service is currently in development but let us know if you want to try it out. [EDIT 7th February 2024 - Metadata Manager has been deprecated. More info about it here.]

Reference Linking

Reference Linking means adding Crossref DOI links to the reference list for journal articles on your article pages as per this example: https://doi.org/10.1088/1367-2630/1/1/006.

How it works

First retrieve DOIs for all available references either through our human or machine interfaces. Then make sure you use the DOI link in your references and on your article landing page using the Crossref DOI display guidelines.

Why it’s useful

Reference Linking:

Enables you to link to more than 10,000 publishers without having to sign multiple agreements
Helps with discoverability, because DOIs don’t break if implemented correctly
Displays your DOIs as URLs so that anyone can copy and share them
Makes your content more useful to readers
Drives traffic to your website from other publishers.

Is it obligatory?

Yes, within a short time after becoming a member you should be including references.

Registering References

Registering references means submitting them as part of your Crossref metadata deposit as per this example: https://www.crossref.org/xml-samples/article_with_references.xml.

How it works

Whenever you register content with us, make sure you include your references in the submission. You can also add references to your existing content via a metadata redeposit, or our resource-only deposit, or our Simple Text Query form.

Why it’s useful

References registered as part of your metadata:

Make your content more discoverable
Make your content richer and more useful
Are required to participate in our Cited-by service (this service shows what articles cite your article)
Enables discovery of research
Enables evaluation of research
Highlights your contents’ provenance
Helps with citation counts.

Is it obligatory?

No, it’s optional, but strongly encouraged. It is ~~required~~ recommended if you are participating in our Cited-by service. [EDIT 7th February 2024 - it is no longer required but highly recommended].

If you have any questions about reference linking or registering your references please get in touch.

SSP roadtrip for the Crossref team

Amanda Bartell — Wed, 30 May 2018 00:00:00 +0000

What do you think of when you think of Chicago? Deep dish pizza? Art Deco architecture?

Well for one week only this year you can add scholarly publishing to the list as the #SSP2018 Conference comes to town. Some Crossref people are excited to be heading out for the conference, and we’re looking forward to meeting as many of our members as possible.

Come along to stand 212A and talk to Anna Tolwinska about Participation Reports. Although this new tool is still in beta, she’s giving SSP attendees a sneak peek and the chance to get an early look at whether they (and over 10 000 other members) are registering the ten key elements that add context and richness to the basic required metadata. You’ll get real insight into what metadata you’re registering, even if this work is done by a third party or other department.

Thinking about registering preprints or including data citations? Want to find out more about our forthcoming Event Data service? Our product director Jennifer Lin will be able to give you the ins and outs of all our latest services so do keep an eye out for her at the conference.

Speaking of third parties, I’ll will be popping along to the “Thinking the Unthinkable, or How to Prepare for a Platform Migration” pre meeting seminar on Wednesday with copies of our new Platform Migration Checklist and lots of hints and tips to help form a new platform migration guide which will help members have a smooth transition when thinking of moving providers.

Shayn Smulyan will be attending the ORCID breakfast meeting on Thursday morning, so come and say hello if you have any questions about how ORCID and Crossref work together. Shayn is one of our support specialists, so he’ll be able to help you with any other technical queries you may have.

Our tech director Chuck Koscher will be keen to hone in on members’ advanced questions about Content Registration, citation matching, and any and all schema deets. So seek him out if you have deep technical questions.

Want to find out more about Metadata 2020, the new campaign to improve metadata for research? Rosa Morais Clark will be able to give you the lowdown, and even better - she has stickers!

And don’t feel left out if you aren’t a member but work closely with Crossref. Jennifer Kemp will be on hand to answer all your metadata use and reuse questions, she’ll be looking forward to chatting with all kinds of service providers, platforms, and tools.

We’re looking forward to seeing you there!

How good is your metadata?

Kirsty Meddings — Thu, 26 Apr 2018 00:00:00 +0000

Exciting news! We are getting very close to the beta release of a new tool to publicly show metadata coverage. As members register their content with us they also add additional information which gives context for other members and for services that help e.g. discovery or analytics.

Richer metadata makes content useful. Participation reports will give—for the first time—a clear picture for anyone to see the metadata Crossref has. This is data that’s long been available via our Public REST API, now visualized.

Who are participation reports for? Everyone!

It’s an opportunity to evaluate and educate. See for yourself where the gaps are, and what our members could improve upon. Understand best practice through seeing what others are doing, and learn how to level-up.

Monitor what metadata is being registered, even if this work is done by a third party or another department. And see what other organisations in scholarly communications see when they use Crossref metadata in their research, tools, and services.

The beta release—expected after acceptance testing some time late May—will let anyone look up any of our 15,000+ members and see whether they are registering ten key elements that add context and richness to the basic required bibliographic metadata.

What do we mean by ‘richer metadata’?

The ten checks for Beta, will be:

References
~~Open references~~ [EDIT 6th June 2022 - all references are now open by default].
ORCID iDs
Funder IDs
Funding award numbers
Crossmark metadata
License information
Full text links
Similarity Check URLs
Abstracts

Each of these additional metadata elements helps increase discovery and wider and more varied use—and usefulness—of research outputs.

Why are we doing this and what do we mean by ‘participation’?

Over the years when we’ve talked with our members about their metadata, we learned that many just can’t be certain exactly how they’re performing. It could be that they’ve outsourced Content Registration to another service provider or larger publisher, or it could be they just weren’t previously aware they could collect and share authors’ ORCID iDs, Funder IDs, and so on. So our primary aim is to give our members the information they need in order to make a case for improving their metadata records. Each check will come with information about why it is important and guidance on how to improve. Additionally, with the growing use of Crossref as a central source of metadata for the research community, it’s in everyone’s interest to be as transparent as possible about what metadata we have - and encourage greater understanding of what’s possible.

Member ‘participation’ is an important concept. Crossref distinguishes itself from other DOI registration agencies by providing this richer infrastructure which allows for things like funding information, license information, links between data and preprints, and so on—all contributing to the research nexus for everyone’s benefit.

Membership of Crossref is not just about getting a persistent identifier for your content, it’s about placing your content in context by providing as much metadata as possible and looking after it long-term.

Here’s a sneak preview of what the report will look like:

So whether you’re a member who wants to run a “health check” on your own metadata, or a consumer of metadata interested in what’s available and from whom, watch this space for Participation Reports!

Would you like a heads-up on your report, pre-beta?

Beta will be released some time in May or June this year, following acceptance testing with members and others. Then we’re looking for about 20 members to have a half-hour phone call with a walk-through ‘health check’. Please contact Anna if you’d like to schedule one.

Redirecting redirection

Geoffrey Bilder — Tue, 24 Apr 2018 00:00:00 +0000

Crossref has decided to change the HTTP redirect code used by our DOIs from 303 back to the more commonly used 302. Our implementation of 303 redirects back in 2010 was based on recommended best practice for supporting linked data identifiers. Unfortunately, very few other parties have adopted this practice.

What’s more, because using a 303 redirect is still unusual, it tends to throw SEO tools into a tizzy- and we spend a lot of time fielding SEO questions from our members about our use of 303s.

TL;DR

At this point, we need to emphasise that we have never seen our use of 303s actually affect page rankings. But at the same time, use of 303 redirects has not had wider uptake. Maintaining this quixotic behaviour just isn’t worth the effort. We hope that, in the future, we can use other techniques (e.g. signposting & cite-as) to achieve some of the things that 303 was supposed to do.

Note that these changes will not affect users or machines using DOIs. The change should be entirely transparent.

Below we provide some background to our decision and after that we provide some detailed technical notes from Jonathan Rees and Henry Thompson who have been very kind in helping to provide Crossref technical guidance on how we can help DOIs best support linked open data and adhere to HTTP best practice.

Background

Back in 2010, Crossref, DataCite (and later, several other RAs) responded to concerns that DOIs were not “linked-data friendly.” There were three problems with DOIs at that time:

It was not clear that DOIs could be used and expressed as HTTP URIs.
There was no standard way to ask a DOI to return a machine-readable representation of the data.
It wasn’t always clear if the DOI resolved to “the thing” (e.g. an article) or “something about the thing” (e.g. a landing page).

On the advice of several people in the linked data community, we proposed some options for fixing this. And we finally settled on:

Recommending that Crossref DOIs be expressed and displayed as HTTP (now HTTPS) URIs. This made it clear that DOIs could be used with HTTP applications.
Enabling DOI registration agencies to support content negotiation. This allowed RAs to support providing machine-readable representations of the data associated with a DOI.
Changing the underlying redirect code from the normal 302 to 303. This was designed to clarify what, at the time, was true- that most DOIs resolved to a landing page, not the article itself.

By any practical measure, machine use of DOIs has exploded since we made these decisions back in 2010. Crossref’s APIs and content negotiation handle over 800 million requests for machine readable data a month. Our sibling organisation, DataCite, has also seen a huge growth in machine use of DOIs. Many applications, from bibliographic management tools, to authoring systems and CRIS systems, make use of machine actionable DOIs all the time. So clearly our work to promote DOIs as machine actionable identifiers is working, but we are certain that our current use of 303 redirects has nothing to do with this growth.

First of all, as we said, very few parties have actually subscribed to the notion of using 303s to help distinguish “the thing” from “something about the thing”.

Secondly, even if they did try to rely on 303s to make this distinction, they would quickly get confused because the DOI is so often just the first in a chain of redirects which do not implement the same semantic distinction. At this point we should be clear - Crossref thinks these kinds of long redirect chains are a bad idea for two main reasons:

They slow down resolution.
They increase the number of potential failure points between the DOI and the item it resolves to.

But we also cannot legislate them away. They exist. And in the real world you will find plenty of DOIs that do a 303 redirect to a system that, in turn, does a 302 redirect to a system that does a 301 redirect and…eventually ends up someplace returning a 200. You get the picture. How on earth is a machine supposed to interpret a 303->302->301->302 redirect chain?

Furthermore - nowadays, after following this chain of redirects, you will often find yourself on a “page” that is both a landing page and the article itself. Dynamic, one-page applications can simply morph the one into the other without the use of additional HTTP requests.

In other words, using 303s is not helping machines interpret what the DOI is pointing at. And yet, people seem to be making good use of machine actionable DOIs and they are not complaining much about it.

Personally, I’d might have just been happy to switch back to using 302s simply so that I could cut down on my conversations with SEO hacks. But that wouldn’t be a principled approach. In 2010 we spent a lot of time considering the initial switch to 303s- we needed to consult with the LOD community on a potential switch back to 302s. At the January 2018 PIDapalooza I had a chance to talk to Henry Thomson about the 302/303 dilemma we faced, and he along with Jonathan Rees very generously provided the following feedback.

Best practices for HTTP redirection by persistent identifier resolvers: 302 vs. 303

Jonathan Rees (MIT CSAIL, https://orcid.org/0000-0001-7694-8250)
Henry Thompson (University of Edinburgh, School of Informatics, https://orcid.org/0000-0001-5490-1347)

If one goes to the trouble to organize an identifier system, then the desire that such a system should last as long as possible leads one to aspirationally say it’s a persistent identifier (PID) system. The unwillingness of the major browser suppliers to implement new URI schemes for PIDs initially hindered their use on the Web and this in turn inhibited widespread adoption. More recently a number of PID approaches have enjoyed very rapid growth as a result of a compromise: these PIDs participate in the World Wide Web by defining simple conversion rules mapping identifiers to actionable (‘http:’ and/or ‘https:’) forms and providing resolution servers that redirect requests for such forms to the appropriate destination.This approach has been widely adopted and is very successful, because it is so useful. An identifier’s actionable form leads, via the HTTP protocol and one or more redirections, to a web page that bears on the ground identity of the associated entity – or perhaps even directly to the entity itself, if the system is one for document entities that are naturally provided as web pages. The nature of the retrieved web page varies from one system to the next.

A confusion arose, however, over claims in various technical specifications (URIs, HTTP, Web Architecture) that the normal case is for the protocol to yield a “representation” of the “resource” “identified” by the URI. None of these terms is adequately defined by the specifications, and initially the language was not taken as normative. Those deploying identifier systems took the HTTP “resource” to be the entity associated with an identifier, and understood the “resource” as being “identified” by the URI, but it was never clear what was, or wasn’t, a “representation” of a given entity/resource: a description of the resource, the resource itself, a version of the resource, instructions on how to find the resource, etc. Sixteen years ago, in an attempt to clarify the intent of this part of the theory of URIs, and to allow applications to usefully and uniformly exploit the idea that an HTTP 200 response must deliver a “representation” of the “resource”, Tim Berners-Lee asked the W3C Technical Architecture Group to consider what came to be known as the httpRange-14 issue. It’s now 13 years after the TAG gave advice which almost no one was happy with, and 5 years after work on issue httpRedirections-57 (which superseded httpRange-14) ground to a halt. There’s still no consensus on whether it’s OK to return landing pages with a 200 status in response to requests for pictures or publications, but the Web seems to be working nonetheless, and no one seems to be bothered much anymore.

The provision of HTTP-based resolution services has stimulated widespread support for the use of identifier systems with Web resolution, particularly in the scholarly journal publication context. Those setting up HTTP resolvers responsible for identifier systems must decide which HTTP response code should be used. The TAG’s advice sows doubt on the use of the 200 response code when the response would have been a landing page, and many resolvers avoid 200 regardless and use redirection for administrative purposes, for example

‘https://dx.doi.org/10.1109/5.771073’ to

‘http://ieeexplore.ieee.org/document/771073/?reload=true’ for the DOI

‘10.1109/5.771073’, or ‘https://identifiers.org/uniprot/A0A022YWF9’ to

‘http://www.uniprot.org/uniprot/A0A022YWF9’ for the Uniprot identifier

‘A0A022YWF9’.

So the response should be a redirection, but what kind, 301, 302, or 303? (Or 307, which is almost the same as 302.) A 301 redirect seems to say that the URI is not persistent (since its target is deemed “more persistent”). A 302 redirect seems to say that the response could have come via a 200, and so suffers the same fate as 200. That leaves 303, as hinted at in the TAG’s advice. This idea got some traction: Ten years ago a Semantic Web interest group promoted the TAG’s advice in a published note, and seven years ago one of us wrote a blog post giving the same advice for resolvers for PIDs in publishing.

However, not only is there neither consensus nor general utility around this strict understanding of the use of the various response codes – that is, that resolution to a landing page is inconsistent with a 200 (and a posteriori therefore with a 302) – but also the range of usage patterns for redirection of HTTP requests has grown and ramified over time as the Web has grown and become more complex. It’s on the face of it unlikely that a mere three response codes can capture all the resulting complexity or cover the space of outcomes (in terms of e.g. what ends up in the browser address bar or what search engines index a page under) that a page owner might like to signal.

We find in practice that some PID redirections are ending up (usually after further publisher-local redirects) at the “identified” document, some at landing pages, and some at one or the other depending on the requesting site, for example in the case of paywalled material.

In the absence of a rethinking of the whole 3xx space, it seems to us that only the 301 vs. 302 distinct ion (roughly, 301 = permanent = please fix the link, and 302 = temporary = don’t change the link) is well understood and more or less consistently treated, whereas for 303, web servers are not very consistent and both search engine and citation crawler behaviours are at best inconsistent and at worst downright unhelpful.

So, we believe it is in both users’ and publishers’ interests for resolvers of actionable-form PIDs to use 302 redirects, not 303.

If we want to help machines better understand the resource that a DOI points at, we have to explore using more nuanced mechanisms.

Just using 302 for the first redirect doesn’t do everything necessary to effectively support the emerging PID+redirection architecture. It’s at the end of the redirect chains that we need more: a standardised way to find the PID back at the start of the chain. The ‘cite-as’ proposal does exactly this, and we hope it’s quickly approved and widely adopted. Once that happens a proposal for augmenting browser (and API) behaviour to prefer, or at least offer, the ‘cite-as’ link for bookmarking and copying will be needed.

Using the Crossref REST API. Part 8 (with Researchfish)

Christine Cormack Wood — Mon, 23 Apr 2018 00:00:00 +0000

Continuing our blog series highlighting the uses of Crossref metadata, we talked to Gavin Reddick, Chief Analyst at Researchfish about the work they’re doing, and how they’re using our REST API as part of their workflow.

Introducing Researchfish

Researchfish is the world’s leading platform for the reporting of the outputs, outcomes and impacts of funded research. It is used by over 100 funding organisations in Europe, North America and Australasia and currently tracks around €50 billion of funding, across 125,000 grants. Researchers have reported around 2.5 million attributed outcomes in Researchfish and roughly half of these are publications with the other half being collaborations, further funding, data sets, policy influences, engagement activities etc.

Funders use Researchfish to ask grantees to report on the outcomes of their grant and Researchfish makes it easy for researchers to do this in a structured way. Researchfish seeks to improve the quality and robustness of the evidence base available for evaluation. It works with funders, research organisations and researchers to present, explain and evaluate the impact of research across all disciplines and a wide range of output types.

How is the Crossref REST API used in Researchfish?

Search
As publications are a major output of research it is important to make the reporting of those publications be as easy as possible and quality of the information on those publications as high as possible. Researchfish integrates with a number of publication APIs, including Crossref, which enables users to enter a number of DOIs or search by author, title, etc. to find their publication.
Direct Harvest
Researchfish uses funding acknowledgements in the Crossref metadata to add publications to researchers’ portfolios and report the publications as arising from the grant. If the acknowledgement exists it’s important to use it instead of asking researchers to report the same thing twice.
Interoperability
Research organisations can upload publications to Researchfish on behalf of researchers, re-using information from their local systems. We use the Crossref REST API to validate the data provided by universities before uploading.
Metadata Enrichment – Open Access
We use the license and embargo period information in the Crossref metadata to help understand the open access status of publications and whether they meet any policy requirements, without researchers having to take any steps to report in this complex area.
Metadata Enrichment – Normalisation/deduplication
As Researchfish allows users to add information from lots of different sources it is very important to normalise the data and prevent the same publication being reported multiple times in different ways. We use the Crossref REST API as part of this process.

What are the future plans for Researchfish?

We are looking to expand the range of integrations to support non-publication outputs and allow some of the same functionality that we have built for publications. We already have integrations to support the reporting of patents, collaborations, further funding and next destinations but are looking to enhance these, along with expanding links to data sets, clinical trials, software and spin out companies.

What else would Researchfish like to see in Crossref?

Crossref is an excellent resource and most of our wish list would be to see more uptake of existing fields e.g. retractions and the ability to use them more flexibly in the REST API. We would also like to see a little more consistency in some of the metadata – publication type is the area that seems to cause the most confusion, particularly around conference proceedings and clinical trials.

Thank you Researchfish! If you would like to contribute a case study on the uses of Crossref Metadata APIs please contact the Community team.

PIDs for conferences - your comments are welcome!

Aliaksandr Birukou — Thu, 19 Apr 2018 00:00:00 +0000

Aliaksandr Birukou is the Executive Editor for Computer Science at Springer Nature and is chair of the Group that has been working to establish a persistent identifier system and registry for scholarly conferences. Here Alex provides some background to the work and asks for input from the community:

Roughly one year ago, Crossref and DataCite started a working group on conference and project identifiers. With this blog post, we would like to share the specification of conference metadata and Crossmark for proceedings and are inviting the broader community to comment.

Why are conferences important?

One common misbelief is that most published research appears in journals. However, next to new ways of communication research results (blogs, presentations,…) and journals there are also other publication options, like books, very important in humanities, or conference proceedings, which are very important in computer science and a couple of related disciplines. Conference proceedings are collections of journal-like papers, often undergoing a more competitive peer review process than in journals. For instance, looking at original research in computer science in Scopus published in CS in 2012-2016, 63% of articles appeared in proceedings, while only 37% were published in journals. DBLP, one of the most important indexing services in CS, lists more than two million conference papers organized in ~5,400 conference series.

So, while it is true that CS has a significant share of conference proceedings, conferences are also relevant in many other disciplines which do not publish formal proceedings. For instance, inSPIRE contains ~23,000 conferences in high-energy physics, the American Society of Mechanical Engineers (ASME) publishes roughly 100 proceedings volumes annually.

Why do we need an open persistent ID for a conference or a conference series?

With publishers, learned societies, indexing services, libraries, conference management systems, research evaluation and funding agencies using conferences directly or indirectly in their daily work, a common vocabulary would simplify data processing, reporting and minimize errors. Right now, a publisher assigns a unique conference ID to the conference to be published, then an indexing service does it, then it is assigned in a library. Wouldn’t it be easier to do this at the very beginning of the process, when the conference planning starts, and keep the same identifier through the whole conference lifecycle?

The joint Crossref and DataCite group on conference and project identifiers has discussed this topic at half a dozen calls and various PID community meetings (PIDapalooza, FORCE conferences, AAHEP Information Provider Summit). The result of those discussions is a draft of the specification of conference metadata and Crossmark for proceedings.

The document first defines the concepts of a conference, conference series, joint and co-located conferences. It then introduces the information we want to store about those entities, e.g., the ID, name, acronym, other IDs, URL and the maintainer of the conference series, or the ID, conf series ID, number, dates, location, and URL for conferences. Such metadata can be submitted to Crossref and DataCite by conference organizers or publishers on their behalf and linked to the existing proceedings metadata, where appropriate. It can be then used for linking research outputs from a conference (beyond formal proceedings), recognizing reviewers via services such as ORCID and Publons, computing metrics of a conference series, conference disambiguation in indexing services and ratings (CORE, QUALIS, CCF), and so on.

The second part of the document introduces Crossmark for conference proceedings. Its goal is to structure and preserve the information about the peer review process of a conference as declared by the general or program chairs. Depending on how much information is available from the conference organizers, one can use the basic or extended versions of Crossmark.

In order to comment, please open the specification and leave comments using “comment” feature of Google Docs. The draft remains open for comments till the 31st of May 2018.

Next steps

After hearing from YOU, we will update the document to reflect the community comments. In parallel, we start a subgroup discussing the governance models, looking into whether we need a new membership category at Crossref, what fees should be covered, etc.

Do you want to be on our Board?

Lisa Hart Martin — Wed, 18 Apr 2018 00:00:00 +0000

Do you want to effect change for the scholarly community?

The Crossref Nominating Committee is inviting expressions of interest to serve on the Board as it begins its consideration of a slate for the November 2018 election.

The key responsibilities of the Board are:

Setting the strategic direction for the organisation;
Providing financial oversight; and
Approving new policies and services.

Some of the decisions the board has made in recent years include:

Introduction of the Metadata APIs Plus service (to provide a paid-for premium service for machine access to metadata);
Updating the policy on open references (to increase links so that more readers can access content);
Establishing the OI Project (to create a persistent organisation Identifier);
Inclusion of preprints in the Crossref metadata; and
Approval to develop Event Data (which will track online activity from multiple sources).

What is expected of a Crossref Board member?

Board members should be able to attend all board meetings, which occur three times a year in different parts of the world. If you are unable to attend in person you must be able to attend via telephone.

Board members must:

be familiar with the three key responsibilities listed above,
actively participate and contribute towards discussions, and
read the board documents and materials provided, prior to attending meetings.

How to submit an expression of interest to serve on the Board

We are seeking people who know about scholarly communications and would like to be part of our future. If you have a vision for the international Crossref community, we are interested in hearing from you.

If you are a Crossref member, are eligible to vote, and would like to be considered, you should complete and submit the expression of interest form with both your organisation’s statement and your personal statement before 18 May 2018.

It is important to note it is your organisation who is the Crossref member—and therefore the seat will belong to your organisation.

About the election and our Board

We have a principle of “one member, one vote”; our board comprises a cross-section of members and it doesn’t matter how big or small you are, every member gets a single vote. Board terms are three years, and one third of the Board is eligible for election every year. There are five seats up for election in 2018.

The board meets in a variety of international locations in March, July, and November each year. View a list of the current Crossref Board members and a history of the decisions they’ve made (motions).

The election opens online in September 2018 and voting is done by proxy online, or in person, at the annual business meeting during ‘Crossref LIVE18’ on 13th November 2018 in Toronto, Canada. Election materials and instructions for voting will be available to all Crossref members online in September 2018.

The role of the Nominating Committee

The Nominating Committee meets to discuss change, process, criteria, and potential candidates, ensuring a fair representation of membership. The Nominating Committee is charged with selecting a slate of candidates for election from those who have expressed an interest.

The selection of the slate (which is likely to exceed the number of open seats) is based on the quality of the expressions of interest and maintaining the balance and diversity of the board—especially in areas of organisational size, gender, geography and sector.

The Committee is made up of three board members not up for election, and two non-board members. The current Nominating Committee members are:

Mark Patterson, eLife (Chair);
Chris Shillum, Elsevier;
Amy Brand, MIT Press;
Vincent Cassidy, The Institution of Engineering & Technology (IET); and
Claire Moulton, The Company of Biologists.

Our board needs to be stay truly representative of Crossref’s global and diverse membership of organisations who publish. Please submit your statements of interest or reply to me with any questions to me at lhart@crossref.org.

Hear this, real insight into the inner workings of Crossref

Joe Wass — Sun, 01 Apr 2018 00:00:00 +0000

You want to hear more from us. We hear you.

We’ve spent the past year building Crossref Event Data, and hope to launch very soon. Building a new piece of infrastructure from scratch has been an exciting project, and we’ve taken the opportunity to incorporate as much feedback from the community as possible. We’d like to take a moment to share some of the suggestions we had, and how we’ve acted on them.

We asked a focus group “What one thing would you change?”. In hindsight, we could have done a better job with the question. We did get some enlightening answers but—for legal and practical reasons—we are unable to end either world hunger or global conflict, or do any of the other things we were invited to do. So we went back to our focus group and asked “What one thing would you change about Crossref?”.

The answers were illuminating. Some of you wanted mundane things like more data dumps. A disappointing number of people wanted us to put the capital ‘R’ back in our name. But two things we heard consistently, loud and clear, were:

“I want to hear more from Crossref”
“I want to know more about what’s going on inside Crossref”

One respondent said:

I like the newsletters, and the Twitter visuals are nice enough, but I want to hear, you know, more from them.

Another:

Crossref is your typical quiet DOI Registration Agency. They make a big thing about being the background infrastructure you don’t notice. But infrastructure doesn’t have to be quiet. I live next to the M25, and I can tell you, that’s the sound of success. I mean, it’s loud.

One final quote which clinched it for us:

The outreach team is doing a great job with their multilingual videos. But you can never cover every world language. In today’s connected world, you should be thinking about the universal language.

She clarified:

No, I don’t mean XML.

We took this advice to heart. When we were building Crossref Event Data, we baked these features right in. Now you can hear what’s going on inside Crossref, any time, day or night.

Introducing the Crossref Thing Action Service!

Turn up your speakers (about half-way, it would be foolhardy to turn them too high) and visit:

live.eventdata.crossref.org/thing-action-service.html

It’s optimized for Google Chrome, but we’ve tested it in Firefox and Safari.

The Thing Action Service shows you, in excruciating sonorous detail, every single action that happens inside the Crossref Event Data system. Every time we receive live data from Twitter or Wikipedia. Every time we check a DOI. Every time we check an RSS feed. Every time we find a link to our Registered Content on the web.

In a pioneering move within the scholarly publishing space, you can hear the data as it’s being processed, live. Furthermore, we think we are the first DOI Registration Agency to offer our services in stereo.

John Chodacki, Professional Working Group Chair, said:

We welcome this innovation. From my experience Chairing, well, everything, I’m certain that hearing-impaired users will like it especially.

So sit back, put the Thing Action Service on the speakers, and relax. You may find it difficult at first, but as you let the sound waves wash over you, think of all that data in flight. That beep could be someone criticizing the article you wrote on Twitter. But don’t worry, the next one might be someone defending it.

Think of it as musique concrète. That’s the Art of Persistence.

Hello, meet Event Data Version 1, and new Product Manager

Christine Buske — Thu, 29 Mar 2018 00:00:00 +0000

I joined Crossref only a few weeks ago, and have happily thrown myself into the world of Event Data as the service’s new product manager. In my first week, a lot of time was spent discussing the ins and outs of Event Data. This learning process made me very much feel like you might when you’ve just bought a house, and you’re studying the blueprints while also planning the house-warming party.

If Event Data is like a house, it’s been built and we’ve recently been putting on a last coat of paint. We’re very happy to announce version 1 of the API today. This is bringing us closer to the launch (house warming party), which will officially present Event Data to the world. Further to that analogy, while I bought into the house, I wasn’t around to see it being built. That’s both incredibly exciting and a little daunting.

Version 1 contains fixes for some challenges we came up against. Like scalability, data modeling for Wikipedia, and polishing. Version 1 is a new release of the data, but it is the same data set you already know and love. It should solve some of the recent stability issues, for which we apologize.

Moving forward, we expect the data model in V1 to persist and are not planning to make further large scale, fundamental changes to the Event Data API. As such, the version 1 release of the API is exceptional and a big step forward. It is important that we address these fixes before we go into production as it affects everyone who uses the service.

Same Event Data, new address

In setting up for the upcoming production service rollout, we have updated the Event Data API domain so that it is in line with Crossref’s suite of APIs. The Query API can now be found at a new URL. Here is an example query: https://api.eventdata.crossref.org/v1/events?rows=1

We have also simplified the standard query parameters in favor of a cleaner filter syntax.

Lastly, we have added a new “Mailto” parameter, just like in our REST API. It is encouraged but optional, so you are not obliged to supply it. We’ll only use it to contact you if there’s a problem.

Changes to the Wikipedia data structure

We’ve done a lot of work to use the canonical URLs for web pages to represent content as consistently as possible. This has entailed updating previously collected Events across data sources. As such, we’ve updated our Wikipedia data model to align with this. Because this update has impacted every Wikipedia Event in the system, we recommend those who have used or saved existing data from the deprecated Query API version to pull a new copy of the data. Read more about the rationale for changing the Wikipedia data model.

Updated data

This then brings me to how we now handle updated data. Sometimes we edit Events to add new features, or we may edit Events if there is an issue processing and/or representing the data when we provision it to the community. And sometimes we must remove Events to comply with a particular data source’s terms and conditions (ex: deleted Tweets). You can read about how updates work in the user guide.

To make life easier moving forward, we’ve split updated Events into two API endpoints. If you are already using Event Data, you will need to make some small updates to your client(s) to align with this. The new endpoints are further described in the documentation.

Event Data beta group

With the version 1 release we are making solid progress towards an official launch (the house-warming party!), we are quite excited to hear how you are using Event Data. Please consider [joining our beta group] (https://groups.google.com/forum/#!forum/crossref-event-data-beta-testers), if you are using the Event Data API or want to hear about updates.

This is also where you can read about these updates in more detail.

For more information and to get started with Crossref Event Data, please refer to the user guide.

I am looking forward to seeing how Event Data is being used, and working with the community to continuously improve what we can offer through this service. Feedback is always welcome, feel free to get in touch with me at eventdata@crossref.org.

A Lustrum over the weekend

Jennifer Lin — Mon, 26 Mar 2018 00:00:00 +0000

The ancient Romans performed a purification rite (“lustration”) after taking a census every five years. The term “lustrum” designated not only the animal sacrifice (“suovetaurilia”) but was also applied to the period of time itself. At Crossref, we’re not exactly in the business of sacrificial rituals. But over the weekend I thought it would be fun to dive into the metadata and look at very high level changes during this period of time.

Crossref provides the latest cumulative stats online. We share news about the work we do along the way in the Crossref blog, including periodic summaries such as the Executive Director’s 2017 end-of-year highlights and the annual review. But what follows is a brief and very informal survey of the population of inhabitants in the Crossref metadata-land for the current lustrum.

Works published

The first thing a census typically asks is population size. We know there are new records arriving each month with 95.7mil to date. And they do so at variable rates. But when the data is visualized, a rough yearly pattern emerges into view. (Data were collected on Mar 25, 2018; results are partial for this month.)

Each year brings with it a significant spike, an influx of new entrants, perhaps reflecting an increase in submissions at the end of the previous year. After January, volume drops down dramatically and gradually rises once more over the course of the year. We see smaller spikes at the March, June, and September mark. (Since this was a brief exercise, I did not dive into any formal research conducted on the nature of publishing cycles.)

Metadata Coverage

The next question is a look at how the population is broken up into different demographics. For this, I analyzed four key sub-populations of ORCID, funding information, license, abstract metadata. The following graph shows the percentage of new parties (i.e., works registered at Crossref containing these metadata) across four specific segments.

I ran Karthik Ram’s script which employed rOpenSci’s r client for the Crossref REST API. Data are based on publication date rather than deposit date and represent all updates to the metadata record for the baseline view.

The census graph shows extensive empty space on the top half, indicating there is ample room for continual growth in these communities. The ORCID population is expanding the fastest, followed by license and funding. Abstracts are a minority group and quite visibly needs a population boost here in Crossref-land.

This view does not capture the percentages across record types nor does it take into account the differential rate of growth between record types (e.g., journal article, book, report, conference proceeding, dissertation, dataset, component, posted content, peer review) as the Crossref corpus has grown. While ORCID, funding, and license information are available for all full record types (viz., excludes components), this matters for abstracts. Abstracts are part of the metadata schema of all relevant record types. This excludes those which do not apply: dataset, component, and peer reviews. All things considered though, the relative impact on the total percentage of metadata deposited (or not deposited) is miniscule given the small sums for these works.

Calling the real demographers & cartographers

This mini-pseudo-lustrum was the result of a few hours of play. The graphs have raised more questions than answers. We welcome more serious and earnest efforts to dive into the metadata and conduct a more detailed, reliable investigation on the size, distribution and composition of the population through our REST API. Next month, we will roll out reports on metadata coverage based on individual members.

This “play” census came out of a session with Karthik Ram, one of the founders of rOpenSci, as we talked about struggle to build better tools for researchers. (rOpenSci is an exciting and influential non-profit that builds open source software for research with a community of users and developers and educates scientists about transparent research practices.) With each round of cocktails, it became clear that a critical subset of the issues boiled down to the problem of limited information about research publications. Why, that is what Crossref does! Indeed. Publishers register their content with Crossref and provide the metadata about the works they publish.

Over the past few years, we have been working with our members to broaden the coverage of the metadata as well as improve their metadata quality. This issue is not exclusive to Crossref - Metadata 2020 rallies stakeholders across the research enterprise to push for change together.

To represent the full breadth and depth of the scholarly communications enterprise, Crossref aims to capture the richness of what our members publish through the content they register. So publishers, powerfully represent your services and make sure your metadata is complete and correct for discovery systems, indexing platforms, research evaluation systems, analytics tools, and the great number of Crossref metadata consumers far and wide.

How we use Crossref metadata

Christine Cormack Wood — Mon, 26 Mar 2018 00:00:00 +0000

Bruce Rosenblum, CEO, Inera Incorporated talks about the work they are doing at Inera, and how they’re using our metadata as part of their workflow.

Can you tell us little bit about Inera, and yourself

I’ve always been fascinated by the intersection of publishing and technology. At Inera I help scholarly and technical publishers improve their workflows through technology, and build editorial and XML software solutions to improve the publication workflow. I lead the development teams for our eXtyles and Edifix products, and I also participate in community projects: co-authoring the original NLM DTD suite, developing the Crossref Metadata Deposit Schema in 2001, and serving for 8 years on the NISO board. I continue to work on JATS and BITS development, and I co-chair the NISO STS Working Group. Before joining Inera, I developed publishing technology such as an Apple II Word processor for Chinese in 1981, and early micro-computer desktop publishing systems in the late 1980s.

At Inera, we develop and license the eXtyles family of Word-based editorial and XML software tools (including eXtyles, eXtyles NLM, eXtyles STS, and eXtyles SI) as well as Edifix, an online bibliographic reference solution. eXtyles and Edifix allow users to automate the most time-consuming aspects of document publication. Publishers of scholarly journals and books, standards, and government documents around the world rely on our software solutions to drive efficient, effective publishing workflows.

Inera and Crossref have collaborated since 2001, and we jointly won the 2014 NEPCo Award for the ongoing symbiotic relationship between our organisations.

What problem is your software and service trying to solve?

Publishers receive manuscripts from authors who have deep knowledge of their subject matter but are sometimes not expert writers and rarely expert users of Microsoft Word. Our eXtyles and Edifix solutions are designed to help publishers rapidly and accurately prepare these manuscripts for publication by automating a lot of technical and editorial cleanup, then producing high-quality JATS and BITS XML.

Within eXtyles and Edifix, we have sophisticated algorithms that heuristically parse bibliographic references, copyedit them automatically to a publisher’s editorial style, and then link them to Crossref and PubMed. These features eliminate a lot of repetitive detail copy editing work so that human editors can focus on higher-level editing tasks, and they produce more accurate bibliographies that include online links, with a fraction of the work that it would take to look up, check, and correct each reference manually.

How are you using Crossref Metadata at Inera?

Simply stated, we use Crossref metadata in our products to ensure that bibliographic reference lists are as complete, correct, and up to date as possible at the point of publication.

Both eXtyles and Edifix use Crossref metadata to improve reference lists. Our reference processing module pulls apart references to journal articles, books, book chapters, conference papers, and standards, applies elements based on the JATS reference model, and then reconstructs them according to the editorial style chosen by the user (e.g., AMA, APA, MLA, or a custom-configured style to meet customers’ requirements).

Crossref metadata is used for two primary purposes. First, we query the Crossref database to obtain DOI links for journal articles, books, conferences, and other types of references. This link lookup helps our customers fulfill their Crossref membership obligations and helps ensure that researchers get appropriate credit for citations of their work. Second, we use the metadata obtained from Crossref to improve the accuracy of author-supplied reference entries.

What values do you pull from our APIs?

The most important metadata value we retrieve is the DOI itself. Because the majority of bibliographic references in author manuscripts do not include DOIs, a key feature of our service is DOI retrieval. However, we use metadata well beyond the DOIs once we’ve matched a record. Even if a reference already has a DOI, we still do a traditional query, using the other available reference elements, to retrieve a DOI and compare the results to flag discrepancies. We’ve found that ~20% of author-supplied DOIs are incorrect, so correcting these discrepancies is one of myriad ways that our software uses Crossref metadata to improve references before publication.

We also pull all of the other fields that are used to build a bibliographic reference—complete author list, title of the item, publication date, volume, pages, and so on—and use these elements to correct and improve the reference. By filling in missing data (e.g., volume, issue, and page numbers) and flagging discrepancies between author-supplied entries and Crossref metadata (e.g., author names in a different order, words missing or misspelled in an article or chapter title), our software assures publishers of a high-quality bibliography with minimal manual effort.

Finally, we use Crossmark metadata to flag references that may have been corrected—or retracted—to inform editors when an item may need further attention from an author. Did the author knowingly cite a retracted article? If not, does that change the science of the paper citing that retracted item?

Have you built your own interface to extract this data?

Yes, we’ve built our own tools to query Crossref’s APIs. In 2002, we used the old “piped-query” API to submit elements of journal references, but we outgrew this API because it returned too many false positive results and missed other DOIs that were correct, and because we wanted to query Crossref for DOIs to other reference types (e.g., books, conference papers, reports) as well as journals. We switched to XML queries in 2006, and the result was a huge improvement in the quality and quantity of DOI links for our customers.

But just moving to XML queries still wasn’t good enough. Eight years ago, we wanted to improve DOI retrieval of non-journal items like reports, and we found that the existing Crossref APIs didn’t provide what we needed. So we collaborated with Crossref CTO Chuck Koscher to create the author–title query as an extension to regular XML queries. The result was a dramatic improvement in our ability to retrieve DOIs to non-journal items. The author–title query was a precursor to Crossref’s current metadata APIs, and it continues to serve us well.

How often do you extract or query data?

All the time! Our customers are located all around the world in more than 25 countries on six continents, so Crossref metadata queries from our software are happening continually, at any time of the day or night, seven days a week, and even on holidays.

There are two other important ways that our software interacts with Crossref APIs every day. First, Crossref’s Simple Text Query (STQ) service, which is used by smaller publishers to meet their Crossref requirement to add DOIs to their reference lists, was built using Inera’s reference parsing engine. In this case, our software runs on Crossref’s servers and is an integral part of the Crossref ecosystem.

Second, to test our products, we run a comprehensive automated quality assurance process every night that tests all aspects of our software and ensures day-over-day stability. When we added Crossref linking functionality in 2003, we began running several thousand Crossref queries per night, looking for consistency in our software’s results. A few months later, we noted an unexpected change: a reference that had previously returned a DOI failed to link! We contacted Crossref about the “lost” DOI, and upon investigation, Crossref discovered that in the process of redepositing 20,000 DOIs, the publisher had accidentally inverted author surnames and given names in all of those records.

Crossref immediately recognized the value of Inera’s automated testing, and its ability to unearth such errors, to Crossref and its members. Over time, the number of DOIs we test nightly has grown to tens of thousands, so we’ve worked with Crossref to develop an automated reporting and analysis process that makes detecting and resolving the issues highlighted by our internal testing more efficient.

The co-development of the author–title query API and the sharing of our nightly test suite results are just two examples that highlight the nature of the Inera–Crossref relationship: it’s characterized by technology integration, bidirectional information exchange, and innovative problem solving.

What are the future plans for Inera?

We’re constantly working to improve eXtyles and Edifix and to develop new and innovative ways to help our customers. Here are a few examples:

Two years ago, at the peak of the Zika outbreak, we received an urgent request from the World Health organisation to help them create DOIs for articles that had been submitted but not yet peer reviewed (see Zika Open). Within 16 hours of their request, we developed, tested, and deployed updated software that allowed WHO to publish information vital to researchers, including DOIs, within hours of receipt.

With respect to Crossref APIs, we plan to integrate the Crossref query features to retrieve DOIs for standards that are deposited by organisations like IEEE, ASTM, and BSI. We also plan to expand our linking and verification capabilities to incorporate newer reference types such as preprints and data citations.

More broadly, we’re very excited about the eXtyles Metadata Extraction technology we released last year. This technology can be used by online submission systems and preprint servers to automatically extract key metadata elements (title, abstract, authors, affiliations, keywords) from author-submitted manuscripts, no matter what “style” the author may have used to format the manuscript. This technology is already in-use at Aries Systems to simplify the submission process. We’re looking forward, soon, to seeing this technology used by preprint servers and institutional repositories to automate the collection and deposit of preprint metadata to Crossref.

New Board Chair Paul Peters shares our mission

Ginny Hendricks — Thu, 22 Mar 2018 00:00:00 +0000

At the end of last year, Paul Peters—CEO of our member Hindawi—became the new Chair of the Crossref Board. The announcement was made in Singapore at our first LIVE Annual ever held in Asia. I caught up with Paul back in London, UK, where he answered a few questions about what he hopes to bring to the Board, and to the Crossref community as a whole.

1. Congratulations, Paul. How delighted were you to be voted in by your fellow board members, old and new?

That’s a rather leading question ;-)

Seriously though, I am incredibly honored to have been chosen to lead Crossref’s board at such an important point in the organisation’s development. The current composition of the board is as diverse as it has ever been, which is essential if the board is to represent Crossref’s global membership, as well as the wide range of business and publication models that our members use. This diversity on the board will help to support Crossref’s aim of encouraging innovation in scholarly communication by providing open infrastructure that benefits all researchers.

2. You’ve been on our board for nine years. How has it changed in that time and what should the board be most proud of?

When I first joined the board, Crossref was at the stage where you had successfully established persistent reference linking as a standard practice among scholarly journal publishers. And, although this was the original purpose of Crossref, it was by no means an easy task, as it required a diverse group of competing publishers to work together in building shared infrastructure for the common good.

In the nine years since then, I’ve seen Crossref continue to build on this core foundation of technological expertise, the trust and goodwill of its membership, and the diverse skills of its small staff. The result has been the development of important new services (such as Similarity Check) that have become an essential component of the scholarly communications system, support new record types (including both preprints and peer review reports) that are becoming increasingly important in the move towards an Open Science future, and the expansion of Crossref’s membership to include almost 10,000 members of all shapes and sizes from 114 countries around the world.

With regard to the board itself, I have been pleased to see Crossref undergo important changes that have provided greater transparency in the organisation’s governance, as well as more active participation from its members. Last year Crossref put out an open call to invite members to put themselves forward for consideration on the board. As a result of holding its first contested election, Crossref saw a dramatic increase in the engagement of members in the election process. Not only is this important for ensuring that the board is truly representative of the diverse membership, but it will also help to actively engage a larger pool of members in the important work that lies ahead.

3. What do you see as Crossref’s strengths and role?

I believe that Crossref’s past and future success relies on two key strengths. The first is its ability to bring together a large and disparate community of organisations and individuals to create tools and services that no single organisation could develop alone. People sometimes overlook how successful Crossref has been in building the trust and support of a diverse group of stakeholders, however I believe this has been an essential ingredient in the organisation’s success and will be essential as Crossref develops new tools and services in the years to come.

Crossref’s other core strength has been the expertise, passion, and ambitious vision of its staff, many of whom I have had the pleasure of knowing since my first days on the board. The ability to develop and maintain real-time infrastructure serving millions of end-users, while simultaneously developing new products and services, requires an incredible range of skills from technology and product development, to marketing, community outreach, and customer support. Moreover, as a growing non-profit organisation with thousands of members around the world, and an international staff working across national boundaries, Crossref’s legal, financial, and administrative support team have also been an essential ingredient in the organisation’s success.

4. We’ve grown beyond just the publisher constituency to libraries, scholars, and platforms and tools, which constituencies do you see us involving next?

Over time I believe that Crossref’s constituency will grow to cover all organisations that contribute to the creation and dissemination of scholarly research, although I recognize this may take several years to achieve.

In the short-term, I believe that research funders are the most important stakeholder group for Crossref to focus on, for the following reasons:

First, with the development of the open Funder Registry and the addition of structured funding data to the Crossref registry, Crossref has already become an important provider of open infrastructure for research funders.
Second, as the result of several key initiatives within the Open Science movement I believe that research funders will play an increasingly important role in determining how scholarly research outputs are created, shared, evaluated, and re-used. Therefore, the active involvement of research funders in Crossref’s membership and governance is essential.
Finally, I believe that there is an important opportunity for Crossref to enable a range of new services across the research lifecycle by providing persistent identifiers and structured metadata research grants. Given how critical grants are within the research process, I’m amazed by the lack of infrastructure to monitor, evaluate, and build upon grants as first-class research objects. In many cases there is minimal, if any, public information about the grants that have been awarded by a particular funder. Even in cases where such data is available, it is rarely structured in a way that enables it to be searched or analyzed across multiple funding agencies.

In the absence of a community-driven, non-profit organisation like Crossref to provide this infrastructure on an open basis, there is a risk that funders will be forced to rely on proprietary alternatives that limit how this information is used and by whom. Fortunately there are already efforts underway within Crossref to develop both the tools and the community of funders that will be required to create persistent identifiers and structured metadata for grants and other forms of research funding.

5. What are the biggest challenges facing Crossref?

I believe that Crossref’s greatest challenge will be to continue to bring together a diverse group of stakeholders, some of whom are regularly at odds with each other, in order to collaborate in developing tools and services for the benefit of the research community.

As challenging as it has been for Crossref to bring together competing publishers to build the shared services that we have all come to depend on, I believe that keeping the community focused towards a common goal will become even more challenging as that community expands to include funders, universities, and the many other organisations involved in the scholarly communications ecosystem. However, I think that Ed and his team have as good of a chance of succeeding as anyone could hope for, which is why I am so excited about Crossref’s future in the years ahead.

6. How will things change with you as Chair? You’ll be busier I guess. But enough about you already, what can we expect as staff and Board?

As my first order of business I’ll be getting rid of Crossref’s corporate jet, lavish office spaces, and executive chef. </sarcasm>.

On a more serious note, my hope is that as Chair I will be able to work with the other members of the board in supporting Crossref’s staff as they work to achieve the ambitious goals we have set out during the past year. I believe that Crossref’s board members and staff are aligned in the desire to significantly expand the range of services Crossref provides, as well as the communities it serves.

The board still has an important role to play in shaping the organisation’s strategic vision, while giving staff ample space to execute on this vision. Said another way, I hope to enable some lively strategic conversations among the board while making sure that we don’t get in the way of Ed and his team once it’s time to put ideas into action.

On a more personal note, I hope to be a good sounding board for Ed on any issues that he faces, either internally or externally, on the road ahead. Given my own experience in leading a growing organisation through a period of significant change, I know how important it can be to have someone to talk to when difficult challenges arise, which they inevitably will. I hope that I can be a good advisor—and also a good friend—to Ed as he leads Crossref into the exciting future that lies ahead.

Ginny: Thanks, Paul. I know Ed will miss his personal chef… but we look forward to working with you too!

Crossref LIVE in Tokyo

Rachael Lammey — Thu, 08 Mar 2018 00:00:00 +0000

What better way to start our program of LIVE locals in 2018 than with a trip to Japan? With the added advantage of it being Valentine’s Day, it seemed a good excuse to share our love of metadata with a group who feel the same way!

We’ve worked closely with the Japan Science and Technology Agency (JST) since 2002, and were delighted when they agreed to collaborate with us on a LIVE event at their offices in Tokyo.

With help from the team at JST, we welcomed around 80 attendees—a mix of editors, publishers and enthusiastic metadata users—who all enjoyed the talks from our guest speakers, Nobuko Miyari from ORCID, Ritsuko Nakajima from JST and Tatsuji Tomioka from Kyoto University Library (who talked about the use of DOIs and metadata in their research information repository, named KURENAI).

Vanessa Fairhurst and I also took part in the days program and talked about the different services that Crossref offers. With many of our members in Japan already well-versed in DOIs, we placed the focus of our sessions around the importance of accurate, complete metadata, and new record types (such as peer reviews and preprints). We also discussed our new community initiatives such as the OI project, identifiers for grants and Metadata2020.

We’d like to say a big thank you to Kentaro Kinoshita from JST for his help with organizing the event. We’d also like to thank the excellent team of translators who assisted us greatly by relaying the content to the audience in Japanese—being able to offer information and take questions in English and Japanese was an invaluable part of the day.

Any questions?

One day is never quite enough to cover all things Crossref, so we were happy to answer questions from the enthusiastic audience:

What metadata is required to register peer review reports with Crossref?
To answer this we pointed them to this informative blog on peer reviews.

How can I find information on using your REST API?
This is a great starting point, and most information can be found here https://api.crossref.org

Is the forthcoming Metadata Manager tool something I can use?
Yes! We hope it will make it much easier for you to deposit good metadata—and if you are in interested in participating in our open beta, let us know.

We’re looking forward to continuing to collaborate with JST, and are really grateful for their help in working with us to make the event go so smoothly. Thank you to those who joined us, and we hope to see you again soon.

Are you having an identity crisis?

Amanda Bartell — Fri, 23 Feb 2018 00:00:00 +0000

We work with a huge range of organisations in the scholarly communications world—publishers, libraries, universities, government agencies, funders, publishing service providers, and researcher services providers—and you each have different relationships with us.

Some of you are members who create and disseminate your own content, register it with us by depositing metadata, and help steer our future by voting in our annual board elections. Some of you don’t vote in our board elections but do play a vital role by registering content on members’ behalf.

And some of you make use of the metadata provided by our members and so perform a key service by getting their published works out into the world, but don’t vote in our board elections.

After a recent review we realized our Member Types weren’t completely clear, and may in fact have led to a bit of confusion. With this in mind, we put some thought into their revision and have now given them the clarity they were missing. Over the course of this year we’ll be checking that everyone is in the right group and getting the appropriate support based on your Member Type.

Former Member Type name	New Member Type name
Publisher	Member
Sponsoring Publisher	Sponsoring Member
Represented Member	Sponsored Member
Sponsoring Entity	Sponsoring organisation
Sponsored Member	Sponsored organisation
Affiliate	Metadata User
Service Provider	(No change to Member Type name)

So, what’s different?

The changes we’ve made help to differentiate if you’re a voting member (and therefore have a say in our future direction), or not. If you are a voting member, you’ll now have the word “Member” in your title—and if you’re not—you won’t, as the diagram below indicates.

Where there are two organisations with a sponsorship arrangement in place (with a sponsoring party and a sponsored party), one of you will always be the voting party, and the other will be non-voting. These partnerships will therefore always contain one “Member” and one “organisation”.

We’ve also stopped using the word “Publisher” in our Member Types as not all our members consider themselves to be publishers — sometimes you’re libraries, funders, scholars, repositories, etc. As it says in one of our truths “Come one, come all: we define publishing broadly. If you communicate research and care about preserving the scholarly record, join us.”

How do you know if you are a voting member?

Voting members fall into three Member Types: Members, Sponsoring Members and Sponsored Members.

This means you are organisations who create and disseminate content, and therefore contribute to the scholarly record. Some of you register your content directly with us and some via a third party, but the key thing is that you’re adding to our metadata records, and as such can have a say in the future direction of Crossref. Voting members can also take metadata out of our system — and many of you do — however, your key relationship with us is as a member who is contributing to the scholarly record.

It also means you have obligations to keep your records up-to-date, and maximize links with other Crossref members.

What’s the difference between the voting categories?

Members
As a Member (formerly known as Publishers), you create and disseminate content, register your own content with us (usually under a single prefix), and are able to vote in our board elections. You pay an annual fee based on your publishing revenue, plus Content Registration fees for all new DOIs.

Sponsoring Members
As a Sponsoring Member (formerly known as a Sponsoring Publisher), you do everything a standard member does, but as well as registering your own content under your own DOI prefix, you also register content on behalf of other, smaller publishers (ideally using separate DOI prefixes so the metadata is accurate and can be reported on separately and relied upon downstream).

When you vote, you vote on behalf of the organisations that you sponsor. You pay an annual fee based on your publishing revenue/expenses plus the publishing revenue of your sponsored organisations, and you also pay Content Registration fees for all new metadata records registered. You look after deposit billing for the organisations you sponsor, and provide technical and language support for them.

Some of our larger members may be thinking that you should be in this Member Type - and you’re probably right! During the course of 2018 we’ll be working with you to transition you over to Sponsoring Membership. If you are a Member who is thinking of becoming a Sponsoring Member, please get in touch.

Sponsored Members
As a Sponsored Member (formerly known as a Represented Member), you create and disseminate content, but you don’t register your content directly with us—this is done by your Sponsoring organisation. Because of this it’s you, the one who creates and disseminates the content and thus contributes to the scholarly record, who can vote.

How do you know if you are a non-voting member?

If you haven’t spotted yourself yet, you may be one of the non-voting organisations we work with — these fall into four Member Types: Sponsoring organisations, Sponsored organisations, Service Providers and Metadata Users.

As a non-voting organisation, you may still register content with us, but you either don’t create and disseminate the content yourselves, or you’re already represented by a voting organisation. Non-voting organisations also include those whose only relationship with us is to make use of our metadata.

What’s the difference between the non-voting categories?

Sponsoring organisations
As a Sponsoring organisation (formerly known as a Sponsoring Affiliate), you don’t create and disseminate content yourself, but you do register content with us on behalf of your Sponsored Members — preferably using distinct DOI prefixes for each member. You also often look after their administrative, technical, billing and language support needs. You’ll pay us an annual fee based on the publishing revenue of all your members, and Content Registration fees for all new DOIs. You might charge the members you work with for this service. You also provide support and promotion of our services and activities.

Sponsored organisations
As a Sponsored organisation (formerly known as a Sponsored Member), you do create and disseminate content yourself, but you don’t register your own content. This is done by a Sponsoring Member, and as they have the member vote, you can’t have one too. For this reason, we’ve removed the word “Member” from your title, to make your voting position clearer. Of course, your Sponsoring Member needs to represent your needs too when voting, so make sure you make them known!

Service Providers
As a Service Provider you work closely with our members to collect and/or host and/or deposit metadata on their behalf. Unlike a Sponsoring organisation however you don’t get involved with administrative, technical, billing or language support for the members you work with, but you’re a key partner in helping them deposit quality metadata and contribute effectively to the scholarly record. During 2018 we’ll be working more closely with you to help you collaborate with us more effectively.

Metadata Users
Metadata Users (formerly known as Affiliates), you are the organisations who don’t register content with us, but you do make use of it through our free and open APIs and search interfaces, or our paid-for Metadata Plus service, giving you access to a premium version of both the REST API and OAI-PMH. Of course all members can get metadata out of our systems as well, but if the only thing you do with us is get metadata out, then you’re a Metadata User.

Don’t know which Member Type you are?

We’re hoping these new names make it clearer, but if you’re still confused, please get in touch with our membership specialist

Wellcome explains the benefits of developing an open and global grant identifier

Ginny Hendricks — Fri, 16 Feb 2018 00:00:00 +0000

Wellcome, in partnership with Crossref and several research funders including the NIH and the MRC, are looking to pilot an initiative in which new grants would be assigned an open, global and interoperable grant identifier. Robert Kiley (Open Research) and Nina Frentrop (Grants Operations) from the Wellcome explain the potential benefits this would deliver and how it might work.

Introduction

As a funder we want to be able to track the outputs that arise from research we have funded. Currently, this is not as straightforward as it should be as researchers do not always cite their funder correctly, let alone their specific grant number. And, even when they do this accurately, because every funder users its own set of grant IDs, these numbers are not unique. For example, we can use EuropePMC to look up outputs from grants with ID 207467, and see that there is one Wellcome grant with this number, and one from the European Research Council.

To resolve such issues, we need a system in which every grant awarded is giving a unique, global ID. Global IDs are already assigned to articles DOIs, people ORCIDs and even biological materials RRIDs. It is time for the funder community to follow suit.

Benefits of an open & global grant identifier system

Once implemented, it would make the identification of grant-specific research outputs more accurate, whilst simultaneously reducing the burden on the researcher.

Currently, researchers are typically asked to manually disclose what outputs have arisen from their funding. In the future, such disclosures would be fully automated. We are already seeing how publishers—who collect ORCIDs through their manuscript submission system—automatically update the author’s ORCID record with details of new publications. If a global ID system for grants was developed, publishers and repositories could also require these to be disclosed on submission, and this data could then programmatically be passed to researcher assessment platforms, like ResearchFish.

How would it work?

For a global grant ID system to work, two things need to happen. First, when a new grant is awarded, that grant must be assigned a unique ID. For the pilot project we plan to contract with Crossref who will register a unique ID, (a DOI) for every grant we register.

Second, every DOI must resolve to a publicly accessible web site, where information about that grant is disclosed. Again, for this pilot we will almost certainly use the Europe PMC Grants Finder Repository, as we already make grant data available from this resource.

A working group has been established to determine precisely what metadata we should make available, but it is likely to include the name of the grant holder, title and value of the award, a short abstract, along with the name of the funder and the unique ID. Mindful that funders already assign IDs to the grants they award and that any changes to this process may be problematic (and certainly time consuming), the plan is to register a DOI which still makes use of the existing grant ID. To make it unique however, the ID will be prefixed with a funder identifier, most likely the Funder Registry ID.

Next steps

Whilst the metadata working group is focusing on the technical aspects of the pilot, a separate “governance group” is examining how a funder might become a member of Crossref and what the business model for registering grant DOIs should be.

In parallel with this, a pilot “proof of concept” initiative is under way, and we anticipate that by autumn 2018 we will have registered DOIs for a defined cohort of grants.

Ultimately we want to get to a situation where every grant has a unique ID, which can then be unambiguously linked to the all outputs – articles, data, code, materials, patents etc. – which arise from it.

And, if every funder were to adopt such a system and expose their grant metadata in a consistent, machine-readable way, it would facilitate the development of applications to help funders get a greatly enhanced picture of the global funding landscape, which in turn would inform strategic planning and resource allocation.

Thanks to guest authors:

Robert Kiley, Head of Open Research, Wellcome [ORCID: 0000-0003-4733-2558] Nina Frentrop, Grants Information & Systems Manager, Wellcome

Please read Crossref for funders for context, and contact Ginny Hendricks at Crossref with any questions.

Meet the members, Part 2 (with protocols.io)

Christine Cormack Wood — Wed, 31 Jan 2018 00:00:00 +0000

Second in our Meet the members blog series is Lenny Teytelman, co-founder and CEO of protocols.io, who gives us a bit of insight into his background and why he started protocols.io, what the future plans for protocols.io are, and how they use and benefit from being a Crossref member.

Can you tell us a little bit about yourself, and why you started protocols.io?

I am a computational and experimental biologist, and it was my struggle with correcting a published research method as a postdoctoral researcher at MIT that led me to co-found protocols.io. I spent a year and a half correcting a single step of a research recipe. Instead of 1ul of a chemical, it needed 5, instead of a 15-minute incubation, it needed an hour. But this was a correction of something previously published, not a new method, so absurdly, it was not a result that I could publish. That means I got no credit for this year and a half, and more importantly, every other scientist using this recipe is either getting misleading results or has to waste 1-2 years rediscovering what I know—rediscovering something that I’d love to share, but have no easy way of doing so.

So, I became obsessed with creating a central place where scientists can easily share and discover detailed research recipes. We’re open access, free-to-read and free-to-publish, with web & mobile apps that make these protocols dynamic and interactive.

What problem is your service trying to solve?

Currently, methods sections of research papers are full of things like “we used a slightly modified version of the method reported in paperX”. Here are two examples:

We are working to increase reproducibility, by encouraging precise detailing of methods and then making it easy to keep these methods up-to-date, long after the paper is published. More broadly, our mission is to accelerate science by getting the detailed knowledge out of paper notebooks, and getting it out in months, instead of years.

Tell us a little bit about what you publish and for whom.

Both the content, and the audience for it, has been expanding recently. When we launched in 2014, the protocols were almost exclusively wetlab biology recipes. In 2015, we added support for computational workflows and began to see bioinformatics methods. More recently, thanks to the referrals from PLOS ONE, we’ve started to see protocols for human trials, medical devices, psychology, and more. About half a year ago, we changed our landing page form “Open Access Repository of Life Science Methods” to the more general “Open Access Repository of Research Methods”.

The readership is also broadening, it’s no longer just professional researchers—we now have protocols and guidelines for undergraduate and high school students, instructions for citizen science projects, and even standard operating procedures for lab management. We’ve also been seeing more off-the-shelf use, with people sharing actual cooking recipes, and we recently began asking authors to classify whether they are sharing “research” or “non-research” instructions.

How would you describe the value of being a Crossref member?

Without a doubt, we would be nowhere close to the adoption and sharing that we have now if we were not members of Crossref, registering DOIs for all public protocols. This is an absolute prerequisite for being included in author guidelines of journals, and we wouldn’t have grown in 2017 from two to over 200 journals that encourage authors to detail their recipes on protocols.io.

In addition to the benefit to protocols.io, there is a benefit to the scientists in terms of the quality control that Crossref ensures among the members. Much of this is behind the scenes and invisible to the researchers visiting protocols.io.

For example, in the beginning, we used to simply delete spam protocols. However, once we started issuing DOIs, we realized that we would be violating the Crossref requirements for minted DOIs if we simply trashed these. As a result, we had to build “retraction” functionality that allows us to take down content, put up a notice explaining the reason for removal, and keep the record so that the respective DOI continues to resolve. This is the correct way to handle removals of scientific content and it is Crossref that made us mature and improve the platform. (We’ve since had to use the retraction functionality at the request of scientists, and we’re glad we implemented it to comply with the Crossref requirements.)

Another example is the resolution report that we routinely get from Crossref, showing us which DOIs are broken. It highlights errors for us and helps us to investigate, identify, and prevent problems with the journal partners.

What do you see as the value of Crossref, beyond protocols.io?

As I argued in my talk at the annual Crossref conference, we are finally in a position to connect scientists with the knowledge they need, automatically. Almost every scientist uses a reference manager such as Mendeley, Zotero, Paperpile, etc. to manage their literature bibliography. In turn, that means that in theory, when something happens to a paper or research objects connected to the paper (retraction, correction, update to the dataset accompanying the manuscript), the reference management platforms could notify every scientist who has that paper in their library.

The problem is that it isn’t feasible for every service like Mendeley to connect to every repository and publisher to track events connected to every paper. This is where Crossref is positioned so powerfully. By collecting the metadata linking papers to the research objects, Crossref can be the single source that the platforms need to query to see if there is news for their users related to any specific published paper. (More of this from my talk was captured really nicely in this blog post by the SMU librarian Aaron Tay.

What are the future plans for protocols.io?

Expanding protocols.io content to include chemistry workflows is an important goal for 2018-19.

We are also eager to start on connecting the protocols directly to the devices that the scientists use. Imagine you need to spin your cells for 30 seconds, but the centrifuge is accidentally set for 3 minutes. Our app should be able to connect to the equipment and alert the researcher to the wrong setting, asking if they are sure they want to proceed.

No longer lost in translation

Christine Cormack Wood — Tue, 30 Jan 2018 00:00:00 +0000

More than 80% of the record breaking 1,939 new members we welcomed in 2017 were from non-English speaking countries, and as our member base grows in its diversity, so does the need for us to share information about Crossref and its services in languages appropriate to our changing audience.

So, early last year we started translating our service videos into six other languages: French, Spanish, Brazilian Portuguese, Chinese, Japanese, and Korean. However, the process of translating from one language to another is not always straightforward—but it is super important—as some things can get seriously lost in translation…

||||

In order to avoid such translation tragedies we created a foolproof process to get the text of the service videos translated and ready for production. (I am, I realize, exposing myself here—see what I did there? —by using a word like foolproof.)

First we produced the videos in English, setting the content to animation and sound (AKA audio visual or A/V to us marketing types), then we brought in a translation company to turn the English content into the six other languages. So far so good. However, as the above examples demonstrate, the meaning of words can get lost in translation. Also, what Crossref does isn’t the easiest thing in the world to translate (are there words for metadata delivery and full-text XML in Japanese?), so we added another stage to the process.

Next, we sent the translated scripts and their English counterparts to some very helpful international members who, as part of the scholarly research community, understand the complexities of our work and are therefore qualified to check that the text had remained in context.

Unfortunately, it hadn’t, as the text came back from them heavily edited. After round two of the editing process, the revised text was applied to the videos—but just to be 100% sure, we sent the completed videos back to our helpful international members for a final run through.

Multiply this painstaking process by 48 videos, throw numerous time zones into the mix and you can see why it took us nearly 12 months to complete them.

And so, it is with great pleasure that today we launch all eight of our service videos in six languages, just click the links below, and enjoy! Découvrez-les! ¡Que los disfrutes! Aproveite! 请欣赏! どうぞお楽しみください！ 즐거운 시간 되세요!

			View videos by language
English	French	Spanish	Brazilian Portuguese	Simplified Chinese	Japanese	Korean
English	français	español	português do Brasil	简体中文	日本語	한국어로

We’d like to thank the following for their help in checking the video translations: Fabienne Meyers from IUCAP for the French versions, our very own resident translator Vanessa Fairhurst for the Spanish versions, Edilson Damasio from the University Library of Maringá for the Brazilian Portuguese versions, Guo Xiaofeng from Wanfang Data for the Chinese versions, Nobuko Miyairi from ORCID for the Japanese versions and Junghyo from Nurimedia and Jae Hwa Chang at infoLumi for the Korean versions.

A year in the life of Crossref

Ed Pentz — Tue, 23 Jan 2018 00:00:00 +0000

We are delighted to report that last year Crossref welcomed a record-breaking 1,939 new members and, because our member base is growing so rapidly in both headcount and geography—with the highest number of new members joining from Asia—we thought it was a good time to reiterate what Crossref is all about, as well as show off a little about the things we are proud to have achieved in 2017.

What is Crossref?

We are an organisation that runs a registry of metadata and DOIs of course, but we are much more than that—staff, board, working groups, and committees as well as a broad range of collaborators, users, and supporters in the wider scholarly communications community. Increasingly, our community includes new contributors like scholars, funders, and universities. Together, we are all working toward the same goal—to enhance scholarly communications. Everything we do is designed to put scholarly content in context so that the content our members publish can be found, cited, used, and re-used.

Here’s how we did that over the past year:

We rallied the community

Rallying the community is all about working together to forge new relationships and pave the way for future generations of researchers—in 2017 we were closely involved with the launch of Metadata 2020; a collaboration that advocates richer, connected, and reusable metadata for all research outputs.

We tagged and shared metadata

To make sure that our APIs continue to have real, genuine utility, we introduced a new service called Metadata Plus in 2017 so that platforms and tools can leverage the power of our rich, immense database to increase the value and discoverability of content.

We played with new technology

To keep pace with changes in the industry and stay true to our mission, we often play with new technology with the goal of offering a bigger and better infrastructure. In 2017 we formed a working group and an advisory group for two new identifiers that will see this infrastructure increase; organisation IDs which became ROR, and Grant IDs which became the Crossref Grant Linking System.

We made new tools and services

Combining our own knowledge and experience with input from the wider community, in 2017 we were able to launch in Beta a new and exciting tool called Event Data. Event Data provides a record of where research has been bookmarked, linked, recommended, shared, referenced, commented on etc, beyond publisher platforms—which is a great example of putting scholarly research in a wider context.

So, while richer metadata (including more record and resource types) remains our focus 2018 and beyond, we also hope that as we become a bigger and more global community we can move beyond the basics and work together to make sure that DOIs, are not the be-all and end-all when they are, in fact, just the beginning.

Bridging Identifiers at PIDapalooza

Joe Wass — Mon, 22 Jan 2018 00:00:00 +0000

Hello from sunny Girona! I’m heading to PIDapalooza, the Persistent Identifier festival, as it returns for its second year. It’s all about to kick off.

One of the themes this year is “bridging worlds”: how to bring together different communities and the identifiers they use. Something I really enjoyed about PIDapalooza last year was the variety of people who came. We heard about some “traditional” identifier systems (at least, it seems that way to us): DOIs for publications, DOIs for datasets, ORCIDs for researchers. But, gathered in Reykjavik, under dark Icelandic skies, I met oceanographic surveyors assigning DOIs to drilling equipment, heard stories of identifiers in Chinese milk production and consoled librarians trying navigate the identifier landscape.

In addition to the usual scholarly publishing and science communication crowd, it was encouraging to see a real diversity of people from different walks of life encounter the same problems and work on them them collaboratively. The thing that brought everyone together was the understanding that if we’re going to reliably reference things – be they researchers, articles they write, or ships they sail – we need to give them identifiers. And those identifiers should be as good as possible: persistent, resolvable, interoperable.

Who cares about PIDs?

At the turn of the century, a handful of publishers came together to create Crossref (or CrossRef as it was in those days). It was becoming increasingly important to be able to store references in machine-readable format, but publishers were faced with a problem. If an author wants to cite an article, they’ll do so without worrying who published it. This means they needed an identifier system that worked across all publishers. Thus the Crossref DOI was born.

Today we’re heading toward 10,000 members, and the thing that they have in common is that they all produce scholarly content and care about how it’s referenced. As a trade association, we effectively act on behalf of all of our members, allowing them to register their content, share metadata and links, and assign an identifier.

But there’s a whole world out there. Publications have never been the be-all and end-all of scholarship, but they have been a backbone. But more and more scholarship, especially science, is done outside journal publishing. Sometimes it’s done on platforms that care about the scholarly record as much as publishers. And sometimes it isn’t.

The Twitterverse

Lots of people use Twitter to talk about science. Some are scientists, some aren’t. Scientific articles are linked from news reports and discussed on blogs. Gone are the days of scholarly articles being cited only by other scholarly articles. We see links coming in from all over the place. And, although not all of this can be counted as the “scholarly record”, some of it could be.

The barrier-to-entry for journals publishing means that science journals contain only science articles. The barrier-to-entry for Twitter means that anyone can, and does, publish there. My Twitter feed is finely balanced between bibliometrics research, marine biology and pictures of snow leopards with Japanese captions. I don’t understand all of it, but I like looking at the pictures.

Back in the days when the only references to scholarly publications were from other scholarly publications, it was easy to keep track of those references. When an article was published, its references went into a citation database. This happened because the publisher considered this important.

But Twitter, the publisher of tweets, doesn’t care. It is used for a huge variety of communications and although some people choose to use it to engage in scholarship, we’re just a blip on their radar. The same goes for Reddit, a platform that describes itself as “the front page of the Internet”. There are communities engaged in scientific discussions, but Reddit doesn’t feel the need to publish its bibliographic references.

Nor should it.

Bridging those who care with those who don’t

The barrier-to-entry for contributing to scientific discussions has lowered, meaning that the role of more non-specialist platforms has increased.

I imagine that there are other communities out there who have their own concerns about the web. Maybe there are model train enthusiasts who want to keep track of every reference to a particular model. Or political commentators who want to keep track of how certain politicians and policies are discussed. As the scholarly community embraces new platforms for communicating, we should recognise that we are part of a broader universe of people using those platforms for more diverse reasons.

Gone are the days when the only way to reply to an article was by writing a letter to the editor. But also gone are the days when you could guarantee that your letter wouldn’t appear next to cat pictures (assuming you weren’t writing to the Journal of Feline Medicine & Surgery). As a specialist community cohabiting online spaces with non-specialists, it falls to us to do whatever we need to adapt that space and make it our own. In our case, this means recording bibliographic references as and where they occur.

Something like this happened once before. As traditional publishers went online, they created Crossref to build and maintain the necessary infrastructure. We’re acting on behalf of the community again to collect links from non-traditional sources. Because we can’t go to platforms like Twitter and say “please deposit your references”, we’re doing the opposite. We identify a platform, then work out how to scrape its content and extract links.

Working at scale

So we’re broadening out the universe of references that we would like to track from “traditional scholarly publishing” to “the entire web”. There are four broad challenges inherent in this, and we think that Crossref infrastructure is the right way to meet them.

The first challenge is physically finding the links. Because social media platforms aren’t specialised for scholarly publishing, they don’t have the same mechanisms in place for capturing bibliographic references. This means that we have to do it ourselves by scraping webpages for references. As the standard-bearer for scholarly PIDs, we think we can do a good job of this.

The second challenge is doing this at the scale of the web. Because we might, in theory, find a link on any webpage, there is a literally infinite number of publishing platforms. From big websites like BBC News down to tiny blogs run out of a bedroom. It would be impossible to partner with each of these individually. The way to solve this is to run a centralised service which goes out and contacts as many sources as possible. This role is a collaborative one. Our system is open to inspection, suggestions and contributions from the community.

The third challenge is the sheer number of publishers. Because they all register content with us, we are in good position to track their DOIs. In addition to that, every member of Crossref publishes content on their own platform, and has their own set of websites to track. We monitor our members’ websites and create a central list of domains that we look for. If this wasn’t done centrally, each publisher would have to run its own web crawlers and perform the same work, only to filter out their own links.

The fourth challenge is how to get all that data to the public. Even if every publisher were able to run their own infrastructure, it would make it very difficult to consume. Through Crossref metadata services, publishers have built a system where you can look up metadata and link to articles without worrying who published them. We think that the same approach should apply to this new link data.

For these reasons, we’re building Crossref Event Data: a system that monitors as many platforms as we can think of, and brings them into one place, and serves the whole community.

Building bridges

If you’ve been following along you’ll know that my last metaphor was the process of refining crude oil. I like metaphors, and mixing them. After all, you can’t mix a good metaphor without breaking a few eggs into the mixing bowl. Today’s metaphors are bridges. And not just one.

Bridge 1: PIDs and URLs

In the world of Persistent Identifiers, we’re quite good at linking. organisations like Crossref, DataCite and ORCID run separate systems but we work together to record and exchange links. But the web is different. There’s no single organisation in control and there are many organisations working to catalogue it. Event Data is our offering: bridging the web with our identifiers.

Bridge 2: Scholarly link providers

Of course, some platforms and systems do care about persistence and Persistent Identifiers. Event Data is an open platform, and we’re collaborating with a few providers to publish links.

We’ve partnered with The Lens to include Patent to DOI references. We’re working with F1000 to include links between reviews and articles. Hopefully we’ll see more organisations use Event Data to publish their links.

Bridge 3: Crossref / DataCite

Event Data is a collaborative project between DataCite and Crossref. When Crossref Registered Content contains a reference to a DataCite DOI we put it into Event Data. DataCite do the same in reverse. This means that Event Data contains a huge number of article - dataset links.

Bridge 4: Traditional discussions vs new ones

At each moment, scholarly discussions are happening in the literature, on various social media platforms and on the web at large. They are all talking about the same thing, but are spread out. Event Data collects links wherever we find them and brings them into one place. By doing this we hope we can help bring those conversations together.

Bridge 5: Bridging bibliometricians and altmetricians to data sources

Capturing links from social media to published literature underpins the field of altmetrics. By collecting this data and making it available under open licenses, we bring it to altmetrics researchers. We don’t provide metrics, but we do provide the data points that can form the basis for research.

Without infrastructure for collecting data, researchers would have to perform the same work over and over again. Because the data is all open, we allow datasets to be republished, reworked and replicated.

Bridge 6: Bridging the Evidence Gap

Running Event Data involves collecting a lot of data - gigabytes per day - and boiling it down into hundreds of thousands of individual Events per day. People consuming the data may want to do further boiling down. At every point of the process we record the input data that we were working from, the internal thought process of the system, and the Events that were produced. A researcher can use the Evidence Logs to trace through the entire process that led to an Event.

We’re a bridge from websites and social media to data consumers. But we take the role very seriously, and there’s nothing hidden. A glass bridge, you could say.

Interesting challenges

It’s not all plain sailing. There are a few challenges along the way to collecting this data which anyone who wanted to collect this kind of information would face. By collecting it in a central place and running an open platform we can solve each problem once, and improve our process as a community.

One problem is choosing what to include. We include any link that we find from a non-publisher website. That means that invariably some of the links are from spam. This problem isn’t new: we see low-quality articles being published in traditional journals from time to time. We try to include all of the data we can find and pass it onto consumers. They might want to whitelist certain sources, or they may want all of the data because they’re trying to study scholarly spam. We have decided to provide data as Events, which strike the balance between atomicity and usefulness.

Another, which I talked about at last year’s PIDapalooza, is how we track article landing pages. Read the blog post, the user guide or hop in a time machine if you’re interested.

The thing about bridges…

… is that they help people get where they’re going. With a few notable exceptions, they’re not the main attraction. We play a humble part in scholarly publishing, helping collect and distribute metadata. Most of what we do goes unseen, and helps people create tools, platforms and research. Event Data is an API, and whilst we hope people will build all kinds of things with it, including altmetrics tools, we’re not making another metric.

PIDapalooza

All of which brings me to my talk, which I’m giving on Wednesday: Bridging persistent and not-so-persistent identifiers. I would tell you about it, but there isn’t much more left to say.

If you want to find out more, we’re currently in Beta, and open for business. Head over to the User Guide to get started!

Crossref ambassador program

Vanessa Fairhurst — Thu, 04 Jan 2018 00:00:00 +0000

We have listened to the feedback from you, our members, and you’ve told us of a need for local experts to provide support in your timezone and language, and to act as liaisons with the Crossref team. You’ve also asked for an increased number of training events both online and in person close to you, and for more representatives from Crossref at regional industry events.

We want to make sure we can reach members around the globe, and as such, a wide team of people is required who are knowledgeable in the languages, cultures, and member needs in a variety of countries. This is why we’re launching our Ambassador Program.

What are Crossref Ambassadors?

Crossref Ambassadors are volunteers who work within the international scholarly research community in a variety of different roles such as librarians, researchers or editors to name but a few. They are individuals who are well connected, value the work that Crossref does and are passionate about improving scholarly communication and the role Crossref plays within this system.

Some of the activities our ambassadors will undertake:

Staying up-to-speed with Crossref developments, for example, by attending webinars and maintaining regular check-ins with the Crossref team.
Engaging in the online community platform; providing feedback, joining in discussions and helping other members to resolve issues posted to the group.
Writing blog posts, or contributing to newsletters.
Participating in beta-testing of new products and services.
Helping with local LIVE events; for example, providing recommendations on speakers or venues, helping with logistics and presenting at the event.
Helping with the translation of Crossref material and content into local languages.
Running webinars on different Crossref services in local languages.
Running training sessions locally with Crossref members
Representing Crossref at relevant industry events

It is important that our ambassadors enjoy the work they are doing with Crossref by contributing in ways in which they feel comfortable, according to their interests, skills and the time they feel they want to contribute. For this reason, the role comes with a high degree of flexibility.

We see our ambassadors as valued members of the Crossref network and will provide them with:

A dedicated contact for any upcoming news, or to share ideas, queries or concerns.
Help with content for proposal calls, presentations, training and written articles.
Crossref materials and giveaways (plus ambassador-branded materials).
Personal endorsement via Crossref
Training on Crossref services and on wider relevant skills as necessary.
First look at new Crossref developments
Certification from Crossref on ambassador and training status.
Personal ambassador logo or badge for use on email, website and profile on the Crossref online community forum (launching later this year).

Crossref Ambassadors will become an increasingly key part of the Crossref community - the first port of call for updates or to test out new products or services, and the eyes and ears within the local academic community - working closely with Crossref to make scholarly communications better for all.

Meet our first ambassadors!

Jae Hwa Chang has been working at infoLumi as a manuscript editor in academic journals since 2010. Prior to joining infoLumi, she was a medical librarian at International Vaccine Institute and was engaged in medical information management and service. Her interests in information control and management started when she was doing work indexing newspaper articles at JoonAng Ilbo. She was fascinated by Crossref’s persistent efforts and contribution in developing new services to “make content easy to find, cite, link, and assess” and has been introducing them to Korean scholarly publishing communities. Jae earned her MA in Library and Information Science from Ewha Womans University, Korea. She serves as a vice chair of the Committee on Planning and Administration at the Korean Council of Science Editors. In her spare time, she enjoys traveling and experiencing new cultures.

장재화는 2010년부터 인포루미에서 의학학술지 원고편집을 담당하고 있다. 그전에는 국제백신연구소 도서관에서 사서로 일하면서 의학정보와 학술지논문 유통에 관심을 가졌으며, 그에 앞서서는 중앙일보에서 신문기사 DB 색인을 하면서 정보관리와 활용에 대해 연구하였다. 정보의 검색, 평가, 활용을 위해 꾸준히 새로운 서비스를 개발하는 Crossref에 매력을 느꼈고, 그 서비스들을 한국의 학술지 출판 관계자들에게 소개해왔다. 이화여자대학교에서 문헌정보학을 전공하였고, 한국과학학술지편집인협의회 기획운영위원회 부위원장을 맡고 있다. 여행과 다양한 문화 체험을 즐긴다.

Edilson Demasio has been a librarian since 1995, with PhD. in Information Science at Federal University of Rio de Janeiro-UFRJ/IBICT. He works in the Department of Mathematics Library of State University of Maringá-UEM, Brazil. With 20 years’ experience in scientific metadata and publishing. His expertise is various including knowledge in scientific communication, Crossref services, research integrity, misconduct prevention in science, publishing on Latin America, biomedical information, OJS-Open Journal Systems, Open Access journals, scientific journals quality and indexing, and scientific bibliographical databases. He is enthusiastic about presenting and disseminating information about Crossref services to his community in Brazil and working within the community, exchanging ideas and experience.

Eu sou bibliotecário desde 1995, Doutor em Ciência da Informação pela Universidade Federal do Rio de Janeiro-UFRJ/convênio IBICT. Eu trabalho na Biblioteca do Departamento de Matemática da Universidade Estadual de Maringá-UEM. Com 20 anos de experiência em metadados científicos e editoração, entre outros. Meus conhecimentos são diversos sobre comunicação científica, cientometria, metadados XML, serviços Crossref, integridade em pesquisa, prevenção de más condutas na ciência, editoração, editoração na América Latina, informação biomédica, OJS-Open Journal Systems, revistas de Acesso Aberto, qualidade de periódicos científicos e indexação, bases de dados bibliográficas. Gosto de disseminar meu conhecimento a outras regiões e pessoas e de trabalhar em comunidade junto as instituições e outros países, de planejar novas apresentações, de trocar experiências como palestrante ou convidado e trabalhar na disseminação do conhecimento para todos.

Lauren Lissaris has dedicated much of her career to the dissemination of valuable content on a robust platform. She takes pride in her achievements as the Digital Content Manager at JSTOR. JSTOR provides access to more than 10 million academic journal articles, books, and primary sources in 75 disciplines. JSTOR is part of ITHAKA, a not-for-profit organisation helping the academic community use digital technologies to preserve the scholarly record and to advance research and teaching in sustainable ways.

Lauren successfully works with all aspects of journal content to effectively assist publishers with their digital content. This includes everything from XML markup, Content Registration/multiple resolution, and HTML website updates. Lauren has been involved in hosting current content on JSTOR since the program’s launch in 2010. She continues to collaborate with organisations to successfully contribute to the evolution of digital content. The natural spread from journals to books has set Lauren up for developing and planning the book Content Registration program for JSTOR. She is a member of the Crossref Books Advisory Group and she helped successfully pilot Crossref’s new Co-access book deposit feature.

If you want to find out more information on the Ambassador Program, or you would like to express your interest in being an ambassador, you can either contact us at [feedback@crossref.org](mailto:feedback@crossref.org?subject=Ambassador Program) or complete our online form.