2014 on Crossref

Linking data and publications

Geoffrey Bilder — Sun, 21 Sep 2014 00:00:00 +0000

Do you want to see if a Crossref DOI (typically assigned to publications) refers to DataCite DOIs (typically assigned to data)? Here you go:

https://web.archive.org/web/20150121025249/http://api.labs.crossref.org/graph/doi/10.4319/lo.1997.42.1.0001

Conversely, do you want to see if a DataCite DOI refers to Crossref DOIs? Voilà:

https://web.archive.org/web/20150321190744/http://api.labs.crossref.org/graph/doi/10.1594/pangaea.185321

Background

“How can we effectively integrate data into the scholarly record?” This is the question that has, for the past few years, generated an unprecedented amount of handwringing on the part researchers, librarians, funders and publishers. Indeed, this week I am in Amsterdam to attend the 4th RDA plenary in which this topic will no doubt again garner a lot of deserved attention.

We hope that the small example above will help push the RDAs agenda a little further. Like the recent ODIN project, It illustrates how we can simply combine two existing scholarly infrastructure systems to build important new functionality for integrating research objects into the scholarly literature.

Does it solve all of the problems associated with citing and referring to data? Can the various workgroups at RDA just cancel their data citation sessions and spend the week riding bikes and gorging on croquettes? Of course not. But my guess is that by simply integrating DataCite and Crossref in this way, we can make a giant push in the right direction.

There are certainly going to be differences between traditional citation and data citation. Some even claim that citing data isn’t “as simple as citing traditional literature.” But this is a caricature of traditional citation. If you believe this, go off an peruse the MLA, Chicago, Harvard, NLM and APA citation guides. Then read Anthony Grafton’s, The Footnote? Are you back yet? Good, so let’s continue…

Citation of any sort is a complex issue- full of subtleties, edge-cases exceptions, disciplinary variations and kludges. Historically, the way to deal with these edge-cases has been social, not technical. For traditional literature we have simply evolved and documented citation practices which generally make contextually-appropriate use of the same technical infrastructure (footnotes, endnotes, metadata, etc.). I suspect the same will be true in citing data. The solutions will not be technical, they will mostly be social. Researchers, and publishers will evolve new, contextually appropriate mechanisms to use existing infrastructure deal with the peculiarities of data citation.

Does this mean that we will never have to develop new systems to handle data citation? Possibly But I don’t think we’ll know what those systems are or how they should work until we’ve actually had researchers attempting to use and adapt the tools we have.

Technical background

About five years ago, Crossref and DataCite explored the possibility of exposing linkages between DataCite and Crossref DOIs. Accordingly, we spent some time trying to assemble an example corpus that would illustrate the power of interlinking these identifiers. We encountered a slight problem. We could hardly find any examples. At that time, virtually nobody cited data with DataCite DOIs and, if they did, the Crossref system did not handle them properly. We had to sit back and wait a while.

And now the situation has changed.

This demonstrator harvests DataCite DOIs using their OAI-PMH API and links them in a graph database with Crossref DOIs. We have exposed this functionality on the “labs” (i.e. experimental) version of our REST API as a graph resource. So…

Now you can get a list of Crossref DOIs that refer to DataCite DOIs using Event Data.

Caveats and Weasel Words

We have not finished indexing all the links.
The API is currently a very early labs project. It is about as reliable as a devolution promise from Westminster.
The API is run on a pair of raspberry-pi’s connected to the internet via bluetooth.
It is not fast.
The representation and the API is under active development.Things will change. Watch the Crossref Labs site for updates on this collaboration with DataCite

Citation needed

Geoffrey Bilder — Thu, 07 Aug 2014 00:00:00 +0000

Remember when I said that the Wikipedia was the 8th largest referrer of DOI links to published research? This despite only a fraction of eligible references in the free encyclopaedia using DOIs.

We aim to fix that. Crossref and Wikimedia are launching a new initiative to better integrate scholarly literature in the world’s largest public knowledge space, Wikipedia.

This work will help promote standard links to scholarly references within Wikipedia, which persist over time by ensuring consistent use of DOIs and other citation identifiers in Wikipedia references. Crossref will support the development and maintenance of Wikipedia’s citation tools on Wikipedia. This work will include bug fixes and performance improvements for existing tools, extending the tools to enable Wikipedia contributors to more easily look up and insert DOIs, and providing a “linkback” mechanism that alerts relevant parties when a persistent identifier is used in a Wikipedia reference.

In addition, Crossref is creating the role of Wikimedia Ambassador (modeled after Wikimedian-in-Residence) to act as liaison with the Wikimedia community, promote use of scholarly references on Wikipedia, and educate about DOIs and other scholarly identifiers (ORCIDs, PubMed IDs, DataCite DOIs, etc) across Wikimedia projects.

Starting today, Crossref will be working with Daniel Mietchen to coordinate Crossref’s Wikimedia-related activities. Daniel’s team will be composed of Max Klein and Matt Senate, who will work to enhance Wikimedia citation tools, and will share the role of Wikipedia ambassador with Dorothy Howard.

Since the beginnings of Wikipedia, Daniel Mietchen has worked to integrate scholarly content into Wikimedia projects. He is part of an impressive community of active Wikipedians and developers who have worked extensively on linking Wikipedia articles to the formal literature and other scholarly resources. We’ve been talking to him about this project for nearly a year, and are happy to finally get it off the ground.

-G

]7 Matt, Max and Daniel at #wikimania2014. Photo by Dorothy.

wikimania2014

♫ Researchers just wanna have funds ♫

Geoffrey Bilder — Thu, 10 Apr 2014 00:00:00 +0000

photo credit

Summary

You can use a new Crossref API to query all sorts of interesting things about who funded the research behind the content Crossref members publish.

Background

Back in May 2013 we launched Crossref’s FundRef service. It can be summarized like this:

Crossref keeps and manages a canonical list of Funder Names (ephemeral) and associated identifiers (persistent).
We encourage our members (or anybody, really- the list is available under A CC-Zero license waiver) to use this list for collecting information on who funded the research behind the content that our members publish.
We then ask that our members deposit this data in their normal Crossref metadata deposits.

And that was cool.

But then people started asking us awkward questions. Questions like “what can I do with the funder data?” and “how do I query it?”.

Stoopit people.

Can’t you just let us bask for a few minutes in the sunny glow of actually conceiving of and launching a project within a year?

But seriously, funders, were interested to see how they could use the funder metadata being collected in Crossref. In particular, some funding agencies were interested in being able to measure Key Performance Indicators (“KPIs” to management wonks) related to recent mandates such as the February 22nd 2013 OSTP memo, Public Access to the Results of Federally Funded Research. Two groups also approached us, CHORUS and SHARE. Both are interested in exploring how to build reporting tools for funders, institutions and researchers and each brought us a gigantic hairball of use-cases they were hoping we would be able to meet.

Conveniently, we were in the process of creating a revised, modern Crossref API that is entirely buzzword-compliant, and so we set to work…

We thought people might be interested in seeing what you can do with the Crossref REST API in relation to funding information and the expectations that are increasingly being attached to them. CHORUS is already using the Crossref REST API heavily and we expect that SHARE will soon start making use of it as well. The feedback from both groups has been very useful, but we are looking for broader feedback as well. The API is still in development, so now is your chance to help us shape it.

Brief Examples

Please note, the following are APIs calls, although you can copy and paste the URIs into your browser, the data is returned in a machine readable representation called JSON. If you want the results to look a little more presentable, we advise you install the JSONView plugin:

Firefox Users: JSONView
Chrome Users: JSONView

Also note that publishers have only just started to deposit the metadata needed for these APIs to work, so the data is currently sparse. We know that many of our members are working feverishly to populate more of the needed metadata, but this requires updates to the their manuscript tracking systems, production systems and hosting systems. It takes time.

But for now you can paste the relevant URIs below into your browser and see the results that we do have. Expect these numbers to increase sharply over the next few months

To start with, you might want to know how many articles in Crossref have FundRef metadata:

https://api.crossref.org/v1/works?filter=has-funder:true&rows=0

You could then be interested in knowing how many works in Crossref use FundRef to credit the United States’ National Science Foundation (NSF) for funding their research? First you need to find out what the FundRef identifier is for the NSF:

https://api.crossref.org/v1/funders?query=NSF

You can see that there are several entries that match “NSF”, and that the one we are looking for has the identifier http://dx.doi.org/10.13039/100000001. Remember, funding agency names can change frequently, the ID provides a persistent link to the funder even if their name changes.

If you are curious, you can see the details for the NSF entry, including its location, parent and child organisations:

https://api.crossref.org/v1/funders/10.13039/100000001

Notice that the results also lists the work-count. This is the number of works in the Crossref metadata that list the US NSF as having funded the research.

So perhaps you would like to see the list of works. The following will list the first twenty:

https://api.crossref.org/v1/funders/10.13039/100000001/works

You can page through the results with the offset argument:

https://api.crossref.org/v1/funders/10.13039/100000001/works?offset=20
https://api.crossref.org/v1/funders/10.13039/100000001/works?offset=40
...

How many works that have listed the NSF as a funder have license information:

https://api.crossref.org/v1/funders/10.13039/100000001/works?filter=has-license:true&rows=0

Lets see the first batch that have license information:

https://api.crossref.org/v1/funders/10.13039/100000001/works?filter=has-license:true

Lets look at the metadata for one of the DOIs returned:

https://api.crossref.org/v1/works/10.1063/1.3593378

Interesting, the metadata shows an article published by AIP. It includes license information (CC-BY 3.0) as well as a link to the full text. If you follow the link to the full text, you can retrieve it:

http://link.aip.org/link/applab/v98/i21/p216101/pdf/CHORUS

Wow- A pretty short article. But you can see that it does credit the NSF and that the award number recorded in the text is the same as the award number recorded in the FundRef section of the Crossref metadata. Yay.

You can see in the brief examples above that there is a lot of other metadata you may want to query on and explore. It can include ORCIDS, information about archiving arrangements- even abstracts. It all depends on what the Crossref member has decided to provide.

You can get a simple overview of what a Crossref member has provided by looking at a member summary. Here is an example for Hindawi:

https://api.crossref.org/v1/members?query=hindawi

Note again that names are fickle, so the above query can also be accomplished using the member identifier like this:

https://api.crossref.org/v1/members/98

Groovy init?

If you want more pointers on where you can learn how to use the API, read on…

The Future of the Crossref REST API.

Our aim is for the Crossref REST API to go into production this Summer (2014). As with most of our newer APIs, there will be a free API for public use and a paid for API for professional use. The only difference between the two will be that the professional version will come with a service level agreement (SLA) covering uptime, response time and support. Naturally, this also means that the professional one will be on dedicated hosting equipment so that we can meet these SLAs, whereas the performance of the free version will be subject to the vicissitudes inherent in using a shared, constrained resource (i.e. the server and network it is running on).

Again, the basics of the API are in place. It should be fairly stable, but we do reserve the right to make changes to it over the next few months. Please send us feedback.

— The Weasel

Many Metrics. Such Data. Wow.

Geoffrey Bilder — Mon, 24 Feb 2014 00:00:00 +0000

[

Crossref Labs loves to be the last to jump on an internet trend, so what better than than to combine the Doge meme with altmetrics?

Note: The API calls below have been superceeded with the development of the Event Data project. See the latest API documentation for equivalent functionality

Want to know how many times a Crossref DOI is cited by the Wikipedia?

http://det.labs.crossref.org/works/doi/10.1371/journal.pone.0086859

Or how many times one has been mentioned in Europe PubMed Central?

http://det.labs.crossref.org/works/doi/10.1016/j.neuropsychologia.2013.10.021

Or DataCite?

http://det.labs.crossref.org/works/doi/10.1111/jeb.12289

Background

Back in 2011 PLOS released its awesome ALM system as open source software (OSS). At Crossref Labs, we thought it might be interesting to see what would happen if we ran our own instance of the system and loaded it up with a few Crossref DOIs. So we did. And the code fell over. Oops. Somehow it didn’t like dealing with 10 million DOIs. Funny that.

But the beauty of OSS is that we were able to work with PLOS to scale the code to handle our volume of data. Crossref contracted with Cottage Labs and we both worked with PLOS to make changes to the system. These eventually got fed back into the main ALM source on Github. Now everybody benefits from our work. Yay for OSS.

So if you want to know technical details, skip to Details for Propellerheads. But if you want to know why we did this, and what we plan to do with it, read on.

Why?

There are (cough) some problems in our industry that we can best solve with shared infrastructure. When publishers first put scholarly content online, they used to make bilateral reference linking agreements. These agreements allowed them to link citations using each other’s proprietary reference linking APIs. But this system didn’t scale. It was too time-consuming to negotiate all the agreements needed to link to other publishers. And linking through many proprietary citation APIs was too complex and too fragile. So the industry founded Crossref to create a common, cross-publisher citation linking API. Crossref has since obviated the need for bilateral linking arrangements.

So-called altmetrics look like they might have similar characteristics. You have ~4000 Crossref member publishers and N sources (e.g. Twitter, Mendeley, Facebook, CiteULike, etc.) where people use (e.g. discuss, bookmark, annotate, etc.) scholarly publications. Publishers could conceivably each choose to run their own system to collect this information. But if they did, they would face the following problems:

The N sources will be volatile. New ones will emerge. Old ones will vanish.
Each publisher will need to deal with each source’s different APIs, rate limits, T&Cs, data licenses, etc. This is a logistical headache for both the publishers and for the sources.
If publishers use different systems which in turn look at different sources, it will be difficult to compare results across publishers.
If a journal moves from one publisher to another, then how are the metrics for that journal’s articles going to follow the journal? This isn’t a complete list, but it shows that there might be some virtue in publishers sharing an infrastructure for collecting this data. But what about commercial providers? Couldn’t they provide these ALM services? Of course - and some of them currently do. But normally they look on the actual collection of this data as a means to an end. The real value they provide is in the analysis, reporting and tools that they build on top of the data. Crossref has no interest in building front-ends to this data. If there is a role for us to play here, it is simply in the collection and distribution of the data.

No, really, WHY?

Aren’t these altmetrics an ill-conceived and meretricious idea? By providing this kind of information, isn’t Crossref just encouraging feckless, neoliberal university administrators to hasten academia’s slide into a Stakhanovite dystopia? Can’t these systems be gamed?

FOR THE LOVE OF FSM, WHY IS CROSSREF DABBLING IN SOMETHING OF SUCH QUESTIONABLE VALUE?

takes deep breath. wipes spittle from beard

These are all serious concerns. Goodhart’s Law and all that… If a university’s appointments and promotion committee is largely swayed by Impact Factor, it won’t improve a thing if they substitute or supplement Impact Factor with altmetrics. Amy Brand has repeatedly pointed out, the best institutions simply don’t use metrics this way at all (PowerPoint presentation). They know better.

But yes, it is still likely that some powerful people will come to lazy conclusions based on altmetrics. And following that, other lazy, unscrupulous and opportunistic people will attempt to game said metrics. We may even see an industry emerge to exploit this mess and provide the scholarly equivalent of SEO. Feh. Now I’m depressed and I need a drink.

So again, why is Crossref doing this? Though we have our doubts about how effective altmetrics will be in evaluating the quality of content, we do believe that they are a useful tool for understanding how scholarly content is used and interpreted. The most eloquent arguments against altmetrics for measuring quality, inadvertently make the case for altmetrics as a tool for monitoring attention.

Critics of altmetrics point out that much of the attention that research receives outside of formal scholarly communications channels can be ascribed to:

Puffery. Researchers and/or university/publisher “PR wonks” over-promoting research results.
Innocent misinterpretation. A lay audience simply doesn’t understand the research results.
Deliberate misinterpretation. Ideologues misrepresent research results to support their agendas.
Salaciousness. The research appears to be about sex, drugs, crime, video games or other popular bogeymen.
Neurobollocks. A category unto itself these days.

In short, scholarly research might be misinterpreted. Shock horror. Ban all metrics. Whew. That won’t happen again.

Scholarly research has always been discussed outside of formal scholarly venues. Both by scholars themselves and by interested laity. Sometimes these discussions advance the scientific cause. Sometimes they undermine it. The University of Utah didn’t depend on widespread Internet access or social networks to promote yet-to-be peer-reviewed claims about cold fusion. That was just old-fashioned analogue puffery. And the Internet played no role in the Laetrile or DMSO crazes of the 1980s. You see, there were once these things called “newspapers.” And another thing called “television.” And a sophisticated meatspace-based social network called a “town square.”

But there are critical differences between then and now. As citizens get more access to the scholarly literature, it is far more likely that research is going to be discussed outside of formal scholarly venues. Now we can build tools to help researchers track these discussions. Now researchers can, if they need to, engage in the conversations as well. One would think that conscientious researchers would see it as their responsibility to remain engaged, to know how their research is being used. And especially to know when it is being misused.

That isn’t to say that we expect researchers will welcome this task. We are no Pollyannas. Researchers are already famously overstretched. They barely have time to keep up with the formally published literature. It seems cruel to expect them to keep up with the firehose of the Internet as well.

Which gets us back to the value of altmetrics tools. Our hope is that, as altmetrics tools evolve, they will provide publishers and researchers with an efficient mechanism for monitoring the use of their content in non-traditional venues. Just in the way that citations were used before they were distorted into proxies for credit and kudos.

We don’t think altmetrics are there yet. Partly because some parties are still tantalized by the prospect of usurping one metric for another. But mostly because the entire field is still nascent. People don’t yet know how the information can be combined and used effectively. So we still make naive assumptions such as “link=like” and “more=better.” Surely it will eventually occur to somebody that, instead, there may be a connection between repeated headline-grabbing research and academic fraud. A neuroscientist might be interested in a tool that alerts them if the MRI scans in their research paper are being misinterpreted on the web to promote neurobollocks. An immunologist may want to know if their research is being misused by the anti-vaccination movement. Perhaps the real value in gathering this data will be seen when somebody builds tools to help researchers DETECT puffery, social-citation cabals, and misinterpretation of research results?

But Crossref won’t be building those tools. What we might be able to do is help others overcome another hurdle that blocks the development of more sophisticated tools; getting hold of the needed data in the first place. This is why we are dabbling in altmetrics.

Wikipedia is already the 8th largest referrer of Crossref DOIs. Note that this doesn’t just mean that the Wikipedia cites lots of Crossref DOIs, it means that people actually click on and follow those DOIs to the scholarly literature. As scholarly communication transcends traditional outlets and as the audience for scholarly research broadens, we think that it will be more important for publishers and researcher to be aware of how their research is being discussed and used. They may even need to engage more with non-scholarly audiences. In order to do this, they need to be aware of the conversations. Crossref is providing this experimental data source in the hope that we can spur the development of more sophisticated tools for detecting and analyzing these conversations. Thankfully, this is an inexpensive experiment to conduct - largely thanks to the decision on the part of PLOS to open source its ALM code.

What Now?

Crossref’s instance of PLOS’s ALM code is an experiment. We mentioned that we had encountered scalability problems and that we had resolved some of them. But there are still big scalability issues to address. For example, assuming a response time of 1 second, if we wanted to poll the English-language version of the Wikipedia to see what had cited each of the 65 million DOIs held in Crossref, the process would take years to complete. But this is how the system is designed to work at the moment. It polls various source APIs to see if a particular DOI is “mentioned”. Parallelizing the queries might reduce the amount of time it takes to poll the Wikipedia, but it doesn’t reduce the work. Another obvious way in which we could improve the scalability of the system is to add a push mechanism to supplement the pull mechanism. Instead of going out and polling the Wikipedia 65 million times, we could establish a “scholarly linkback” mechanism that would allow third parties to alert us when DOIs and other scholarly identifiers are referenced (e.g. cited, bookmarked, shared). If the Wikipedia used this, then even in an extreme case scenario (i.e. everything in Wikipedia cites at least one Crossref DOI), this would mean that we would only need to process ~ 4 million trackbacks.

The other significant advantage of adding a push API is that it would take the burden off of Crossref to know what sources we want to poll. At the moment, if a new source comes online, we’d need to know about it and build a custom plugin to poll their data. This needlessly disadvantages new tools and services as it means that their data will not be gathered until they are big enough for us to pay attention to. If the service in question addresses a niche of the scholarly ecosystem, they may never become big enough. But if we allow sources to push data to us using a common infrastructure, then new sources do not need to wait for us to take notice before they can participate in the system.

Supporting (potentially) many new sources will raise another technical issue- tracking and maintaining the provenance of the data that we gather. The current ALM system does a pretty good job of keeping data, but if we ever want third parties to be able to rely on the system, we probably need to extend the provenance information so that the data is cheaply and easily auditable.

Perhaps the most important thing we want to learn from running this experimental ALM instance is: what it would take to run the system as a production service? What technical resources would it require? How could they be supported? And from this we hope to gain enough information to decide whether the service is worth running and, if so, by whom. Crossref is just one of several organisations that could run such a service, but it is not clear if it would be the best one. We hope that as we work with PLOS, our members and the rest of the scholarly community, we’ll get a better idea of how such a service should be governed and sustained.

Details for Propellerheads

Warning, Caveats and Weasel Words

The Crossref ALM instance is a Crossref Labs project. It is running on R&D equipment in a non-production environment administered by an orangutang on a diet of Redbulls and vodka.

So what is working?

The system has been initially loaded with 317,500+ Crossref DOIs representing publications from 2014. We will load more DOIs in reverse chronological order until we get bored or until the system falls over again.

We have activated the following sources:

PubMed

DataCite

PubMedCentral Europe Citations and Usage

We have data from the following sources but will need some work to achieve stability:

Facebook

Wikipedia

CiteULike

Twitter

Some of them are faster than others. Some are more temperamental than others. WordPress, for example, seems to go into a sulk and shut itself off after approximately 1,300 API calls.

In any case, we will be monitoring and tweaking the sources as we gather data. We will also add new sources as we get requested API keys. We will probably even create one or two new sources ourselves. Watch this blog and we’ll update you as we add/tweak sources.

Dammit, shut up already and tell me how to query stuff.

You can login to the Crossref ALM instance simply using a Mozilla Persona (yes, we’d eventually like to support ORCID too). Once logged-in, your account page will list an API key. Using the API key, you can do things like:

http://det.labs.crossref.org/api/v5/articles?ids=10.1038/nature12990

And you will see that (as of this writing), said Nature article has been cited by the Wikipedia article here:

https://en.wikipedia.org/wiki/HE0107-5240#cite_ref-Keller2014_4-0;

PLOS has provided lovely detailed instructions for using the API- So, please, play with the API and see what you make of it. On our side we will be looking at how we can improve performance and expand coverage. We don’t promise much- the logistics here are formidable. As we said above, once you start working with millions of documents, the polling process starts to hit API walls quickly. But that is all part of the experiment. We appreciate your helping us and would like your feedback. We can be contacted at:

2014 on Crossref

Linking data and publications

Background

Technical background

Caveats and Weasel Words

Citation needed

wikimania2014

♫ Researchers just wanna have funds ♫

Summary

Background

Brief Examples

More examples and documentation.

The Future of the Crossref REST API.

Many Metrics. Such Data. Wow.

Background

Why?

No, really, WHY?

What Now?

Details for Propellerheads

Warning, Caveats and Weasel Words

So what is working?

Dammit, shut up already and tell me how to query stuff.