Text and data mining for members
Even if you already have an API, ours provides additional benefits as it’s a common, standards-based API that works across all members and records. Researchers having to learn many different member APIs for TDM projects doesn’t scale well.
It is up to you to decide formats for your full-text in: some offer PDF, others XML, and some plain text. Some members vary what they deliver depending on the age of the content or other variables. Our API does not provide automatic access to subscription content - access to subscription content is managed on your site using your existing access control systems.
As a member, you need to do two things to enable text and data mining for the metadata records that you have registered with us:
- Include the link to full-text in the metadata for each DOI so researchers can follow it to access your content
- Include a license URL in the metadata for each DOI so researchers can use this to find out if they have permission to carry out TDM with your content item
Register this information with us using a resource-only deposit or by uploading a .csv file containing the URLs and the related DOIs.
If you are concerned about the impact of automated TDM harvesters on your site performance, you may choose to implement rate-limiting headers.
Rate limiting
TDM may change the volume of traffic that your servers have to handle when researchers download large numbers of files in bulk. You can mitigate performance issues with rate limiting.
We have defined a set of standard HTTPS headers that can be used by servers to convey rate-limiting information to automated text and data mining tools. Well-behaved TDM tools can simply look for these headers when they query member sites in order to understand how to behave so as not to affect the site’s performance. The headers allow a member to define a rate limit window - a time span, such as a minute, an hour, or a day. The member can then specify:
| Header name | Example value | Explanation |
|---|
| CR-TDM-Rate-Limit | 1500 | Maximum number of full-text downloads that are allowed to be performed in the defined rate limit window |
| CR-TDM-Rate-Limit-Remaining | 76 | Number of downloads left for the current rate limit window |
| CR-TDM-Rate-Limit-Reset | 1378072800 | Remaining time (in UTC epoch seconds) before the rate limit resets and a new rate limit window is started |
We do not provide or enforce this rate limiting - it’s up to you to implement it if required, and to define a rate limit appropriate for your servers.
Example member site
We have created TinyPub to show an implementation of our API, including rate limiting and IP-based subscription access. You can download this code for reference, but please note that it’s just to illustrate the workings of the system, and is not intended for production.
