how to wget or curl from a download button with no attached content URL?

user_3696204 0
Viewed 859

I'm trying to download some files from the ICEWS dataset at https://dataverse.harvard.edu/dataverse/icews. I'm trying to do it on a Google VM instance because I am on a slow connection with data limits that are expensive to break. I have tried both curl and wget without success. For any given file I'd like to download, there is a download button with a URL attached to it, for example:

http://dataverse.harvard.edu/file.xhtml?persistentId=doi:10.7910/DVN/28075/K7L9Y8#

But this URL appears to only be a link to the location of the button on the current page. Using this URL with either wget or curl results in the download of the HTML of the current page rather than the download of the file content. I can find no direct link to the content itself, and also cannot find a way to print out the redirect chain that results in clicking the download button.

What is a good solution to this problem? Command-line would be preferred, but if there is some GCP specific workaround, that would be good too. I'm a bit of a newb at both, so simple is better.

wget curl
2 Answers

Taking a look at this link it'll explain how to use wget to download a package. Going to the link you've provided here, in the 'metadata Tab' I copied the 'Download URL' >> used wget to download the file:

wget https://dataverse.harvard.edu/api/access/datafile/:persistentId?persistentId=doi:10.7910/DVN/28075/K7L9Y8

and was able to see the the data just fine:

:persistentId?persistentId=doi:10.7910%2FDVN%2F28075% 100%[======================================================================================================================>] 221.56M  4.02MB/s    in 84s     

2020-10-05 14:36:43 (2.62 MB/s) - ‘:persistentId?persistentId=doi:10.7910%2FDVN%2F28075%2FK7L9Y8.1’ saved [232325743/232325743]

th@instance-667:~$ ls
':persistentId?persistentId=doi:10.7910%2FDVN%2F28075%2FK7L9Y8'  ':persistentId?persistentId=doi:10.7910%2FDVN%2F28075%2FK7L9Y8.1'

th@instance-667:~$ cat ':persistentId?persistentId=doi:10.7910%2FDVN%2F28075%2FK7L9Y8.1'
Event ID        Event Date      Source Name    Source Sectors   Source Country  Event Text      CAMEO Code      Intensity       Target Name     Target Sectors  Target Country  Story ID        Sentence Number Publisher    City    District        Province       Country  Latitude        Longitude
20718170        2014-01-01      Police (Australia)      Police,Government       Australia

I think this is what you needed?

user_118661150

You can also use the Dataverse API to download a file (or a dataset), as long as you know its PID. Take a look at the "Basic File Access" section in the API documentation. Note that in Havard's Dataverse, this API endpoint is open an works without authentication.

How to get your file

Using curl, you can supply a persistent identifier (in this case the DOI) to access the file. Your file has the DOI 10.7910/DVN/28075/K7L9Y8 so using

curl -L https://dataverse.harvard.edu/api/access/datafile/:persistentId/\?persistentId\=doi:10.7910/DVN/28075/K7L9Y8 --output events.2014.20160121105408.tab

downloads your desired file.

Some interesting details about the file location

The link above (as well as the download link from the website) do not point directly to the data file. The file itself is stored in some Amazon S3 cloud storage. You can see this when run curl with the -v parameter (and without the -L):

curl -v https://dataverse.harvard.edu/api/access/datafile/:persistentId/\?persistentId\=doi:10.7910/DVN/28075/K7L9Y8
< HTTP/2 303 
< date: Fri, 19 Nov 2021 16:50:00 GMT
< content-length: 0
< location: https://dvn-cloud.s3.amazonaws.com/10.7910/DVN/28075/15264f0277f-8738b46ba751[...very long URL that I shortened]

The initial reply from Dataverse is HTTP 303 (See Other) and the location entry points to the actual location of the file. Handing curl the -L parameter tells it to follow this location link.

Downloading whole datasets

Note that in your case, you are downloading a single file from a dataset. To download the whole dataset, you need to use a different API endpoint (basically replacing datafile with dataset in the URL) and use the right DOI. For your dataset, the file and dataset DOIs are:

10.7910/DVN/28075/K7L9Y8  File DOI
10.7910/DVN/28075         Dataset DOI

In other words, files from a dataset get an additional suffix element that is appended to the Dataset DOI. To download the whole dataset (careful, its 1.5 GB) you would then use:

curl -L https://dataverse.harvard.edu/api/access/dataset/:persistentId\?persistentId\=doi:10.7910/DVN/4OYCQK -v --output ICEWS_Coded_Event_Data.zip
user_28627190
Related