Show chuncked list of files in folders with PROPFIND

user_739897 0
Viewed 891

We're using PROPFIND request to get list of all files in specific folder.

curl --location --request PROPFIND 'https://example.com/some_folder' \
    --header 'Authorization: Basic eHh4Onl5eQ=='

But in some folders we have about 1M files and request timeout. Is there any way to set start position and file count limit per request?

webdav curl
2 Answers

RFC6578 could be used for paging, but I would bet that support for this is somewhat rare.

user_809110

resuming incomplete downloads really isn't curl's thing, it's wget's thing (curl does not have native support for automatically retrying/resuming incomplete downloads, but wget does support it. to do the same thing in curl, you would need some scripting language alongside curl, like php/python/whatever)

by default curl downloads to stdout, and wget downloads to files, to get wget to download to stdout, add the argument -O- , wget does curl's equivalent of --location by default, so no translation needed there, --request PROPFIND translates to --method=PROPFIND, and --header 'Authorization: Basic eHh4Onl5eQ==' roughly translates to --auth-no-challenge --http-user='xxx' --http-password='yyy' , putting it all together we get

wget --tries=10 -O- --method=PROPFIND --auth-no-challenge --http-user='xxx' --http-password='yyy' 'https://example.com/some_folder'

which should automatically resume the download up to 10 times before giving up, which you can change with the --tries=10 argument

for completeness, here's the request wget will send with the above invocation:

PROPFIND /some_folder HTTP/1.1
User-Agent: Wget/1.19.1 (cygwin)
Accept: */*
Accept-Encoding: identity
Authorization: Basic eHh4Onl5eQ==
Host: example.com
Connection: Keep-Alive

Is there any way to set start position and file count limit per request?

.. having skimmed RFC1918 and RFC5689, i don't think so; it's HTTP so you can issue HTTP Range requests to download only parts of the list, but then you won't be able to parse it as well-formatted XML, you'll have to fuzzy-parse it like browsers parse HTML.. (PS libxml2 has good support for parsing broken XML, and PHP has great bindings for libxml2, probably wouldn't be difficult to fuzzy-parse with PHP's DOMDocument::loadHTML() & co)

user_10670030
Related