Scraping from web site into HDFS

Question

I'm trying to scrap data from website into HDFS, at first it was working well the scraping, and then I added the line of storing data into HDFS it's not working:

import requests
from pathlib import Path
import os
from datetime import date
from hdfs import InsecureClient
date= date.today()
date

def downloadFile(link, destfolder):
    r = requests.get(link,stream=True)
    filename="datanew1"+ str(date)+".xls"
    downloaded_file = open(os.path.join(destfolder, filename), 'wb')
    client= InsecureClient('http://hdfs-namenode.default.svc.cluster.local:50070', user='hdfs')
    with client.download('/data/test.csv')
        for chunk in r.iter_content(chunk_size=256):
            if chunk:
                downloaded_file.write(chunk)
           
link="https://api.worldbank.org/v2/fr/indicator/FP.CPI.TOTL.ZG?downloadformat=excel"
Path('http://hdfs-namenode.default.svc.cluster.local:50070/data').mkdir(parents=True, exist_ok=True)
downloadFile(link, 'http://hdfs-namenode.default.svc.cluster.local:50070/data')

There is no error in the code, just I can't found the data scraped!

Scraping from web site into HDFS

0 Answers