Issue with downloading a file from a given URL and saving it using Google Colab

user_10532751 0
Viewed 1020

I have some URLs of some java files located in GitHub. I want to pass the URL to a method, download the file and save it with a different unique name.

This is the code I wrote and it has the issue mentioned below the code snippet.

!pip install wget

import wget
from datetime import datetime


def download_file(url):

   # Creating file name
   now_time =datetime.now()
   millisec = now_time.timestamp() * 10000
   millisec = str(millisec).split('.')[0]
   partial_name = url.split('/')[-1].split('.')[0]
   file_name = partial_name + millisec + '.java'

   # Download the file and save in colab location
   wget.download(url, file_name)

   return file_name

A sample URL I pass to this method is 'https://github.com/e32wong/CloCom/blob/master/CloneDigger.java'

The issue is, the downloaded java file's content appears as an HTML content full of tags. But, the original file on GitHub is a simple java file.

I want to save the file as it is. Any solution to this, please.

google-colaboratory wget file python
4 Answers

You are trying to download a Github page and not the source file. So it will be downloaded as an HTML page. You have to use a raw URL to download any source file from Github. So, if you want to write an automatic script to download the source file then you can first scrape the raw URL from the given GitHub page and then the raw URL can be passed as an argument to your method.

So, you can change the script as follows

import wget
from datetime import datetime
from bs4 import BeautifulSoup
import requests

HEADER = {
    "User-Agent": 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
}

def find_raw_url(url):

    response = requests.get(url, headers=HEADER)
    soup = BeautifulSoup(response.text, "html.parser")
    if soup is not None:
        rawUrl = soup.find(id='raw-url')
        if rawUrl is not None: 
            rawUrl = rawUrl['href']
            if rawUrl is not None:
                return 'https://github.com' + rawUrl
            
    return None


def download_file(url):
    
   raw_url = find_raw_url(url)
   
   if raw_url is not None:
       # Creating file name
       now_time =datetime.now()
       millisec = now_time.timestamp() * 10000
       millisec = str(millisec).split('.')[0]
       partial_name = url.split('/')[-1].split('.')[0]
       file_name = partial_name + millisec + '.java'
    
       # Download the file and save in colab location
       wget.download(raw_url, file_name)
   else:
       print('Cannot find raw url to download from github')


download_file('https://github.com/e32wong/CloCom/blob/master/CloneDigger.java')

Thank you.

user_90890150

The problem is with the URL. You must pass the raw URL of the file.

Page URL:'https://github.com/e32wong/CloCom/blob/master/CloneDigger.java'

Raw URL: 'https://raw.githubusercontent.com/e32wong/CloCom/master/CloneDigger.java'

You can easily modify the url by replacing it. Try this.

def download_file(url):

   # Creating file name
   now_time =datetime.now()
   millisec = now_time.timestamp() * 10000
   millisec = str(millisec).split('.')[0]
   partial_name = url.split('/')[-1].split('.')[0]
   file_name = partial_name + millisec + '.java'

   #Edit url as the raw file url of github
   url = url.replace('github.com', 'raw.githubusercontent.com', 1)
   url = url.replace('/blob/', '/',1)

   # Download the file and save in colab location
   
   wget.download(url, file_name)

   return file_name
user_157560970

Since you said the file is storing the HTML output, the problem may be you are giving the HTML link of the file instead of giving the raw file link. So, wget downloads the webpage and stores it. Instead, you should give the raw link, which you can find in the header section of a file as shown below - enter image description here

How do you know if it is a raw file or just an URL?

Well you just see the last part of the URL, if it is a link to the file it will end with the file extension eg - .jgp, .cpp, .py etc

Example of a raw link for an image - https://helpx.adobe.com/content/dam/help/en/photoshop/using/convert-color-image-black-white/jcr_content/main-pars/before_and_after/image-before/Landscape-Color.jpg

Notice it ends with .jpg

user_163613440

You can't access the actual content of the Java files programmatically through github.com URLs. It is meant for showing content to the end users through the browser (Github site). To get the raw content of a file stored in a Github repo, one must use "raw.githubusercontent.com" URLs.

The following is the refactoring of your download_file function. It downloads the file you want to a specific google drive folder.

import os
from google.colab import drive
 
drive.mount('/content/drive')

# notice the difference in the URL.
url = "https://raw.githubusercontent.com/e32wong/CloCom/master/CloneDigger.java"

def download_file(url):
  destination_path = "/content/drive/My Drive/code_files"
  os.chdir(destination_path)
  !wget "$url"

download_file(url)
user_88109410
Related