I'm pretty much brand new to Python, but I'm looking to build a webscraping tool that will rip data from an HTML table online and print it into a CSV in the same format.
Here's a sample of the HTML table (it's enormous, so I'm going to provide only a few rows).
<div class="col-xs-12 tab-content">
<div id="historical-data" class="tab-pane active">
<div class="tab-header">
<h2 class="pull-left bottom-margin-2x">Historical data for Bitcoin</h2>
<div class="clear"></div>
<div class="row">
<div class="col-md-12">
<div class="pull-left">
<small>Currency in USD</small>
</div>
<div id="reportrange" class="pull-right">
<i class="glyphicon glyphicon-calendar fa fa-calendar"></i>
<span>Aug 16, 2017 - Sep 15, 2017</span> <b class="caret"></b>
</div>
</div>
</div>
<table class="table">
<thead>
<tr>
<th class="text-left">Date</th>
<th class="text-right">Open</th>
<th class="text-right">High</th>
<th class="text-right">Low</th>
<th class="text-right">Close</th>
<th class="text-right">Volume</th>
<th class="text-right">Market Cap</th>
</tr>
</thead>
<tbody>
<tr class="text-right">
<td class="text-left">Sep 14, 2017</td>
<td>3875.37</td>
<td>3920.60</td>
<td>3153.86</td>
<td>3154.95</td>
<td>2,716,310,000</td>
<td>64,191,600,000</td>
</tr>
<tr class="text-right">
<td class="text-left">Sep 13, 2017</td>
<td>4131.98</td>
<td>4131.98</td>
<td>3789.92</td>
<td>3882.59</td>
<td>2,219,410,000</td>
<td>68,432,200,000</td>
</tr>
<tr class="text-right">
<td class="text-left">Sep 12, 2017</td>
<td>4168.88</td>
<td>4344.65</td>
<td>4085.22</td>
<td>4130.81</td>
<td>1,864,530,000</td>
<td>69,033,400,000</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
I'm particularly interested in recreating the table with the same column headers provided: "Date," "Open," "High," "Low," "Close," "Volume," "Market Cap." Currently, I've been able to write a simple script that will essentially go to the URL, download the HTML, parse with BeautifulSoup, and then use 'for' statements to get the td elements. Below a sample of my code (URL omitted) and the result:
from bs4 import BeautifulSoup
import requests
import pandas as pd
import csv
url = "enterURLhere"
page = requests.get(url)
pagetext = page.text
pricetable = {
"Date" : [],
"Open" : [],
"High" : [],
"Low" : [],
"Close" : [],
"Volume" : [],
"Market Cap" : []
}
soup = BeautifulSoup(pagetext, 'html.parser')
file = open("test.csv", 'w')
for row in soup.find_all('tr'):
for col in row.find_all('td'):
print(col.text)
Anyone have any pointers on how to at least reformat the data pull into the table? Thanks.