How to scrape values of a tags attribute?

Brad Cunningham 0
Viewed 50

I am trying to extract the data located in the labels area of this tag and I am not quite sure how to extract it. I am new to web scraping and I am using Scrapy in the shell.

<object id="svg_chart" class="map-svg uk-animation-fade" type="image/svg+xml" data="https://services.dat.com/svg/graph.svg?showAllLabels=true&amp;vgrid=true&amp;lineWidth=4&amp;op_min=75&amp;minYValue=0&amp;maxYValue=10&amp;yStep=2&amp;title=Van%20Load-to-Truck%20Ratio&amp;labels=Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec&amp;colors=_F89929,_57CAF1,_B686BB&amp;ds_3=2020|2.23,1.84,2.89,0.98,1.91,3.52,4.40,5.31,5.45,4.33,4.49,4.84&amp;ds_2=2021|4.27,7.54,5.78,4.79,6.12,5.56,5.81,6.46,6.32,5.59,5.19,6.54&amp;ds_1=2022|9.34,7.33,4.57,3.42,4.39,3.88,3.84,3.54,null,null,null,null&amp;copyright=%40%202022%20DAT%20Freight%20%26%20Analytics" width="" height=""></object>

I was able to extract the main tag into the response and I have tried to use .get() and .extract() to get at the data. I have not tried .re(), so if this is the answer then apologies.

scrapy web-scraping python
1 Answers

Try bs4

    from bs4 import BeautifulSoup
    import lxml
    
    mystr = '<object id="svg_chart" class="map-svg uk-animation-fade" type="image/svg+xml" data="https://services.dat.com/svg/graph.svg?showAllLabels=true&amp;vgrid=true&amp;lineWidth=4&amp;op_min=75&amp;minYValue=0&amp;maxYValue=10&amp;yStep=2&amp;title=Van%20Load-to-Truck%20Ratio&amp;labels=Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec&amp;colors=_F89929,_57CAF1,_B686BB&amp;ds_3=2020|2.23,1.84,2.89,0.98,1.91,3.52,4.40,5.31,5.45,4.33,4.49,4.84&amp;ds_2=2021|4.27,7.54,5.78,4.79,6.12,5.56,5.81,6.46,6.32,5.59,5.19,6.54&amp;ds_1=2022|9.34,7.33,4.57,3.42,4.39,3.88,3.84,3.54,null,null,null,null&amp;copyright=%40%202022%20DAT%20Freight%20%26%20Analytics" width="" height=""></object>'
    soup = BeautifulSoup(mystr, 'lxml')
    
    obj = soup.find('object')
    
    print(obj['id'])
    print(obj['class'])
    print(obj['type'])
    print(obj['data'])

... result

'''
svg_chart
['map-svg', 'uk-animation-fade']
image/svg+xml
https://services.dat.com/svg/graph.svg?showAllLabels=true&vgrid=true&lineWidth=4&op_min=75&minYValue=0&maxYValue=10&yStep=2&title=Van%20Load-to-Truck%20Ratio&labels=Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec&colors=_F89929,_57CAF1,_B686BB&ds_3=2020|2.23,1.84,2.89,0.98,1.91,3.52,4.40,5.31,5.45,4.33,4.49,4.84&ds_2=2021|4.27,7.54,5.78,4.79,6.12,5.56,5.81,6.46,6.32,5.59,5.19,6.54&ds_1=2022|9.34,7.33,4.57,3.42,4.39,3.88,3.84,3.54,null,null,null,null&copyright=%40%202022%20DAT%20Freight%20%26%20Analytics
'''

This is because obj.attrs is a dictionary and you can do whatever you want with it:

print(type(obj.attrs))
#  result
#  <class 'dict'>

#  You can get all of the attrs like this too
for k, v in obj.attrs.items():
    print(k + ' == ' + str(v))

#  result
'''
id == svg_chart
class == ['map-svg', 'uk-animation-fade']
type == image/svg+xml
data == https://services.dat.com/svg/graph.svg?showAllLabels=true&vgrid=true&lineWidth=4&op_min=75&minYValue=0&maxYValue=10&yStep=2&title=Van%20Load-to-Truck%20Ratio&labels=Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec&colors=_F89929,_57CAF1,_B686BB&ds_3=2020|2.23,1.84,2.89,0.98,1.91,3.52,4.40,5.31,5.45,4.33,4.49,4.84&ds_2=2021|4.27,7.54,5.78,4.79,6.12,5.56,5.81,6.46,6.32,5.59,5.19,6.54&ds_1=2022|9.34,7.33,4.57,3.42,4.39,3.88,3.84,3.54,null,null,null,null&copyright=%40%202022%20DAT%20Freight%20%26%20Analytics
width ==
height ==
'''
d r0
Related