I have been trying to use some code I found in this answer to recursively find all links from a given URL:
import urllib2
from bs4 import BeautifulSoup
url = "http://francaisauthentique.libsyn.com/"
def recursiveUrl(url,depth):
if depth == 5:
return url
else:
page=urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
newlink = soup.find('a') #find just the first one
if len(newlink) == 0:
return url
else:
return url, recursiveUrl(newlink,depth+1)
def getLinks(url):
page=urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
links = soup.find_all('a')
for link in links:
links.append(recursiveUrl(link,0))
return links
links = getLinks(url)
print(links)
and besides a warning
/usr/local/lib/python2.7/dist-packages/bs4/__init__.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.
The code that caused this warning is on line 28 of the file downloader.py. To get rid of this warning, change code that looks like this:
BeautifulSoup(YOUR_MARKUP})
to this:
BeautifulSoup(YOUR_MARKUP, "lxml")
I get the following error:
Traceback (most recent call last):
File "downloader.py", line 28, in <module>
links = getLinks(url)
File "downloader.py", line 25, in getLinks
links.append(recursiveUrl(link,0))
File "downloader.py", line 11, in recursiveUrl
page=urllib2.urlopen(url)
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 396, in open
protocol = req.get_type()
TypeError: 'NoneType' object is not callable
What is the problem?