How to read the header content of a webpage in Django?

Goune Kenfack 0
Viewed 44

I created a search engine in Django and bs4 that scrapes search results from the Ask.com search engine. I would like when Django fetches search results from Ask, it checks the value of the X-Frame-Options header in order to give a value to my notAccept boolean depending on the result of the condition.

I took inspiration from this page of the Django documentation and also from this other page and after testing a proposed answer, I modified my code like this:

def search(request):
        . . .
        final_result = []
        for page_num in range(1, max_pages_to_scrap+1):
            url = "https://www.ask.com/web?q=" + search + "&qo=pagination&page=" + str(page_num)
            res = requests.get(url)
            soup = bs(res.text, 'lxml')
            result_listings = soup.find_all('div', {'class': 'PartialSearchResults-item'})

            for result in result_listings:
                result_title = result.find(class_='PartialSearchResults-item-title').text
                result_url = result.find('a').get('href')
                result_desc = result.find(class_='PartialSearchResults-item-abstract').text
                final_result.append((result_title, result_url, result_desc))

                for header in final_result[1]: #the error is generated here
                 response = requests.get(header)
                if response['X-Frame-Options'] in ["DENY", "SAMEORIGIN"]:
                    head = True
                    notAccept = bool(head)
                else:
                    notAccept = bool(False)

But when I test, I get in the terminal the following errors:

Internal Server Error: /search
Traceback (most recent call last):
  File "C:\Python310\lib\site-packages\django\core\handlers\exception.py", line 55, in inner
    response = get_response(request)
  File "C:\Python310\lib\site-packages\django\core\handlers\base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "C:\Users\user\Documents\AAprojects\Whelpsgroups1\searchEngine\search\views.py", line 29, in search
    for header in final_result[1]:
IndexError: list index out of range
[17/Sep/2022 23:28:40] "GET /search?csrfmiddlewaretoken=t57w6QZSzkgsKoNJWXrjotBRdKSNxBBxzfzPnUC6N7kXfs5pMyGWcjQUYcrT13Ds&search=moto HTTP/1.1" 500 88593

I don't know how to solve this problem.Thank you!

http-headers django python
1 Answers

This is more of a Python Requests issue than Django.. from my limited knowledge I don't believe you can get header information of pages just by looking at the Links. You need to actually send a GET request because it's in the Header of the request:

for l in links:
    response = requests.get(l)
    if response['X-Frame-Options'] in ["DENY", "SAMEORIGIN"]:
        head = True
        notAccept = bool(head)
    else:
        notAccept = bool(False)

I was unable to find anything about CSP_FRAME_ANCESTORS tho..

Hopefully you can find some of this useful.. at the very least now you know to search python requests {x} on the topic

Edit

I'll explain the error that you've added, invalid Index:


# Final Result is an Array Filled with Tuples
#   OR just think of it as an Array filled with Arrays
# ---

# This would be the result on the first loop:

final_result = [
    (result_title0, result_url0, result_desc0), # index 0
    ]

# You used:
final_result[1] # => Undefined

# Correct Way:
# ---

# Grab first item in Array:
final_result[0] # => (result_title0, result_url0, result_desc0)

# Grab first item in Array + and then 2nd item in list:
final_result[0][1] # => result_url0

# Next Issue
# ---

# But you will run into this issue / always grabbing the first item
final_result = [
    (result_title0, result_url0, result_desc0), # index 0
    (result_title1, result_url1, result_desc1), # index 1
    ]


final_result[0][1] # => result_url0 **Wrong!**

# -1 should be used instead // (Last Item in list)
final_result[-1][1] # => result_url1 **Correct!**

# ^ Actual solution
# ---

But because you already have result_url as a variable in the loop you might has well use it in the GET instead of trying to fetch it from that Nested Array

for result in result_listings:
    result_title = result.find(class_='PartialSearchResults-item-title').text
    result_url = result.find('a').get('href')
    result_desc = result.find(class_='PartialSearchResults-item-abstract').text
    final_result.append((result_title, result_url, result_desc))

    # Ping URL found here: result.find('a').get('href')
    response = requests.get(result_url)

    # Check for header information in the response
    if response['X-Frame-Options'] in ["DENY", "SAMEORIGIN"]:
        # head = True
        notAccept = True
    else:
        notAccept = False

and you might as well wait until the very end to add that tuple to the final_result list- maybe even use a dictonary

Wait until end

for result in result_listings:
    result_title = result.find(class_='PartialSearchResults-item-title').text
    result_url = result.find('a').get('href')
    result_desc = result.find(class_='PartialSearchResults-item-abstract').text

    # Ping URL found here: result.find('a').get('href')
    response = requests.get(result_url)

    # Check for header information in the response
    if response['X-Frame-Options'] in ["DENY", "SAMEORIGIN"]:
        # head = True
        notAccept = True
    else:
        notAccept = False

    # Add here! Last second.
    final_result.append((result_title, result_url, result_desc, notAccept))

Wait until end + Dict

for result in result_listings:
    result_title = result.find(class_='PartialSearchResults-item-title').text
    result_url = result.find('a').get('href')
    result_desc = result.find(class_='PartialSearchResults-item-abstract').text

    # Ping URL found here: result.find('a').get('href')
    response = requests.get(result_url)

    # Check for header information in the response
    if response['X-Frame-Options'] in ["DENY", "SAMEORIGIN"]:
        # head = True
        notAccept = True
    else:
        notAccept = False

    # Dict makes Code a little more readable last on when you use this data
    final_result.append({
        'title':result_title,
        'url': result_url,
        'desc': result_desc,
        'x-frame': notAccept
        })
Nealium0
Related