How to Extract HTML from JSON Response

user_1292262 0
Viewed 455

I'm using a web scraping API at https://scrapfly.io, which is returning the contents of the page as an item in a JSON response. I'm able to extract the item but it's formatted with '\n' and escape characters. I just want to return raw HTML so that I can parse it as normal.

This is what I have at the moment...

todos = json.loads(result.text)
output = json.dumps(todos['result']['content'], ensure_ascii=True)
print(output)

And the output looks like this...

"\n<!DOCTYPE html>\n<html lang=\"en-us\">\n<head>\n<title>Google Data...

What I want is...

<!DOCTYPE html>
<html lang="en-us">
<head>
<title>Google Data...

Any help would be greatly appreciated.

Lux

PS: As I suspected, I wasn't processing the response correctly. The correct solution is below. Hope it helps other people.

python
3 Answers

You could check for "\n" in your output and replace it.

result = output.replace("\n", "")
user_160474870

I hired somebody in the end and they solved the issues easily.

Just for anybody else who's using https://scrapfly.io and has trouble decoding the 'content' variable returned, here's how it's done...

from bs4 import BeautifulSoup
import requests

def send_request(url):

    scrapfly_api = 'YOUR_API'
    scrapfly_url = 'https://api.scrapfly.io/scrape'

    resp = requests.get(
        url= scrapfly_url,
        params={
            "key": scrapfly_api,
            "url": url,
            "country": 'us',
        },
        timeout = 15,
    )

    json_code = resp.json()
    html = json_code['result']['content']
    soup = BeautifulSoup(html, 'lxml')
    print(str(soup).encode('ascii', 'ignore').decode())

Hope that helps and thanks to the guys that tried to help me.

Lux

user_12922620

Scrapfly responds in JSON and the content scraped is also in JSON, Scrapfly doesn't touch the response content (All decode stuff is done as mentioned in the documentation https://www.scrapfly.io/docs/scrape-api/getting-started#faq_find_scrape_content, so it's a JSON inside a JSON response. You have to decode the content according to the content type, here you have JSON, so you have to JSON decode it.

Another thing, I don't know if it's intended, you set 15s as timeout which might not recommend their documentation

They have a python SDK https://www.scrapfly.io/docs/sdk/python

user_148887300
Related