How to get title from URL in PHP from sites returning 403 Forbidden

user_12018281 0
Viewed 814

I am trying to get the title of a few pages in PHP with this code. It works fine with almost every link except for a few, for example, with 9gag.

function download_page($url)
{
    $agent = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36';

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_VERBOSE, true);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_USERAGENT, $agent);
    curl_setopt($ch, CURLOPT_URL, $url);
    $data = curl_exec($ch);

    return $data;
}

function get_title_tag($str)
{
    $pattern = '/<title[^>]*>(.*?)<\/title>/is';

    if(preg_match_all($pattern, $str, $out))
    {
        return $out[1][0];
    }
    return false;
}

$url = "https://9gag.com/gag/avPBX3b";

$data = download_page($url);

echo $extracted_title = get_title_tag($data);

It echoes

Attention Required! | Cloudflare

which seems to be protected by a Cloudflare bot verification page. But when I try to post this link on any social network, they are able get the title and all the metadata required. How is it possible?

Edit:

Even if I use the opengraph.io API, I get:

"root":{
    "error":{
        "code": 2005
        "message": "Got 403 error from server."
    }
}
cloudflare web-crawler url php
2 Answers

just replace agent string and it should work OK, from:

$agent = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36';

to:

$agent = 'facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)';

I see that CloudFlare has enabled captcha verification if standard agent strings are present so this will easily bypass this. I'm puzzled with security here but that is out of scope of this question

user_8534280

You can make use of Facebook's Graph API.

https://graph.facebook.com/v7.0/?fields=og_object&id=https://9gag.com/gag/avPBX3b

JSON Output:

{
   "og_object": {
      "id": "994417753967326",
      "description": "More memes, funny videos and pics on 9GAG",
      "title": "32 Places People Have Mispronounced Their Entire Life",
      "type": "article",
      "updated_time": "2020-06-12T15:54:27+0000"
   },
   "id": "https://9gag.com/gag/avPBX3b"
}

You can read more about it's usage here.

user_50732810
Related