Scrapy bypass data usage consent wall

I am scraping yahoo finance news using the code below.

class YfinNewsSpider(scrapy.Spider):
    name = 'yfin_news_spider'
    custom_settings = {'DOWNLOAD_DELAY': '0.5', 'COOKIES_ENABLED': True, 'COOKIES_DEBUG': True}

    def __init__(self, month, year, **kwargs):
        self.start_urls = ['https://finance.yahoo.com/sitemap/2020_03_all']
        self.allowed_domains = ['finance.yahoo.com']
        super().__init__(**kwargs)

    def parse(self, response):
        all_news_urls = response.xpath('//ul/li[@class="List(n) Py(3px) Lh(1.2)"]')
        for news in all_news_urls:
            news_url = news.xpath('.//a[@class="Td(n) Td(u):h C($c-fuji-grey-k)"]/@href').extract_first()
            yield scrapy.Request(news_url, callback=self.parse_news, dont_filter=True)

    def parse_news(self, response):
        news_url = str(response.url)
        title = response.xpath('//title/text()').extract_first()
        paragraphs = response.xpath('//div[@class="caas-body"]/p/text()').extract()
        date_time = response.xpath('//div[@class="caas-attr-time-style"]/time/@datetime').extract_first()

        yield {'title': title, 'url': news_url, 'body_text': paragraphs, 'timestamp': date_time}

However, when I run my spider it give me below results.

2020-11-28 20:42:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_05cc09ea-0bc0-439d-8b4c-2d6f20f52d6e> (referer: https://finance.yahoo.com/sitemap/2020_03_all)
2020-11-28 20:42:40 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://finance.yahoo.com/news/onegold-becomes-first-company-offer-110000241.html>
Cookie: B=cnmvgrdfs5a0r&b=3&s=o1; GUCS=ASXMbR9p
2020-11-28 20:42:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_05cc09ea-0bc0-439d-8b4c-2d6f20f52d6e>
{'title': 'Yahoo er nu en del af Verizon Media', 'url': 'https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_05cc09ea-0bc0-439d-8b4c-2d6f20f52d6e', 'body_text': [], 'timestamp': None}
2020-11-28 20:42:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_d6731ce6-78bc-4222-914f-24cf98f874b8> (referer: https://finance.yahoo.com/sitemap/2020_03_all)

This seems to indicate that when my spider go to https://finance.yahoo.com/news/onegold-becomes-first-company-offer-110000241.html found in https://finance.yahoo.com/sitemap/2020_03_all. It tried sending cookie to https://finance.yahoo.com/news/onegold-becomes-first-company-offer-110000241.html, but was redirected to consent accepting wall https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_05cc09ea-0bc0-439d-8b4c-2d6f20f52d6e.

I open this consent wall https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_05cc09ea-0bc0-439d-8b4c-2d6f20f52d6e in browser and found data consent accepting screen. When I clicked accept, it brought me to the correct site that I want to scrape. The scraping results also exactly the content in this consent screen.

I have tried setting COOKIES_ENABLED to True, but it did not work. So, is there anyway to bypass this accepting screen in scrapy?

Thank you.