Unable to boost the performance while parsing links from landing pages

Since your crawler is using threading, why not "spawn" more workers to process the follow URLs from the landing pages?

For example:

import concurrent.futures as futures
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

base = "https://stackoverflow.com"
links = [
    f"{base}/questions/tagged/web-scraping?tab=newest&page={i}&pagesize=30"
    for i in range(1, 5)
]

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36",
}


def threader(function, target, workers=5):
    with futures.ThreadPoolExecutor(max_workers=workers) as executor:
        jobs = {executor.submit(function, item): item for item in target}
        futures.as_completed(jobs)


def make_soup(page_url: str) -> BeautifulSoup:
    return BeautifulSoup(requests.get(page_url).text, "html.parser")


def process_page(page: str):
    s = make_soup(page).find("div", class_="grid--cell ws-nowrap mb8")
    views = s.getText() if s is not None else "Missing data"
    print(f"{page}\n{' '.join(views.split())}")


def make_pages(soup_of_pages: BeautifulSoup) -> list:
    return [
        urljoin(base, item.select_one("a.question-hyperlink").get("href"))
        for item in soup_of_pages.select(".summary > h3")
    ]


def crawler(link):
    while True:
        soup = make_soup(link)
        threader(process_page, make_pages(soup), workers=10)
        next_page = soup.select_one(".pager > a[rel='next']")
        if not next_page:
            return
        link = urljoin(base, next_page.get("href"))


if __name__ == '__main__':
    threader(crawler, links)

Sample run output:

https://stackoverflow.com/questions/66463025/exporting-several-scraped-tables-into-a-single-csv-file
Viewed 19 times
https://stackoverflow.com/questions/66464511/can-you-find-the-parent-of-the-soup-in-beautifulsoup
Viewed 32 times
https://stackoverflow.com/questions/66464583/r-subscript-out-of-bounds-for-reading-an-html-link
Viewed 22 times

and more ...

Rationale:

Essentially, what you're doing in your initial approach is spawning workers to fetch question URLs from search pages. You don't process the follow URLs.

What I've suggested is to spawn additional workers to process what the crawling workers collected.

In your question you mention:

I wish to get the titles from landing pages

That's what a tuned version of your initial approach is trying to accomplish by utilizing the threader() function, which is basically a wrapper for ThreadPool().

1. I hope you didn't actually spawn a single thread as shown in your example :)

future_to_url = {executor.submit(get_titles,url): url for url in [link]}

2. You'll speed it up A LOT by simply using a Session (which means reusing the connection, explicitly claiming that you may query this site once again until the session object appeared to be explicitly finalised or garbage-collected) instead of a plain requests.get() calls.
3. Indeed the python threads ain't good at CPU-bound tasks (like parsing of HTML) because of GIL, and you should probably use ProcessPoolExecutor for parsing, and leave to a ThreadPoolExecutor (or even to a single thread) HTTP requests handling only.
4. After all, I strongly recommend to take a look at the aiohttp as a non-blocking successor of requests (of urllib actually, but nvm) - it is built on top of the asyncio, so that it allows to forget about thread-safety problems, implicit locks, and so on...

   In [3]: import aiohttp, asyncio, time
      ...: 
      ...: t0 = time.monotonic()
      ...: 
      ...: 
      ...: async def do_the_request(session):
      ...:     async with session.get("http://www.google.com") as resp:
      ...:         content = await resp.read()
      ...: 
      ...: 
      ...: async def main():
      ...:     async with aiohttp.ClientSession() as session:
      ...:         tasks = {asyncio.create_task(do_the_request(session)) for _ in r
      ...: ange(100)}
      ...:         await asyncio.wait(tasks, return_when=asyncio.ALL_COMPLETED)
      ...: 
      ...: 
      ...: t0 = time.monotonic()
      ...: asyncio.run(main())
      ...: t1 = time.monotonic()
      ...: print(f"Time elapsed: {t1 - t0:.3f}")
      Time elapsed: 0.571
   
      In [4]: 
   

5. Finally, if you'll hit the limit of the CPU core (I bet you'll exhaust you bandwidth earlier) running your process - just start another one with ProcessPoolExecutor.

Python doesn't handle concurrency very well, You could bypass that by making your Python script handle a single link, and then use Bash to implement concurrency. Here's an example:

The python code, let's call it crawlLink.py:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
import sys

base = 'https://stackoverflow.com'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.190 Safari/537.36',
}

def get_titles(link):
    res = requests.get(link,headers=headers)
    soup = BeautifulSoup(res.text,"html.parser")
    for item in soup.select(".summary > h3"):
        post_title = item.select_one("a.question-hyperlink").get("href")
        print(urljoin(base,post_title))

link = sys.argv[1]
get_titles(link)

The bash script:

#! /bin/bash

links=""

for page in {1..5}
do
    links="${links} https://stackoverflow.com/questions/tagged/web-scraping?tab=newest&page=${page}&pagesize=30"
done

echo "${links}" | xargs  -i --max-procs=5 bash -c '/usr/bin/python3 crawlLink.py "{}"'