Elasticsearch query with filter take more time than query without filter. Why?

I am using Elasticsearch Version (7.6.1)

Query with Filter is

GET mark13/_search
{
    "explain": false,
    "from": 0,
    "size": 500,
    "track_scores": true,
    "stored_fields": [
        "_source"
    ],
    "sort": {
        "_script": {
            "type": "number",
            "script": {
                "id": "sorting_algo",
                "params": {
                    "query": "abhinav keshri"
                }
            },
            "order": "desc"
        }
    },
    "script_fields": {
        "poca_score": {
            "script": {
                "id": "field",
                "params": {
                    "query": "abhinav keshri"
                }
            }
        }
    },

    "query": {
        "bool": {
            "filter": [
              {
                "term": {
                  "class" : "42"
                }
              }

            ], 
            "should": [
                {
                    "match": {
                        "applied_for": {
                            "query": "abhinav keshri",
                            "boost": 118,
                            "fuzziness": 0
                        }
                    }
                },
...
...
...

Output of the above query is

{
  "took" : 45414,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 4730.905,
    "hits" : [
      {
...
...
...

It took 45414 ms to execute.

Query without Filter is

GET mark13/_search
{
    "explain": false,
    "from": 0,
    "size": 500,
    "track_scores": true,
    "stored_fields": [
        "_source"
    ],
    "sort": {
        "_script": {
            "type": "number",
            "script": {
                "id": "sorting_algo",
                "params": {
                    "query": "abhinav keshri"
                }
            },
            "order": "desc"
        }
    },
    "script_fields": {
        "poca_score": {
            "script": {
                "id": "script",
                "params": {
                    "query": "abhinav keshri"
                }
            }
        }
    },

    "query": {
        "bool": {
            "should": [
                {
                    "match": {
                        "applied_for": {
                            "query": "abhinav keshri",
                            "boost": 118,
                            "fuzziness": 0
                        }
                    }
                },
...
...
...

Output of the above query is

{
  "took" : 7104,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 9920,
      "relation" : "eq"
    },
    "max_score" : 4730.905,
    "hits" : [
      {

It took 7104 ms to execute.

My Expectation was that , Filter query would take less time as compared to non-Filter query, since there is less results to apply bool query on.

Also i have tried to execute filter query in different formats (one given above)-

also i have tried following format. (it gives me the same results ).

{
    "from": 0,
    "size": 50,
    "track_scores": true,
    "stored_fields": [
        "_source"
    ],
    "sort": {
        "_script": {
            "type": "number",
            "script": {
                "id": "sorting_algo",
                "params": {
                    "query": "abhinav keshri"
                }
            }
        }
    },
    "script_fields": {
        "poca_score": {
            "script": {
                "id": "script",
                "params": {
                    "query": "abhinav keshri"
                }
            }
        }
    },
    "query": {
        "bool": {
            "should": [

            ],
            "filter": [
                {
                    "bool": {
                        "should": [
                            {
                                ...
                                ...
                            }
                        ],
                        "must": {
                            "bool": {
                                "should": [
                                    {
                                        "terms": {
                                            "class": [
                                                "42"
                                            ]
                                        }
                                    },
                                    ...
                                    ...

                                ]
                            }
                        }
                    }
                }
            ]
        }
    }
}

Question Why Filter query take longer than non-Filter query?