Not able to fetch polyline polypoints because of restricted Google Maps API Key: Flutter

I am using a Google Maps API Key which permits calls through SDK only. I want to display polyline between source and destination location but I get the following message with my code-

This IP, site or mobile application is not authorized to use this API key. Request received from (IP ADDRESS), with empty referer status:REQUEST_DENIED points:[]

I am using the most used package: https://pub.dev/packages/flutter_polyline_points
But, I cannot get the poly points using the provided method-

 _getPolyline() async {
    PolylineResult result = await polylinePoints.getRouteBetweenCoordinates(
        googleAPiKey,
        PointLatLng(_originLatitude, _originLongitude),
        PointLatLng(_destLatitude, _destLongitude),
        travelMode: TravelMode.driving,
        wayPoints: [PolylineWayPoint(location: "Sabo, Yaba Lagos Nigeria")]);
    if (result.points.isNotEmpty) {
      result.points.forEach((PointLatLng point) {
        polylineCoordinates.add(LatLng(point.latitude, point.longitude));
      });
    }
    _addPolyLine();
  }

The method polylinePoints.getRouteBetweenCoordinates() uses a URL to fetch responses, but this is not permitted for restricted API Keys, which allow only SDKs. Please help me find an alternative so that I can still get the poly points using my restricted API Key. I tried finding some alternatives but did not come across any different methods.