Why does the toPandas() function in PySpark throw a connection error if the size of the file is too large?

I have been trying to apply the toPandas() function to a file that is 5GB in size and I keep getting a connection refused error.

ConnectionRefusedError                    Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/py4j/clientserver.py in connect_to_java_server(self)
    436                 self.socket = self.ssl_context.wrap_socket(
    437                     self.socket, server_hostname=self.java_address)
--> 438             self.socket.connect((self.java_address, self.java_port))
    439             self.stream = self.socket.makefile("rb")
    440             self.is_connected = True

ConnectionRefusedError: [Errno 111] Connection refused

I have tested the function for files upto a few 100 MB and it seems to work. Is there no workaround for using the toPandas() function for files of larger sizes?