Python multiprocessing - Subprocesses with same PID

Viewed 64

Basically I'm trying to open a new process every time I call a function. The problem is that when I get the PID inside of the function, the PID is the same as in another functions even if the other functions haven't finished yet.

I'm wrapping my function with a decorator:

def run_in_process(function):
    """Runs a function isolated in a new process.

    Args:
        function (function): Function to execute.
    """

    def wrapper(*args):
        parent_connection, child_connection = Pipe()
        process = Process(target=function, args=(*args, child_connection))
        process.start()
        response = parent_connection.recv()
        process.join()
        return response

    return wrapper

And declaring the function like this:

@run_in_process
def example(data, pipe):
    print(os.getpid())
    pipe.send("Just an example here!")
    pipe.close()

Obs1.:This code is running inside a AWS Lambda.

Obs2.: Those lambdas didn't finish before the other one starts, because this tasks takes at least 10 seconds.

Log of execution 1

Log of execution 2

Log of execution 3

You can see at the logs that each one is a different execution and they are executed at the "same" time.

The question is: Why they have the same PID even knowing that they are running concurrently? Shouldn't they have different PIDs?

I obligatorily need to execute this function in an isolated process

1 Answers

Your Lambda function could have been running in multiple containers at once in the AWS cloud. If you've been heavily testing your function with multiple concurrent requests, it is quite possible that the AWS orchestration created additional instances to handle the traffic.

With serverless, you lose some of the visibility into exactly how your code is being executed, but does it really matter?

Related