Shared-memory IPC solution for both Linux and Windows

Question

Below is a simple and perfect solution on Windows for IPC with shared memory, without having to use networking / sockets (that have annoying limits on Windows). The only problem is that it's not portable on Linux:

Avoiding the use of the tag parameter will assist in keeping your code portable between Unix and Windows.

Question: is there a simple way built-in in Python, without having a conditional branch "if platform is Windows, if platform is Linux" to have a shared-memory mmap?

Something like

mm = sharedmemory(size=2_000_000_000, name="id1234")  # 2 GB,  id1234 is a global
                                                      # id available for all processes
mm.seek(1_000_000)
mm.write(b"hello")

that would internally default to mmap.mmap(..., tagname="id1234") on Windows and use /dev/shm on Linux (or maybe even a better solution that I don't know?), and probably something else on Mac, but without having to handle this manually for each different OS.

Working Windows-only solution:

#server
import mmap, time
mm = mmap.mmap(-1, 1_000_000_000, tagname="foo")
while True:
    mm.seek(500_000_000)
    mm.write(str(time.time()).encode())
    mm.flush()
    time.sleep(1)
# client
import mmap, time
mm = mmap.mmap(-1, 1_000_000_000, tagname="foo")
while True:
    mm.seek(500_000_000)
    print(mm.read(128))
    time.sleep(1)

user_7756864 · Answer

The easiest way is to use python with version >=3.8, it has added a built-in abstraction for shared memory, it works on both windows and linux https://docs.python.org/3.10/library/multiprocessing.shared_memory.html

The code will look something like this:

Process #1:

  from multiprocessing import shared_memory
  # create=true to create a new shared memory instance, if it already exists with the same name, an exception is thrown
  shm_a = shared_memory.SharedMemory(name="example", create=True, size=10)
  shm_a.buf[:3] = bytearray([1, 2, 3])
  while True:
      do_smt()
  shm_a.close()

Process #2:

  from multiprocessing import shared_memory
  # create=false, use existing
  shm_a = shared_memory.SharedMemory(name="example", size=10)
  print(bytes(shm.buf[:3]))
  # [0x01, 0x02, 0x03]
  while True:
      do_smt()
  shm_a.close()

Otherwise, I think there are no common good solutions and you will need to reinvent the wheel :)

user_6085193 · Answer

Personally this has worked well for me

Option 1: http://www.inspirel.com/yami4/

The YAMI4 suite for general computing is a multi-language and multi-platform package.

Several Operating systems:

Sample code

Microsoft Windows, POSIX (Linux, Max OS X, FreeBSD, ...), QNX (with native IPC messaging), FreeRTOS, ThreadX, TI-RTOS. Programming languages: C++, Ada, Java, .NET, Python, Wolfram.

Shared-memory IPC solution for both Linux and Windows

2 Answers

Option 1: http://www.inspirel.com/yami4/

Option 2: ZeroMq https://zeromq.org/