I have a data acquisition app which populates some shared memory (/dev/shm). In that shared mem is a numpy array (with a fixed size) which is constantly updated. So far, so good.
Working on a gunicorn + plotly + react app which plots the data in that shared mem. Gunicorn reads data from /dev/shm via the SharedArray package makes a plotly json object and react shows it.
If I use yahoo finance to get some data everything works correct :
def mountain_day(symbol: str) -> dict:
data1 = yf.download(tickers=pair1, period='1d', interval='1m')
print (str(data1.tail(7)))
...
If I read the shared memory (sa.attach is the SharedArray package) and convert the numpy array to a df
def mountain_day(symbol: str) -> dict:
barscols = ['startTime', 'Time', 'Open', 'High', 'Low', 'Close', 'OpenA', 'HighA', 'LowA', 'CloseA', 'Spread', 'Ticks']
a = sa.attach("shm://Vol200 (1s) Index|MT5|622|C15S-SA")
data1 = pd.DataFrame(a, columns = barscols)
print (str(data1.tail(7)))
...
I get an OSError: [Errno 9] Bad file descriptor
Full traceback below
/real-time-plot/src# gunicorn -b 0:5000 --reload main:app
[2022-08-28 00:21:59 +0200] [18124] [INFO] Starting gunicorn 20.1.0
[2022-08-28 00:21:59 +0200] [18124] [INFO] Listening at: http://0.0.0.0:5000 (18124)
[2022-08-28 00:21:59 +0200] [18124] [INFO] Using worker: sync
[2022-08-28 00:21:59 +0200] [18174] [INFO] Booting worker with pid: 18174
[CONNECTED] sid='0mKWKQN-_ejlI5zIAAAB'
[2022-08-28 00:22:06 +0200] [18174] [ERROR] Socket error processing request.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/sync.py", line 136, in handle
self.handle_request(listener, req, client, addr)
File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/sync.py", line 194, in handle_request
util.reraise(*sys.exc_info())
File "/usr/local/lib/python3.8/dist-packages/gunicorn/util.py", line 626, in reraise
raise value
File "/usr/local/lib/python3.8/dist-packages/gunicorn/workers/sync.py", line 186, in handle_request
resp.close()
File "/usr/local/lib/python3.8/dist-packages/gunicorn/http/wsgi.py", line 391, in close
self.send_headers()
File "/usr/local/lib/python3.8/dist-packages/gunicorn/http/wsgi.py", line 322, in send_headers
util.write(self.sock, util.to_bytestring(header_str, "latin-1"))
File "/usr/local/lib/python3.8/dist-packages/gunicorn/util.py", line 287, in write
sock.sendall(data)
OSError: [Errno 9] Bad file descriptor
ril.ns
Attempting to ping sid='0mKWKQN-_ejlI5zIAAAB' with symbol='ril.ns'
startTime Time Open High Low Close OpenA HighA LowA CloseA Spread Ticks
86394 0.0 1.661639e+12 260.99 261.14 260.82 260.93 261.11 261.26 260.94 261.05 12.0 14.0
86395 0.0 1.661639e+12 261.04 261.06 260.65 260.81 261.16 261.18 260.77 260.93 12.0 16.0
86396 0.0 1.661639e+12 260.78 261.15 260.72 261.05 260.90 261.27 260.84 261.17 12.0 14.0
86397 0.0 1.661639e+12 260.99 261.02 260.34 260.44 261.11 261.14 260.46 260.56 12.0 16.0
86398 0.0 1.661639e+12 260.46 261.05 260.27 261.05 260.58 261.17 260.39 261.17 12.0 14.0
86399 0.0 1.661639e+12 261.10 261.20 260.81 260.81 261.22 261.32 260.93 260.93 12.0 14.0
86400 0.0 1.661639e+12 260.83 261.19 260.83 261.11 260.95 261.31 260.95 261.23 12.0 8.0
@sio.event
def ping_graph(sid, data):
symbol = data['symbol']
print(symbol)
try:
while sid in ACTIVE_CLIENTS:
print(f"Attempting to ping {sid=} with {symbol=}")
graph = plot.mountain_day(symbol)
sio.emit('graph_plot', graph, to=sid)
sio.sleep(61)
except Exception as e:
print(f"[EXCEPTION] {e}")
Gunicorn version is
gunicorn (version 20.1.0)
I know its quite non traditional code, semaphores, mutexes or locks are not needed as the np array [0,-2:,] in shared mem changed only every 20sec. The last line is updated every few millisec but is not used in the live graphing.
Reason for using the shared mem aproach instead of serialisation, packing, sockets, ipc or other things is speed while data acq and graphing application are completely independent of each other.
Anyone a clue to fix this and why this happens ?
Edit : fixed : -k eventlet did the trick