Python serial read is extremely slow, how to speed up?

Admin 1
Viewed 16470

I'm using the following code to communicate with a Digital Multi Meter (DMM).
It works ok. I can send commands and read results. I don't use readline because I read binary data.

The problem:

The problem is that it is very very slow. Same code written in Ruby is much faster. When it takes 30 sec in python, it takes 2 or 3 sec in ruby (using same speed). So it's not a hardware issue.
The only difference between Ruby code and Python is that in Python I use inWaiting() for reading all characters available. In Ruby, read() function reads all of them and not just one.

The code:

Here's the code: In read_retry function I check how many characters I have to read. I read them, and then call data_is_ok function to check if it's finished or not.
As you can see, there are '\r' embedded in the data returned. Reading is finished when the last character is '\r' (no more data available).
So there's a loop to read numerous chunks. 

import serial
[...]

def data_is_ok(data):
  # No status code yet
  if len(data) < 2: return False

  # Non-OK status
  if len(data) == 2 and data[0] != "0" and data[1] == "\r": return True

  # Non-OK status with extra data on end
  if len(data) > 2 and data[0] != "0":
    raise ValueError('Error parsing status from meter (Non-OK status with extra data on end)')

  # We should now be in OK state
  if data[0] != "0" or data[1] != "\r":
    raise ValueError('Error parsing status from meter (status:%c size:%d)' % (data[0], len(data)))

  return len(data) >= 4 and data[-1] == "\r"

def read_retry():
  retry_count = 0
  data = ""

  while retry_count < 500 and not data_is_ok(data):
    bytesToRead = ser.inWaiting()
    data += ser.read(bytesToRead)
    if data_is_ok(data): return data
    time.sleep (0.001)
    retry_count += 1
  raise ValueError('Error parsing status from meter:  %c %d %r %r' % (data[0],len(data),data[1] == '\r', data[-1] == '\r'))

[...]

# Serial port settings
try:
  ser = serial.Serial(port='/dev/cu.usbserial-AK05FTGH', baudrate=115200, bytesize=8, parity='N', stopbits=1, timeout=5, rtscts=False, dsrdtr=False)
except serial.serialutil.SerialException, err:
  print "Serial Port /dev/cu.usbserial-AK05FTGH doesn't respond"
  print err
  sys.exit()

[...]

ser.write(cmd+'\r')
data = read_retry()

I have used cProfile profiler. Most of the time is spent in time.sleep

Here's an extract:

         363096 function calls (363085 primitive calls) in 28.821 seconds

   Ordered by: internal time
   List reduced from 127 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    19050   25.245    0.001   25.245    0.001 {time.sleep}
        1    1.502    1.502    1.502    1.502 {posix.open}

The question:
Is it possible to make the code faster ?

python performance
3 Answers

A couple of things you can do.

You shouldn't need time.sleep(). If you know you only want 2 bytes then do ser.read(2) and if you want to limit the wait time ser.timeout = 0.01 EDIT unless you are in a separate thread. Python threads are greedy. I/O operations release the thread so the main thread can run. However, in your case you are always reading the data that is in the buffer ser.read(ser.inWaiting()). I've found that you need to force the serial port to wait on I/O ser.read(ser.inWaiting() + 32). Just make sure you also have a timeout, so you aren't waiting for the I/O forever. Also your current timeout is 5 seconds which is a long time to wait. end edit

You might be able to use readline ser.readline() this may only read to '\r\n'. I'm not sure.

If you are on Python3 then ser.read() returns bytes, so any comparison will be false. data[0] == '0' would need to be data[0:1] == b'0'

The other thing to do is to run the code through the debugger to make sure you are getting the data you are expecting. It is possible that a string comparison or something is wrong which would make you loop a bunch of times needlessly.

user_19652880

I have encountered the same problem that pySerial library exhibit very slow reading when using serial.readline method.

I am running Windows 7 and have tried different versions of pySerial, from 2.7 to 3.4.

Let me demonstrate my concern. A transmitter transmits new bunch of data each 0.02s with baud rate = 115200.

If I execute following

import serial

if __name__ == '__main__':

    serial_port = serial.Serial('COM26', 115200)   

    while(True):            
        print(serial_port.is_open)
        serial_port.readline()        
        print(serial_port.in_waiting)

I see the number of bytes in buffer is growing. Writing to buffer is faster than reading.

Solution to this problem is shown in following snippet that one must use serial.read method and read all buffered bytes at once. If I execute this

import serial

if __name__ == '__main__':

    serial_port = serial.Serial('COM26', 115200)

    while(True):        
        print(serial_port.is_open)
        serial_port.read(serial_port.in_waiting)                   
        print(serial_port.in_waiting)

then I see that serial_port.read(serial_port.in_waiting) can read very fast and it avoids the problem with accumulation of bytes in buffer.

I am fine with the solution for now but maybe somebody can explain why it behaves this way.

Thanks.

user_70922000

in read_retry() try using a timeout and not retry count

import time

def read_retry():
  timeout = 500 * 0.001
  data = ""
  tic = time.time()
  while toc - tic < timeout and not data_is_ok(data):
    bytesToRead = ser.inWaiting()
    data += ser.read(bytesToRead)
    if data_is_ok(data): return data
    toc = time.time()
  raise ValueError('Error parsing status from meter:  %c %d %r %r' % (data[0],len(data),data[1] == '\r', data[-1] == '\r'))
user_32059090
Related