How to optimize python code in pure pythonic way without cythonizing it

user_17375573 0
Viewed 140

Edited:

(This post has been largely edited compared to its previous version).

For test purpose I have a module, say, ptest.py :

# ptest.py
def testfunc(*args) :
    # Some process that can take significant time. There may be some integers larger than 2^64.

After cythonizing it to, say, ctest.pyx I got an increase in performance with some further tests (as expected).

Now for some integer value large enough the program may throw OverflowError while in pure python that might not happen. In this case, can I raise SomeError with information (say, ValueError('The number should not be greater than some value')) in cython as we do in python ?

Even I did the same,

if n > 2**64:
    raise ValueError("Number should be a non negative integer less than 2^64.")

it kept showing: OverflowError: Python int too large to convert to C unsigned long. Seems to ignore the condition completely.

So, how can I inform the user ? And finally, can I use cython in pure python without cythonizing the entire program (that is without setup, build etc.) as I don't want to lose those functionality of python (in this case raising Exception, handling sufficiently large integer etc.)?

I tried other methods from cython module (following Cython doc.) like, cython.declare, cython.exceptval etc. in .py file but they all seem to be a failure in improving the performance. I haven't approached ctypes yet as I want the proper technique beforehand.

cython optimization python
2 Answers

I timed and ran the OP's code with a value of n==1_000 10_000 times. The duration was ~0.77s

Changing the code to use a list comprehension as follows:

def testfunc(n: int) -> list :
    return [n_ for n_ in range(n, 0, -1)]

...reduced the duration to ~0.26s

user_175803810

The only way to do what you're asking is to program better. In this example you can just do:

list(range(n,0,-1))

And that's the thing here: by thinking about what you want to do you can do the same thing smarter and clearer. That's what it means to work in a pythonic way.

Other things you can do to optimize code pythonically:

  • think of smarter algorithms (with pen and paper)
  • vectorize (use numpy) whenever you can
  • use the right type, if you want to check if an item is in a list, a set might be more appropriate
  • don't index lists (this is relatively slow), you can usually iterate over them
  • there's usually a library that does what you want faster than you can program it (often written in C)
user_67664570
Related