me and some friends are doing a small language competition to calculate some neural networks. Some doing in C other in fortran, and me: Python.
The code is simple, is just a bunch of vector dot operations and a summation after that apply a signal function and return -1 or 1 (activated or not).
With that we are sending a bunch of random numbers and checking (right now only single process) which language do it faster.
My code is simple as this:
def sgn(h):
"""Signal function"""
return -1 if h < 0 else 1
def lincomb(A, B):
"""Linear combinator between two matrices"""
return np.einsum('ji,ij->', A, B)
def lincombrav(A, B):
return A.ravel().dot(B.ravel('F'))
def functional_test():
w1 = np.random.random(50**2).reshape(50,50)
w2 = np.random.random(50**2).reshape(50,50)
return sgn(lincombrav(w1, w2))
Where A and B are matrices that represent each layer in a neural network. then we dot the ith-column of the first matrix with the ith-row for the second matrix, sum all results and send to signal function. Something like:
w1 = 2*np.random.random(100**2).reshape(100,100)-1
w2 = 2*np.random.random(100**2).reshape(100,100)-1
then we time it with
%timeit sgn(lincomb(w1, w2))
Python is losing to Fortran by 38x :-(
Is there anyway to improve that Python "code".
EDIT: Added timeit results:
Python version (already with the ravel mode)
In [10]: %timeit functional_test()
8.72 µs ± 406 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
Python version (with einsum)
In [16]: %timeit functional_test()
10.27 µs ± 490 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
Fortran version
In [13]: %timeit fort.test()
235 ns ± 12.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
Fortran version was created using "f2py" program, to generate a python loadable module from fortran code.
The test functions do the following (in each language):
- Create the matrix A
- Create the matrix B
- call sgn(lincomb(A,B)) # from each respective language implementation
I also moved the matrix creation to outside, to run only the mathematical operation instead also handling memory. Still, python is behind by same magnitude.
EDIT2: Good python news. Python has won in all but the small matrix tests. Here will follow the whole code:
Python functions (bla.py)
import numpy as np
from numba import jit
import timeit
import matplotlib.pyplot as plt
def sgn(h):
"""Signal function"""
return -1 if h < 0 else 1
def lincomb(A, B):
"""Linear combinator between two matrices"""
return np.einsum('ji,ij->', A, B)
def lincombrav(A, B):
return A.ravel().dot(B.ravel('F'))
def functional_test_ravel(n):
"""Functional tests (Victor experiment)"""
w = 2*np.random.random(n**2).reshape(n,n)-1
x = 2*np.random.random(n**2).reshape(n,n)-1
return sgn(lincombrav(w, x))
def functional_test_einsum(n):
"""Functional tests (Victor experiment)"""
w = 2*np.random.random(n**2).reshape(n,n)-1
x = 2*np.random.random(n**2).reshape(n,n)-1
return sgn(lincomb(w, x))
@jit()
def functional_test_numbaein(n):
"""Functional tests (Victor experiment)"""
w = 2*np.random.random(n**2).reshape(n,n)-1
x = 2*np.random.random(n**2).reshape(n,n)-1
return sgn(lincomb(w, x))
@jit()
def functional_test_numbarav(n):
"""Functional tests (Victor experiment)"""
w = 2*np.random.random(n**2).reshape(n,n)-1
x = 2*np.random.random(n**2).reshape(n,n)-1
return sgn(lincombrav(w, x))
Fortran functions (fbla.f95)
module fbla
implicit none
integer, parameter::dp = selected_real_kind(12,100)
public
contains
real(kind=dp) function sgn(x)
integer, parameter::dp = selected_real_kind(12,100)
real(kind=dp), intent(in):: x
if(x >= 0.0 ) then
sgn = +1.0
else if (x < 0.0) then
sgn = -1.0
end if
end function sgn
real(kind=dp) function lincomb(A, B, n)
integer, parameter :: sp = selected_int_kind(r=8)
integer, parameter :: dp = selected_real_kind(12,100)
integer(kind=sp) :: i
integer(kind=sp), intent(in):: n
real(kind=DP), intent(in) :: A(n,n)
real(kind=DP), intent(in) :: B(n,n)
lincomb = 0
do i=1,n
lincomb = lincomb + dot_product(A(:,i),B(i,:))
end do
end function lincomb
real(kind=dp) function functional_test(n)
integer, parameter::dp = selected_real_kind(12,100)
integer, parameter::sp = selected_int_kind(r=8)
integer(kind=sp), intent(in):: n
integer(kind=sp):: i, j
real(kind=dp), allocatable, dimension(:,:):: x, w, wt
ALLOCATE(wt(n,n),w(n,n),x(n,n))
do i=1,n
do j=1,n
w(i,j) = 2*rand(0)-1
x(i,j) = 2*rand(0)-1
end do
end do
wt = transpose(w)
functional_test = sgn(lincomb(wt, x, n))
end function functional_test
end module fbla
Test execution functions (tests.py)
import numpy as np
import timeit
import matplotlib.pyplot as plt
import bla
from fbla import fbla
def run_test(test_functions, N, runs=1000):
results = []
global rank
for n in N:
rank = n
for t in test_functions:
# print(f'Rank {globals()["rank"]}')
print(f'Running {t} to matrix size {rank}', end='')
r = min(timeit.Timer(t , globals=globals()).repeat(repeat=5, number=runs))
print(f' total time {r} per run {r/runs}')
results.append((t, n, r, r/runs))
return results
def plotbars(results, test_functions, N):
Nsz = len(N)
M = len(test_functions)
fig, ax = plt.subplots()
ind = np.arange(int(Nsz))
width = 1/(M+1)
p = []
for n in range(M):
g = [ w*1000 for (x,y,z,w) in results if x==test_functions[n]]
p.append(ax.bar(ind+n*width, g, width, bottom=0))
ax.legend([ l[0] for l in p ], test_functions)
ax.set_xticks(ind-width/2+((M/2) * width))
ax.set_xticklabels(np.array(N).astype(str))
ax.set_xlabel('Rank of square random matrix')
ax.set_ylabel('Average time(ms) per run')
ax.set_yscale('log')
return fig
N = (10, 50, 100, 1000)
test_functions = [
'bla.functional_test_einsum(rank)',
'fbla.functional_test(rank)'
]
results = run_test(test_functions, N)
plot = plotbars(results, test_functions, N)
plot.show()
The results are:
[('bla.functional_test_einsum(rank)', 10, 0.023221354000270367, 2.3221354000270368e-05),
('fbla.functional_test(rank)', 10, 0.005375514010665938, 5.375514010665938e-06),
('bla.functional_test_einsum(rank)', 50, 0.07035048000398092, 7.035048000398091e-05),
('fbla.functional_test(rank)', 50, 0.1242617039824836, 0.0001242617039824836),
('bla.functional_test_einsum(rank)', 100, 0.22694124400732107, 0.00022694124400732108),
('fbla.functional_test(rank)', 100, 0.5518505079962779, 0.0005518505079962779),
('bla.functional_test_einsum(rank)', 1000, 37.88827919398318, 0.03788827919398318),
('fbla.functional_test(rank)', 1000, 74.09929457501858, 0.07409929457501857)]
Some standard timeit output from a ipython3 session. fbla is the fortran library while bla is standard python library.
In : n=1000
In : w1 = 2*np.random.random(n**2).reshape(n,n)-1
In : w2 = 2*np.random.random(n**2).reshape(n,n)-1
In : bla.sgn(bla.lincomb(w1,w2))
Out: -1
In : fbla.sgn(fbla.lincomb(w1,w2))
Out: -1.0
In : %timeit fbla.sgn(fbla.lincomb(w1,w2))
11.3 ms ± 430 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In : %timeit bla.sgn(bla.lincomb(w1,w2))
3.81 ms ± 573 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
