Tuesday, January 27, 2015

Scale your projects with Eventlet ( concurrent networking library ) + monkeypatching without changing existing code base

So what if you have an existing python code base but you are unable to scale it because of varoius dependencies like blocking I/O, network connections e.t.c

Look into eventlet !!! (thanks to my co-worker who suggested not looking into twisted to solve the problem since it was just plainly painful .. but look into eventlet)

easy to install , easy to integrate into existing code base ( if its not too late and if you are not using any C based libraries in your python code base)

They call it "greening"

So I ran into a similar blocking issue and the way i greened my  application is simply by monkeypatching the standard library.

pip install eventlet

 

Then add these lines as soon as your programme starts ( to avoid late binding problems) 

 
import eventlet
eventlet.monkey_patch() 

Now try to think how on highlevel you can sort of create threads on the blocking process and spawn threads of that blocking call.

( for ex:- lets say the blocking function  has a network connection or some other blocking I/O and it is called foo(p1,p2) and it is using a standard python library  )

 pool = eventlet.GreenPool()
 pool.spawn_n(foo, p1, p2)
 pool.waitall()

voila! you are async again!!! no matter how much time foo takes to complete .. you can move ahead :)

No comments:

Post a Comment