Debugging openstack .. not a cake!

So I have been playing with openstack for few days and as cool as it is ..i have to admit its a nightmare to debug.

so how do you start ?

1) find out all the major player nodes
2) go to each node and look into /var/log/
3) find out which one was written when your task failed

for examples

 I spinned up a vm and it was taking forever to come up.
 I logged into my compute node and found this

2015-02-18 10:07:26.586 3215 TRACE nova.compute.manager [instance: 0d28a0ad-93ce-43f5-b984-3c33f614861c] RemoteError: Remote error: OperationalError (OperationalError) (1048, "Column 'instance_uuid' cannot be null") 'UPDATE instance_extra SET updated_at=%s, instance_uuid=%s WHERE instance_extra.id = %s' (datetime.datetime(2015, 2, 18, 16, 7, 26, 576147), None, 152339L)

Now a friend pointed that this could be because of version mismatched.

So i logged into my controller node and checked the version

root@njain-compute:~# dpkg -l nova-compute
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================================================-==============================-==============================-============================================================================================================
ii nova-compute 1:2014.2.1-0ubuntu1~cloud0 all OpenStack Compute - compute node base

So i logged into my controller node and checked the version

root@control-\:~# dpkg -l |grep nova 
ii  nova-api                            1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - API frontend
ii  nova-cert                           1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - certificate management
ii  nova-common                         1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - common files
ii  nova-conductor                      1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - conductor service
ii  nova-consoleauth                    1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - Console Authenticator
ii  nova-novncproxy                     1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - NoVNC proxy
ii  nova-scheduler                      1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - virtual machine scheduler
ii  python-nova                         1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute Python libraries
ii  python-novaclient                   1:2.19.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

So i logged into my cinder node and checked the version

root@cinder-controller-mel01-1:~# dpkg -l|grep cinder
ii cinder-api 1:2014.2-0ubuntu1~cloud0 all Cinder storage service - API server
ii cinder-common 1:2014.2-0ubuntu1~cloud0 all Cinder storage service - common files
ii cinder-scheduler 1:2014.2-0ubuntu1~cloud0 all Cinder storage service - Scheduler server
ii python-cinder 1:2014.2-0ubuntu1~cloud0 all Cinder Python libraries
ii python-cinderclient 1:1.1.0-0ubuntu1~cloud0 all python bindings to the OpenStack Volume API

and as you can see cinder is out of sync .. and its version needs to match  1:2014.2.1


 well thats one way to debug(primarily since i was told version mismatch might be the issue)  though I wish debugging openstack was easier. 
Often people say reinstalling/rebuilding is better than debugging openstack and google is not yet abuzz with many clues .. here is to hoping!

 

Scale your projects with Eventlet ( concurrent networking library ) + monkeypatching without changing existing code base

So what if you have an existing python code base but you are unable to scale it because of varoius dependencies like blocking I/O, network connections e.t.c

Look into eventlet !!! (thanks to my co-worker who suggested not looking into twisted to solve the problem since it was just plainly painful .. but look into eventlet)

easy to install , easy to integrate into existing code base ( if its not too late and if you are not using any C based libraries in your python code base)

They call it "greening"

So I ran into a similar blocking issue and the way i greened my  application is simply by monkeypatching the standard library.

pip install eventlet

 

Then add these lines as soon as your programme starts ( to avoid late binding problems) 

import eventlet
eventlet.monkey_patch() 

Now try to think how on highlevel you can sort of create threads on the blocking process and spawn threads of that blocking call.

( for ex:- lets say the blocking function  has a network connection or some other blocking I/O and it is called foo(p1,p2) and it is using a standard python library  )

 pool = eventlet.GreenPool()

 pool.spawn_n(foo, p1, p2)

 pool.waitall()

voila! you are async again!!! no matter how much time foo takes to complete .. you can move ahead :)

Metrics on your way : What do you when you want to see statistics of you project (graphite+statsd)

Okay so you have project working spik and spank .. but now you want to monitor the health of your project . you want to know how much data is it handling? how well is it handled? how long does this call take? how many success and how many failures? e.t.c e.t.c

Basically its like getting the health stats of your project .. so how do you do it ?

These are the phases my project saw :-

1. we all start with good old pint statements :)
2.  at some point they have to be removed .. so introduced logging.
3.  big log files were generated .. we needed better understanding of all this data.
4.  Splunk(www.splunk.com) cam to rescue .. feed it any log file and put in proper   queries and it will give you nice data. graphs., trends e.t.c
5. Now if  splunk and big log data file is the concern .. I introduce a database (sqllite) ..did some smart queries and displayed is neatly on a django dashboard. worked as a charm ( i still love this option) and looked very professional but then scalability issues alas!
6.  The came "statsd+graphite" ..Generate counters/stats using stasd. Run statsd, configure and feed it to graphite ..run graphite and watch the magic.

Here are very nice instructions to " installing and configuring statsd and graphite" thanks digital ocean "https://www.digitalocean.com/community/tutorials/installing-and-configuring-graphite-and-statsd-on-an-ubuntu-12-04-vps"

I used the python "statsd" library and to use it was as simple as that :-

>>> pip install statsd
>>> import statsd
>>> c = statsd.StatsClient('localhost', 8125)
>>> c.incr('foo')  # Increment the 'foo' counter.
>>> c.timing('stats.timed', 320)  # Record a 320ms 'stats.timed'.

one issue which i hit during the installation and configuration part was a twisted error of "unknown command : carbon-cache"

more about it here http://stackoverflow.com/questions/27951317/install-graphite-statsd-getting-error-unknown-carbon-cache?noredirect=1#comment44298261_27951317

I was able to bypass is by manually deleting twisted ( i was not using twisted anywhere)

when i ran graphite locally i was abel to see my stats where and graph them.

Pretty cool!

Happy new year

Here are few notes to self:-
1) learn more new stuff
2) openstack
3) more python practice
4) more outreach "girls/women in tech"
5) more volunteering
6) network more and increase visibility
7) have fun!
8) take some relevant Moocs

Push notifications on APNS , GCM and MPNS(WNS)

If you need to send Push Notifications for your application you will realize that you need to support a number of device types( ios, windows, android to name a few)

Its interesting how push notification differ and resemble on these different platforms.

One option is to use things like urbansirship and outsource the job.

The other option is to build on free python libraries to send messages to apps like pushco, nma(notify my android), growl(windows)  e.t.c


But if you need to send pushnotifications to your applications things are different.

You will need to look into APNS, GCM/C2DM, MPNS/WNS  e.t.c  (follow the links for details on them) and if you are looking for a python library for windows notification https://github.com/Neetuj/python-wns


I will be concentrating on the sending part of the push notification in this post.
....more to come



Comparision of  pushnotification  on different platforms(apart from the attachment):-


ID:-
      GCM:-  Android requires the device to contact its respective push message provider (GCM ) to receive a special string Device ID   identifies a specific app on a specific device. 183-byte string [a-zA-Z0-9_\-]
      APNS:-  ios requires the device to contact its respective push message provider (APNS ) to receive a special string Device token,  identifies a specific app on a specific device.64-byte hex-string
      MPNS:-  Windows requires the device to contact its respective push message provider (MPNS ) to receive a special string URI,  identifies a specific app on a specific device.


Message:-
        GCM:- json paylaod  (registration_ids, data=None, collapse_key=None, delay_while_idle=False, time_to_live=None, is_json=True, dry_run=False)
        APNS:-string or dictionary (alert (body, action-loc-key,loc-key,loc-args,launch-image)  , sound, badge,  token)
        MPNS:-string

Feedback:-
        GCM:- return success/failure
        APNS:- no response .. listen to APNs feedback server for deprecated tokens
        MPNS:- http_status_code will be returned for success/failure

Limitations:-
        GCM:- payload size<=4 KB
        APNS:- payload size<=256 bytes (alert 75-100 bytes)  ios8+ it was increased to 2 kb
        MPNS:- payload size<=3kb

Customized data:-
        GCM:- send in json message
        APNS:- "extra" key is there is space
        MPNS:-  "param" key (string)





Side note:- APNS disconnects if there are failures so listen to feedback server and try to get a persistent connection




more here (http://csce.uark.edu/~cwt/COURSES/2014-01--CSCE-4543--SW-ARCH/03--CHAPTERS/Chapter%2004--Push%20Notification%20Services--Cleaver.pdf)