Beginners guide to cloud and openstack

Disclaimer :-- all the content is sourced from various pages on openstack .. I am just putting together that information in a crash course :)

 

What is Cloud?

Let's quickly review just what exactly we mean by cloud technologies?
 Cloud technologies are built on existing technologies such as virtualization and clustering to virtualize hardware, software, storage, and networking resources into flexible units that are quickly allocated to meet demand. So rather than the old static model of dedicated hardware servers for various tasks, and static network and storage configurations, all of those formerly specialized devices are assimilated into a common resource pool. It's a more efficient use of hardware, and very fast to scale up or down according to demand. You can even configure self-service for users so they can grab whatever they need when they need it.
Types of cloud ?

• Private
• Public
• Hybrid
• 
  
 

 

All computing resources are shareable in a cloud, and there are three basic service models:

• SaaS, software as a service
• PaaS, platform as a service
• IaaS, infrastructure as a service

What is Openstack? 
Open Source Cloud Computing platform (short answer).
OpenStack is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a datacenter, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface.
https://en.wikipedia.org/wiki/OpenStack (long answer)

What are  Openstack components? 

core projects (short answer):

• Identity (Keystone)
• Dashboard (Horizon)
• Orchestration (Heat)
• Metering (Celiometer)
• Object stroage (Swift)
• Image service (Glance)
• Networking (Quantum now Neutron)
• Compute (Nova)

https://en.wikipedia.org/wiki/OpenStack (not so long answer)
https://www.openstack.org/software/ (long answer)

How do these Openstack Components talk to each other?

1. Messaging Queues: AMQP is the messaging technology chosen by the OpenStack cloud.The OpenStack components such as Nova , Cinder , Quantum communicates internally via AMQP(Advanced Message Queue Protocol) and through eachother using REST-call. The AMQP broker, RabbitMQ , sits between any two internal Openstack components and allows them to communicate in a loosely coupled fashion i.e its components has, or makes use of, little or no knowledge of the definitions of other separate components. More precisely, Nova components (nova-api, nova-scheduler , nova-compute) use Remote Procedure Calls (RPC) to communicate to one another.
   In OpenStack the Nova ,Cinder and Quantum implements RPC (both request+response, and one-way, respectively nicknamed ‘rpc.call’ and ‘rpc.cast’) over AMQP by providing an adapter class which take cares of marshaling and unmarshaling of messages into function calls. Each Nova components (for example api ,compute, Scheduler, etc.) , Cinder components ( for example volume, Scheduler) , Quantum Components( for example quantum-server , agents ,plugins )  create two queues at the initialization time, one which accepts messages with routing keys ‘NODE-TYPE.NODE-ID’ (for example compute.hostname) and another, which accepts messages with routing keys as generic ‘NODE-TYPE’ (for example compute) .
2.  

Beginner's links to find out about openstack

https://opensource.com/business/14/2/openstack-beginners-guide

https://eloiguillaud.wordpress.com/2014/10/12/cloud-computing-solutions-comparative-study-openstack-13/

https://www.linux.com/learn/docs/720624-introduction-to-openstack-from-zero-to-domination

https://www.linux.com/learn/tutorials/866101-how-to-test-drive-openstack

https://wiki.openstack.org/wiki/Main_Page

playing with openstack HEAT : beginners introduction

• Install heat
• Heat templates reside here

1.  https://github.com/openstack/heat-templates
2. which can be seen on the vm its in /vagrant/heat_templates
3. there are test templates in root dir as well ex:- /root/heat_test_template.yml

• on the controller vm this is how to use heat to spin up nodes
  
  1. heat stack-create -f heat_templates/test.yml test
  2. once its created you should be able to see it using CLI like this :-
     [root@controller-01 vagrant(keystone_admin)]$ heat stack-list
     
      +--------------------------------------+------------+---------------+----------------------+
     | id                                   | stack_name | stack_status  | creation_time        |
     +--------------------------------------+------------+---------------+----------------------+
     | xxxx-xxxxx | test       | CREATE_FAILED | 2015-11-16T20:59:49Z |
     +--------------------------------------+------------+---------------+----------------------+
  3. on horizon you can see it here 
  
  

          7.  If you click the "stack name" link you can get into more details of your stack.
          8. There are Actions like "resume stack, suspect stack , delete stack" e.t.c that you can perform
         9.  You can get different details on the stack on horizonw including the template itself and you can change the template
     

• General Heat Resources :-

http://docs.openstack.org/developer/heat/template_guide/

https://wiki.openstack.org/wiki/Heat

https://github.com/openstack/heat

why use HSM(Hardware Security Module) ? What to look for in a HSM?

Let's answer first why ? Why a HSM should be used?
We all know security has sort of become a myth. Specially in today's interconnected, big data, cloud crazy world. The questions is How to secure data ? documents ? keys? passwords?

Very sensitive applications have to protect several different forms of data and in some industries (banking, health, complaint heavy ones) we absolutely can not overlook the possibility of security compromises.

Solutions with various encryption technologies came up when the field of security was burgeoning but they were only safe as long as the keys are safe . How do you keep the keys safe? Store your software key in some software? NAAA!! that won't work.
Store them in tapes like Millitary ? NAAA!! that won't work
Store them in a tamper proof hardware with multiple level of authentications? May be !!  atleast to some extent!

That's where Hardware Security Module HSM comes into play (added bonus it can help you clear  your industry's security compliance needs )

Now that we know why ( and there are more use cases here  if you still need more reasons) the nest question is what to look for in an HSM?

Lets list the technical factors that might help  :-

• Performance -  encryption/decryption/key generation/signing, symmetric, asymmetric, decide based on your application needs
• Redundancy - consider the failover cases, how often can it happen? How easy is it to replace it? Assess the impact of it on your operations and have a plan in place and in case that happens or look for a more robust one. Dual NIC or not , down time e.t.c are questions to be asked.
• Backups - No matter how robust HSM is you can not afford to not think about backups because of the very real possibility of loosing access to your data forever. Consider how easy it is to backup your data (in a secure way ofcourse) , restore your data in HSM. 
• Scalability -  Most HSMs have limit on slots available, number of keys you can stores, number of users you can add e.t.c e.t.c Have the scale in mind. What if you need to scale up or down ( for cost cutting) 
• Procedures - Lets face it .. using HSM is not gonna make things simple .. its gonna make them more complex.  The real cost of using HSM is felt in operations because of the complexity that is introduced.A good HSM will come with a detailed usage manual which describes how things should be done to make sure its used correclty. It's not the hardware which matters, but how you use it. So consider APIs, Device management options , customer support , policy level controls , configuration options  e.t.c
• Physical/Logical Security -  Ofcourse all HSMs should be able to tackle physical or logical hacking in to the hardware/software of HSM. Most HSMs as a result have tamper proof cases, hardcase ,  multiple level of authentications and "wipe everyhting" attitude for intrusion attacks. 
• Complaince - If you have any compliance requirements make sure HSM supports it.

Mixed expereinces of my first hackathon last year: How AT&T can legally steal your idea and own it

I wanted to share my unexpected and mixed experiences from my hackathon last year.
 http://www.eventbrite.com/e/att-mobile-app-hackathon-dallas-connected-car-tickets-12156796305

I attend that hackathon in aug 2015..pitched an idea for connected car theme "how about smart cars save babies in locked in carseat when parents are away from the carseat?"  we called it "childcheck" and talked about sensors detecting temperature rise and visual monitoring of the car and sending voivemails/pusnotifications/alerts to car owners( parents)
 and in 2015 a year later
At&t comes up with the same idea :)
http://mobile.pcmag.com/news/56056-at-and-ts-hot-car-sensor-could-save-babies-lives?origref=http:%2F%2Fabout.att.com%2Finnovation%2Ffoundry

and here are we presenting out project on the same idea at At&T hackathon

https://www.facebook.com/attdeveloper/photos/a.757760704270652.1073741892.151603081553087/757760794270643/?type=3&theater

so this is how it started:-
I was always cribbing not to be able to go to these cool weekend hackathons since the other half of me would strictly instruct me to spend all my weekend with my darling son .. who does not get enough of me on weekdays :) (neither do i get enough of him)

One day one of my good friend tells me that she quit her job and she is gonna look into starting a startup. She asked me if i was interested ( in working/investing) to which i was not very receptive ( good thing it had no impact on your friendship) and  she also asked if I would be interested in coming to this hackathon taking place over the weekend in Dallas. She is not a techie (she is the queen of business ;)) and i though to myself .. if she is going why don't i give it a shot ? ( I will get a ride with her as well ;))

So that's how I checked one more thing on my bucket list .. "attend a hackathon".

Since i had absolutely not idea about the hackathon i was going to .. I was totally unprepared. I did not have an idea .. and I just assumed I will just join somebody's  team  and work on their idea (whichever idea appealed to me). We had just bought a new laptop ..I ripped its packaging took it to the hackathon.

This was the hackathon we attended :- AT&T connected car hackathon 2014
 http://www.eventbrite.com/e/att-mobile-app-hackathon-dallas-connected-car-tickets-12156796305

Once we reached there I found a lot of folks ..sitting around .. not talking too much and still mingling around with "Hi" and Hellos.  I was happy to see guacamole and chips ( I was hungry !!)  I think there are many others like me who would go to hackathons jsut for free food ;) ( if only they served good vegetarian food and did not have families to take care of)

As per my nature .. and since I had not looked into the topic at all before ..I started asking people questions.."what is a connected car?", "what can a connected car do?" " do we have a cpapabilities document on connected car?" "what is your idea?" e.t.c e.t.c   Sometimes I got smirks, sometimes unrelated comments and sometimes a big laugh .. and out of all those was the most useful comment "its a hackathon .you can make the connected car do whatever you want .. hypothetically speaking..nobody has specs/asnwers/documentation e.t.c in a hackathon ..just a theme"

That comment was pretty revealing and got my brain cells racing .. then came the pitch time and there were a handful of ideas ( i guess 4-5 ) that were pitched .. i was suprised since there were soo many people out there i expected more ideas i could choose from.
and the sad part was that none of the ideas were on which i would want to work on .. since most of them were jsut about more and more data consumption. "A freind is going on a road trip .. this idea is to make the song playlist of the road-tripper crowd sourced from his friends"  and there were  3-4 more ideas along the same line of though .. "speeding and cop around ..raise an alert" e.t.c  Later on i was told that not everybody who comes with an idea lieks to pitch it .. they like to keep their idea secret and work in secret .. ( to me that totally did not make sense ..since I though attendees came here to work together on some idea ..not in silos) ..Respectfully all were nice ideas but none good enough to entice me  and now I had to make a choice .. either i work for their idea ( which i do not like ) or I leave or the last option is I come up with a idea i like and pitch it!!

I took the last option and since connected car can do anything you want ..had my brain cells still racing my idea was ..."can we make a smart car realize/detect there is a baby in the car locked in high heat and alert the parent?"  I scoped it out more while pitching .. How about we put something in place in baby's carseats to detect that baby is in the car alone and the car is locked and may be a sensor on car to detect car's temperature and then If all these conditions are satisfied then "extreme temp+baby in locked car alone" -> send message to car's oswner's cellphone (text/call/pushnotification/anything)

I gathered all my courage and pitched the idea since i know as a mother .. this was my biggest fear.. so many babies die every summer in texas in a matter of minutes because of being locked in car accidentally .. specially for new parents exhaustion is very real .. you are soo exhausted/absent mindede with soo many things to take care of it is a very realistic scenario that you forget that you have a little passenger in the back.  By the time you come back its too late .. sometimes the temperature rises too quickly and it happens all too fast !!

Finally I was the last one to pitch this new minted idea by my brain and people heard it .. some complained that i am not speaking loud enough .. so I shouted on top of my voice .. the idea (with passion :) )  ...Could not entice many .. some looks were encouraging but not many came forward . I assumed nobody is interested. My friend was .. and she said she could market the idea :) ..then 3 more folks came (all parents i think ..though i did not ask .. but who could relate to the idea i guess)

AND we have a team!! I was happy .. we all went home to start working on the idea next morning. ( while the winning team stayed overnight and worked on the idea right away :))

Next day we worked on the idea .. I must admit we did make good progress but not a stellar progress on the idea I admit and I realized how true the saying is "ideas are dime a dozen .. its the execution that matters".
By the time it was demo time in the evening we were able to "simulate temperature rising in a car" through a script and send pushnotification and place calls  one a given cellphone number when the temperature crosses the threshold. We had many other things we wanted to work on "a thermal sensor in to place in car" for temperature detection , "a visual sensor in car" (camera?) to detect baby in the car seat. (later one some people suggested we should also consider pets for this idea ..though detecting a baby was easier since baby would be in car seat visual matching is easier) ...but there is limited time and energy.

When the time for presentations came .. it was extremely interesting to see what other people had some up with. even though the ideas in my opinion were not that impressive .. some of the exceution of ideas totally blew me over .. like wow .. they did all this in last 24 hrs? amazing!!
Then came a team of 2 girls .. who basically pretty much copied our idea and implemented the hardware side of it. I could not believe my eyes!! None of us could .. they blatantly copied our idea ..put a spin on it and basically demo'd who their sensor with detect a change in temperature in the car.

I felt sooooo disappointed .. are like back to high-school .. with petty jealousies and competition wars ? why the hell did these 2 girls not join our team rather than copying( or should i call stealing) the idea? we had the software side figured out ..they had the hardware side .. we could have actually presented  and working end to end solution!!!

That was my biggest experience .. that no matter your age .. deep inside we are all insecure scared and competitive :)

we did not win  .. the idea with the crowd-sourced play list on a road trip won ..  http://www.eventbrite.com/e/att-mobile-app-hackathon-dallas-connected-car-tickets-12156796305


I wished there were more ideas on solving real-life problems .. rather than jsut more data consumption apps :( but then who am I to judge ? FB, twitter all are succesfull.

  I think in these things, more realistic and practical ideas don't get the attention or winning results but only those that seem cool do.

The most interesting thing is that recently I came across this :)
:-

http://mobile.pcmag.com/news/56056-at-and-ts-hot-car-sensor-could-save-babies-lives?origref=http:%2F%2Fabout.att.com%2Finnovation%2Ffoundry


Is it too much of a coincidence ?? That i pitch the same idea a year ago in a AT&T hackathon and then they come up with a technology on the same idea? Or are ideas everywhere dime a dozen??

Coming across this post is what actually made me realise that i should have written this post last year rather than this year .. but then as somebody wisely said i guess ..its the execution that matters and I am glad at&t took the idea ahead ..

Lessons learned from my first hackathon:-

1) be discreet about your idea
2) go well prepared with softwares , IDE e.t.c that you will need
3) may be not go to hackathons .. to reveal your real idea but just learn other's ideas and execute their ideas (since they might never find out)
4) blog about your idea .. before the company that sponsored your hackathon claims it as theirs.
5) Get some students on yrou team who can work overnight on your idea ;)

Safenet Hardware Security Module configuration for the first time.

Safenet HSM comes with a lot of documentation .. which is good .. the sad part is  I found it a little too verbose.
if you want to read it all here is the link
http://cloudhsm-safenet-docs-5.3.s3-website-us-east-1.amazonaws.com/007-011136-006_lunasa_5-3_webhelp_rev-c/startpage.htm

If you juts got the hardware and want to know how to configure it quickly here are the steps

1. connect to the serail console 
2. login with default appliance credentials if you are logging in for the first time (admin , PASSWORD)
3. Change Password:as soon as you do that you will be asked to change the password ( atleast 3 characters from the these categories letters, numbers, special characters)
4. using h or ? you can get a list of commands
5. You can change/recover passwords (Note is this "appliance admin password" write it down carefully)
6. Sett time zone Next check date, time, zone :-
   lunash:>status date
   lunash:>status time
   lunash:>status zone
7. if its incorrect set it up correctly using :-
   lunash:>sysconf timezone set America/Chicago
   *** note above 7 steps will be done the first time HSM is powered on****
8. configure network (hostname, domain, interfaces)
   lunash:> network hostname yourhostname 
   lunash:> network domain yourdomain
   lunash:>net interface  -device eth0 -ip 1.1.1.1  -netmask 255.255.255.255 -gateway   1.2.2.3.
9. verify network  configuration (Note: make sure links are up..else go check on the eth port connections, ping ips to verify )
   lunash:>net show
   
   net-show

   
   

     [yourHSM] lunash:>net show                        
   
   
      Hostname:          "yourhostname"
      Domain:            "yourdomain"
   
      IP Address (eth0): 1.1.1.1
      HW Address (eth0): 00:00:00
      Mask (eth0):       255.255.255.255
      Gateway (eth0):    1.2.2.3
   
      Name Servers:      10.0.80.11        10.0.80.12    
      Search Domain(s):  softlayer.local 
   
   Kernel IP routing table
   Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
   1.1.1.1        0.0.0.0         255.255.255.255          U     0      0        0 eth0
   1.0.0.0        1.2.2.3          255.0.0.0                  UG    0      0        0 eth0
   
   Link status
     eth0: Configured   
           Link detected: yes
   
     eth1: Not configured
10. Generate certificate: Certificate can be generated following ways (use the ip if you do not have DNS configured)
    lunash:>sysconf regenCert
    lunash:>sysconf regenCert 11.1.1.1
11. lunash:>ntls bind eth0

Now if you want you can go further and start playing with it.

1. Initialize HSM :
   lunash:>hsm init -label myhsm -domain thisiscliningdomain -password *** -force
2. Create Partition:
   lunash:>hsm login
   lunash:>partition create -partition myPartition1
    

Hardware Security Module :- lock your master key and never let it leave

A hardware security module (HSM) is a physical computing device that safeguards and manages digital keys for strong authentication and provides cryptoprocessing. These modules traditionally come in the form of a plug-in card or an external device that attaches directly to a computer or network server.  (from wiki)




A hardware security module (HSM) is a physical device that provides secure storage for encryption keys. It also provides secure computational space (memory) to perform encryption and decryption operations.

HSM provides secure cryptographic key storage and operations within a tamper-resistant hardware device.HSM is worth looking into ,Whenever security, compliance, key management e.t.c are of utmost importance.
A hardware security module can be employed in any application that uses digital keys. Typically the keys must be of high-value - meaning there would be a significant, negative impact to the owner of the key if it were compromised.
The functions of an HSM are:

• onboard secure cryptographic key generation
• onboard secure cryptographic key storage and management
• use of cryptographic and sensitive data material
• offloading application servers for complete asymmetric and symmetric cryptography.

Here we list some uses cases, practical/possible applications of HSM :-

• Database -TDE( transparent data encryption):-
  Several commercial database engines support a feature called transparent data encryption (TDE) that can seamlessly encrypt the data in a database.Transparent data encryption can use HSM to provide enhanced security for sensitive data. An HSM is used to store the master encryption key used for transparent data encryption. The key is secure from unauthorized access attempts as the HSM is a physical device and not an operating system file. All encryption and decryption operations that use the master encryption key are performed inside the HSM. This means that the master encryption key is never exposed in insecure memory.
• Cryptographic accelerator:-
  HSM delivers significant performance benefits and reduces hardware costs. HSMs can provide significant CPU offload for asymmetric key operations. Can not compete with hardware-only solutions for symmetric key operations though. Performances ranges from 1 to 7,000 1024-bit RSA signs/second.
•  Compliance sensitive industries:-
  wherever data is sensitive , critical and highly personal it needs to be encrypted. Healthcare industry , Legal industry, Finance industry, government, banks... all have to deal with lot of compliance clearance to make sure their data/documents/information is all using high standard encryption and key management standards.. HSMs can help in enforcing PCI DSS, PCI PTS compliance , DES compliance, FIPS compliance , HIPAA compliance,  TR-39 compliance. Mostly all follow the sign and decrypt data, but the private keys are located at a HSM model
• Certificate Management:-
   HSM  can also be used as service to generate certificates, sign certificate signing requests (CSRs), and to store private keys used with certificates. The CloudHSM is typically used as an architectural building block and root of trust in these applications
•  EMV data preparation and card personalization
•  Remote key loading for ATM networks
•  Point-to-Point Encryption (P2PE) of cardholder data
•  Credit, debit, and prepaid card fraud prevention
•  MAC calculation to ensure integrity of data in transit and at rest
•  Dynamic key exchange for Point of Sale and ATM
•  PIN translation and verification
•  One-time password generation for online security :-
  Most OTP (One Time Password) solutions are based on a long secret random seed value. Somehow this needs to be stored at the validating server as well to be able to verify the submitted OTP value.
  In a way these seeds have the same role as user supplied passwords and need to be stored equally secure. Salting & hashing will not work here as this will break the OTP algorithm.
  Most small OTP-tokens are physically secured by being more or less tamper-proof but this does not apply to the server.Do all cryptographic computations in the HSM
• SSL offloading

• Sample Applications:

  • PKI key generation & key
  • Storage (online CA keys & offline CA keys)
  • Certificate validation & signing
  • Document signing
  • Transaction processing
  • Database encryption
  • Smart card issuance

  - See more at: http://www.safenet-inc.com/data-encryption/hardware-security-modules-hsms/luna-hsms-key-management/luna-sa-network-hsm/#content-left
  
  
  

python-wns to send push notifications to windows

Hey folks

Here is an easy way to send push notification to windows mobiles.

https://github.com/Neetuj/python-wns

This is a follow up for my previous post

http://workstuffcareer.blogspot.com.br/2014/11/push-notifications-on-apns-gcm-and.html

and now you can install it from pip

pip install python-wns

When I was working on push notifications it was interesting to see how there are packages for GCM , IOS..andriod apple phones but none for windows phones
well yeah windows has earned a bad name in the opensource world but lets me nice to the new guy in our world :) I guess they are trying to make friends in here .. so here is my welcome to them warm wishes from python-wns

How to do the reality check on all of these "click based" data/social media analysis and predictions?

I got a great opportunity to attend pydata conference ( thanks to pyladies for supplying the free ticket and my manager to let me attend it :))

It was a cool conference with great workshops, talks and presentations!  (liked it more than the pytexas conference last year :)). Shout out to the organisers!



Heard many  great/innovative talks about data analysis and predictions and machine learning based on that analysis-prediction model and the privacy concerns that come with open data  at Pydata-dallas conference .. but not much on how to tackle/prevent the false predictions, their negative impacts, how are we doing the reality check on these "click-based analysis"

How do we predict that out of these 20 people only 2 are gonna  show up on your meetup .. based on theit history ?

How do we predict and take out clicktivists from activists .. to make sure we are not  grossly overestimating out support in this cause .. to be able to plan accuratly ?


How do you make sure doing a search on pressure cooker .. will not land you in some "special files" because of some social media analysis algorithm's incorrect prediction ?


How do we make sure that X's brand/popularity is not just "click popularity" ?

How do we establish actual learning assessments with Moocs?

Now  the next question that comes to my mind is ..

 Is it really that easy to manipulate the market sentiments by just some clicks?

Many  invest in shares/funds e.t.c largely based on the market sentiment , brand's perception .. is it really that volatile ? 

Install barbican

Barbican is a ReST API designed for the secure storage, provisioning and management of secrets. It is aimed at being useful for all environments, including large ephemeral Clouds. All documentation and work can be found on either Launchpad or Github at the following locations:

• Source Code
• Launchpad
• Wiki
• Contributing

Step-by-step guide for ubuntu

For some reason I ran into issues while installing Barbicun from these instructions(https://github.com/cloudkeep/barbican/wiki/Barbican-Quick-Start-Guide) hence I am adding here what all I did to make it work

1. Install prereqs 

   sudo apt-get install python-dev
   sudo apt-get install libsqlite3-dev

   sudo apt-get install libxml2-dev
   sudo apt-get install libxslt1-dev
   sudo apt-get install libbz2-dev  libldap2-dev  libsasl2-dev
   

   sudo apt-get install libffi-dev
   sudo apt-get install libssl-dev

2.  Install pyenv
   curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
3.  install  pyenv-virtualenv
   

   git clone https://github.com/yyuu/pyenv-virtualenv.git ~/.pyenv/plugins/pyenv-virtualenv

   echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash_profile

   exec "$SHELL"

4. pyenv install 2.7.5
5. pyenv virtualenv 2.7.5 barbican27
6. pyenv shell barbican27
7. bin/barbican.sh install
   Note:- if you run into any "module not found" kind of error .. means its time to restart . remove virtualenv , install what is missing (in my case it was biz2 so installing libbz2-dev helped) and then redo all the steps from step 2.
8. you can test step 7 once you see " [emperor] vassal barbican-api.ini is ready to accept requests"  by "curl -H 'X-Project-Id:12345' localhost:9311"
   response looks like {"v1": "current", "build": "2015.1.dev103"} on client
   and "[emperor] vassal barbican-api.ini is now loyal" on server side ( somebody knows what's good humour ) out there

Step-by-step guide for centos

For some reason I ran into issues while installing Barbicun from these instructions(https://github.com/cloudkeep/barbican/wiki/Barbican-Quick-Start-Guide) hence I am adding here what all I did to make it work

1. Install prereqs  
   

   sudo yum install openssl

   sudo yum install libffi-devel
   sudo yum install openssl-devel

   sudo yum install python-devel
   sudo yum install sqlite-devel 

   sudo yum install openldap-devel

2.  Install pyenv
   curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
3. if above does not work run this script https://gist.github.com/ysaotome/7956676 
4.  install  pyenv-virtualenv
   

   git clone https://github.com/yyuu/pyenv-virtualenv.git ~/.pyenv/plugins/pyenv-virtualenv

   echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash_profile

   exec "$SHELL"

5. pyenv install 2.7.6
6. pyenv virtualenv 2.7.6 barbican27
7. pyenv activate  barbican27
8. pyenv shell barbican27
9. bin/barbican.sh install
   Note:- if you run into any "module not found" kind of error .. means its time to restart . remove virtualenv (rm -rf /root/.pyenv/versions/barbican27/), install what is missing (in my case it was biz2 so installing libbz2-dev helped) and then redo all the steps from step 2.
10. you can test step 7 once you see " [emperor] vassal barbican-api.ini is ready to accept requests"  by "curl -H 'X-Project-Id:12345' localhost:9311"
    response looks like {"v1": "current", "build": "2015.1.dev103"} on client
    and "[emperor] vassal barbican-api.ini is now loyal" on server side ( somebody knows what's good humour ) out there

logstash -forwarder on centos7: how to run as a service

If you want to run logstash-forwarder agent to run on your vm everytime it comes up you will have to jump through a lot of hoops :)

here are some of them ( hopefully this is all you will need)

     yum install git
     git clone https://github.com/elasticsearch/logstash-forwarder.git
     cd logstash-forwarder/
     go build

     gem install fpm pleaserun
     make rpm

     yum install rpm-build
     make rpm

    yum install logstash-forwarder-0.4.0-1.x86_64.rpm
    vim /opt/logstash-forwarder/logstash-forwarder.conf

   yum install supervisor
   systemctl enable supervisord
   echo_supervisord_conf > /etc/supervisord.conf
   sudo /usr/bin/echo_supervisord_conf > supervisord.conf
   vim supervisord.conf
   sudo cp supervisord.conf /etc/supervisord.conf
   sudo cp supervisord.conf /etc/supervisor.d/supervisord.conf

Edit the  logstash-forwarder.conf with server info and the cert, key files e.t.c (make sure you scp the cert and key file)

{
 "network": {
  "servers": [ "serverip:port" ],
  "ssl ca": "/opt/logstash-forwarder/ssl/logstash-forwarder.crt",
  "ssl key": "/opt/logstash-forwarder/ssl/logstash-forwarder.key",
  "timeout": 15
},
"files": [
 {
   "paths": ["/home/centos/ubtest.log"],
   "fields": { "type": "staging" }
 }
]
}

Edit the supervisor.conf with this in the end

[program:logstash-forwarder]
command=sudo /opt/logstash-forwarder/bin/logstash-forwarder -config=/opt/logstash-forwarder/logstash-forwarder.conf
stdout_logfile=/var/log/logstash-forwarder.log
stderr_logfile=/var/log/logstash-forwarder.err
autorestart=true
autostart=true
user=root
directory=/opt/logstash-forwarder 

How to monitor your vms ? use zabbix

If you want to monitor your vms at one central place.
Use Zabbix

1) Bring up a zabbix server

https://www.digitalocean.com/community/tutorials/how-to-install-zabbix-on-ubuntu-configure-it-to-monitor-multiple-vps-servers

2)  install zabbix-agents on your vms and edit the (hostname, server, serveractive) fields in config files

    rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/7/x86_64/zabbix-release-2.2-1.el7.noarch.rpm
     yum install zabbix zabbix-agent
     vim /etc/zabbix/zabbix_agentd.conf 

3) configure the agents to run everytime the vm comes up

     systenctl start zabbix-agent.service
     systemctl start zabbix-agent.service
     systemctl status zabbix-agent.service
     ps -ax|grep zabbix

4) add your hosts to zabbix servers  now you can add some active checks(to push data) if you want else zabbix server will keep polling agent

5) Note:- if you want to automate the hostname and do not want to manually go edit it .. use HostnameItem = system.host   instead of Hostname

Image and openstack : how to get a centos7 image and change its credentials?

So I wanted  to create vms with centos7 on it so the steps I thought i would take was

$ nova boot --flavor FLAVOR_ID --image IMAGE_ID --key-name KEY_NAME \
  --user-data USER_DATA_FILE --security-groups SEC_GROUP_NAME --meta KEY=VALUE \
  INSTANCE_NAME

Now it seemed logical to look for IMAGE_ID in

$ nova image-list  

$glance  image-list

But the catch is centos image was not there

you can find it here :-

http://cloud.centos.org/centos/7/images/

$glance image-create imageName --location IMAGE_URL

Now go ahead get the image-id and create a vm as specified above.

Next thing you will run into is credentials. How do i log into the box ( lets say if you can not ssh for some reason into it)

Cloudinit comes to rescue a simple

mydata.file

$ nova boot --image ubuntu-cloudimage --flavor 1 --user-data mydata.file

mydata.file will look like this

#cloud-config
password: yourpwd
chpasswd: { expire: False }
ssh_pwauth: True

Now you can loginto your box with these credentials

 username:centos
 password:yourpwd

 

Debugging openstack .. not a cake!

So I have been playing with openstack for few days and as cool as it is ..i have to admit its a nightmare to debug.

so how do you start ?

1) find out all the major player nodes
2) go to each node and look into /var/log/
3) find out which one was written when your task failed

for examples

 I spinned up a vm and it was taking forever to come up.
 I logged into my compute node and found this

2015-02-18 10:07:26.586 3215 TRACE nova.compute.manager [instance: 0d28a0ad-93ce-43f5-b984-3c33f614861c] RemoteError: Remote error: OperationalError (OperationalError) (1048, "Column 'instance_uuid' cannot be null") 'UPDATE instance_extra SET updated_at=%s, instance_uuid=%s WHERE instance_extra.id = %s' (datetime.datetime(2015, 2, 18, 16, 7, 26, 576147), None, 152339L)

Now a friend pointed that this could be because of version mismatched.

So i logged into my controller node and checked the version

root@njain-compute:~# dpkg -l nova-compute
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================================================-==============================-==============================-============================================================================================================
ii nova-compute 1:2014.2.1-0ubuntu1~cloud0 all OpenStack Compute - compute node base

So i logged into my controller node and checked the version

root@control-\:~# dpkg -l |grep nova 
ii  nova-api                            1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - API frontend
ii  nova-cert                           1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - certificate management
ii  nova-common                         1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - common files
ii  nova-conductor                      1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - conductor service
ii  nova-consoleauth                    1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - Console Authenticator
ii  nova-novncproxy                     1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - NoVNC proxy
ii  nova-scheduler                      1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute - virtual machine scheduler
ii  python-nova                         1:2014.2.1-0ubuntu1~cloud0            all          OpenStack Compute Python libraries
ii  python-novaclient                   1:2.19.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

So i logged into my cinder node and checked the version

root@cinder-controller-mel01-1:~# dpkg -l|grep cinder
ii cinder-api 1:2014.2-0ubuntu1~cloud0 all Cinder storage service - API server
ii cinder-common 1:2014.2-0ubuntu1~cloud0 all Cinder storage service - common files
ii cinder-scheduler 1:2014.2-0ubuntu1~cloud0 all Cinder storage service - Scheduler server
ii python-cinder 1:2014.2-0ubuntu1~cloud0 all Cinder Python libraries
ii python-cinderclient 1:1.1.0-0ubuntu1~cloud0 all python bindings to the OpenStack Volume API

and as you can see cinder is out of sync .. and its version needs to match  1:2014.2.1


 well thats one way to debug(primarily since i was told version mismatch might be the issue)  though I wish debugging openstack was easier. 
Often people say reinstalling/rebuilding is better than debugging openstack and google is not yet abuzz with many clues .. here is to hoping!

 

Scale your projects with Eventlet ( concurrent networking library ) + monkeypatching without changing existing code base

So what if you have an existing python code base but you are unable to scale it because of varoius dependencies like blocking I/O, network connections e.t.c

Look into eventlet !!! (thanks to my co-worker who suggested not looking into twisted to solve the problem since it was just plainly painful .. but look into eventlet)

easy to install , easy to integrate into existing code base ( if its not too late and if you are not using any C based libraries in your python code base)

They call it "greening"

So I ran into a similar blocking issue and the way i greened my  application is simply by monkeypatching the standard library.

pip install eventlet

 

Then add these lines as soon as your programme starts ( to avoid late binding problems) 

import eventlet
eventlet.monkey_patch() 

Now try to think how on highlevel you can sort of create threads on the blocking process and spawn threads of that blocking call.

( for ex:- lets say the blocking function  has a network connection or some other blocking I/O and it is called foo(p1,p2) and it is using a standard python library  )

 pool = eventlet.GreenPool()

 pool.spawn_n(foo, p1, p2)

 pool.waitall()

voila! you are async again!!! no matter how much time foo takes to complete .. you can move ahead :)

Metrics on your way : What do you when you want to see statistics of you project (graphite+statsd)

Okay so you have project working spik and spank .. but now you want to monitor the health of your project . you want to know how much data is it handling? how well is it handled? how long does this call take? how many success and how many failures? e.t.c e.t.c

Basically its like getting the health stats of your project .. so how do you do it ?

These are the phases my project saw :-

1. we all start with good old pint statements :)
2.  at some point they have to be removed .. so introduced logging.
3.  big log files were generated .. we needed better understanding of all this data.
4.  Splunk(www.splunk.com) cam to rescue .. feed it any log file and put in proper   queries and it will give you nice data. graphs., trends e.t.c
5. Now if  splunk and big log data file is the concern .. I introduce a database (sqllite) ..did some smart queries and displayed is neatly on a django dashboard. worked as a charm ( i still love this option) and looked very professional but then scalability issues alas!
6.  The came "statsd+graphite" ..Generate counters/stats using stasd. Run statsd, configure and feed it to graphite ..run graphite and watch the magic.

Here are very nice instructions to " installing and configuring statsd and graphite" thanks digital ocean "https://www.digitalocean.com/community/tutorials/installing-and-configuring-graphite-and-statsd-on-an-ubuntu-12-04-vps"

I used the python "statsd" library and to use it was as simple as that :-

>>> pip install statsd
>>> import statsd
>>> c = statsd.StatsClient('localhost', 8125)
>>> c.incr('foo')  # Increment the 'foo' counter.
>>> c.timing('stats.timed', 320)  # Record a 320ms 'stats.timed'.

one issue which i hit during the installation and configuration part was a twisted error of "unknown command : carbon-cache"

more about it here http://stackoverflow.com/questions/27951317/install-graphite-statsd-getting-error-unknown-carbon-cache?noredirect=1#comment44298261_27951317

I was able to bypass is by manually deleting twisted ( i was not using twisted anywhere)

when i ran graphite locally i was abel to see my stats where and graph them.

Pretty cool!

Happy new year

Here are few notes to self:-
1) learn more new stuff
2) openstack
3) more python practice
4) more outreach "girls/women in tech"
5) more volunteering
6) network more and increase visibility
7) have fun!
8) take some relevant Moocs