Monday, April 27, 2015

How to do the reality check on all of these "click based" data/social media analysis and predictions?


I got a great opportunity to attend pydata conference ( thanks to pyladies for supplying the free ticket and my manager to let me attend it :))

It was a cool conference with great workshops, talks and presentations!  (liked it more than the pytexas conference last year :)). Shout out to the organisers!



Heard many  great/innovative talks about data analysis and predictions and machine learning based on that analysis-prediction model and the privacy concerns that come with open data  at Pydata-dallas conference .. but not much on how to tackle/prevent the false predictions, their negative impacts, how are we doing the reality check on these "click-based analysis"

How do we predict that out of these 20 people only 2 are gonna  show up on your meetup .. based on theit history ?

How do we predict and take out clicktivists from activists .. to make sure we are not  grossly overestimating out support in this cause .. to be able to plan accuratly ?


How do you make sure doing a search on pressure cooker .. will not land you in some "special files" because of some social media analysis algorithm's incorrect prediction ?


How do we make sure that X's brand/popularity is not just "click popularity" ?

How do we establish actual learning assessments with Moocs?

Now  the next question that comes to my mind is ..

 Is it really that easy to manipulate the market sentiments by just some clicks?

Many  invest in shares/funds e.t.c largely based on the market sentiment , brand's perception .. is it really that volatile ? 

Tuesday, April 7, 2015


Install barbican

Barbican is a ReST API designed for the secure storage, provisioning and management of secrets. It is aimed at being useful for all environments, including large ephemeral Clouds. All documentation and work can be found on either Launchpad or Github at the following locations:

Step-by-step guide for ubuntu

For some reason I ran into issues while installing Barbicun from these instructions(https://github.com/cloudkeep/barbican/wiki/Barbican-Quick-Start-Guide) hence I am adding here what all I did to make it work
  1. Install prereqs 
    sudo apt-get install python-dev
    sudo apt-get install libsqlite3-dev
    sudo apt-get install libxml2-dev
    sudo apt-get install libxslt1-dev
    sudo apt-get install libbz2-dev  libldap2-dev  libsasl2-dev
    sudo apt-get install libffi-dev
    sudo apt-get install libssl-dev
  2.  Install pyenv
    curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
  3.  install  pyenv-virtualenv
    git clone https://github.com/yyuu/pyenv-virtualenv.git ~/.pyenv/plugins/pyenv-virtualenv
    echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash_profile
    exec "$SHELL"
  4. pyenv install 2.7.5
  5. pyenv virtualenv 2.7.5 barbican27
  6. pyenv shell barbican27
  7. bin/barbican.sh install
    Note:- if you run into any "module not found" kind of error .. means its time to restart . remove virtualenv , install what is missing (in my case it was biz2 so installing libbz2-dev helped) and then redo all the steps from step 2.
  8. you can test step 7 once you see " [emperor] vassal barbican-api.ini is ready to accept requests"  by "curl -H 'X-Project-Id:12345' localhost:9311"
    response looks like {"v1": "current", "build": "2015.1.dev103"} on client
    and "[emperor] vassal barbican-api.ini is now loyal" on server side ( somebody knows what's good humour (smile)) out there


Step-by-step guide for centos

For some reason I ran into issues while installing Barbicun from these instructions(https://github.com/cloudkeep/barbican/wiki/Barbican-Quick-Start-Guide) hence I am adding here what all I did to make it work
  1. Install prereqs  
    
    sudo yum install openssl
    sudo yum install libffi-devel
    sudo yum install openssl-devel
    sudo yum install python-devel
    sudo yum install sqlite-devel 
    sudo yum install openldap-devel
  2.  Install pyenv
    curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
  3. if above does not work run this script https://gist.github.com/ysaotome/7956676 
  4.  install  pyenv-virtualenv
    git clone https://github.com/yyuu/pyenv-virtualenv.git ~/.pyenv/plugins/pyenv-virtualenv
    echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash_profile
    exec "$SHELL"
  5. pyenv install 2.7.6
  6. pyenv virtualenv 2.7.6 barbican27
  7. pyenv activate  barbican27
  8. pyenv shell barbican27
  9. bin/barbican.sh install
    Note:- if you run into any "module not found" kind of error .. means its time to restart . remove virtualenv (rm -rf /root/.pyenv/versions/barbican27/), install what is missing (in my case it was biz2 so installing libbz2-dev helped) and then redo all the steps from step 2.
  10. you can test step 7 once you see " [emperor] vassal barbican-api.ini is ready to accept requests"  by "curl -H 'X-Project-Id:12345' localhost:9311"
    response looks like {"v1": "current", "build": "2015.1.dev103"} on client
    and "[emperor] vassal barbican-api.ini is now loyal" on server side ( somebody knows what's good humour (smile)) out there