logstash -forwarder on centos7: how to run as a service

If you want to run logstash-forwarder agent to run on your vm everytime it comes up you will have to jump through a lot of hoops :)

here are some of them ( hopefully this is all you will need)

     yum install git
     git clone https://github.com/elasticsearch/logstash-forwarder.git
     cd logstash-forwarder/
     go build

     gem install fpm pleaserun
     make rpm

     yum install rpm-build
     make rpm

    yum install logstash-forwarder-0.4.0-1.x86_64.rpm
    vim /opt/logstash-forwarder/logstash-forwarder.conf

   yum install supervisor
   systemctl enable supervisord
   echo_supervisord_conf > /etc/supervisord.conf
   sudo /usr/bin/echo_supervisord_conf > supervisord.conf
   vim supervisord.conf
   sudo cp supervisord.conf /etc/supervisord.conf
   sudo cp supervisord.conf /etc/supervisor.d/supervisord.conf

Edit the  logstash-forwarder.conf with server info and the cert, key files e.t.c (make sure you scp the cert and key file)

{
 "network": {
  "servers": [ "serverip:port" ],
  "ssl ca": "/opt/logstash-forwarder/ssl/logstash-forwarder.crt",
  "ssl key": "/opt/logstash-forwarder/ssl/logstash-forwarder.key",
  "timeout": 15
},
"files": [
 {
   "paths": ["/home/centos/ubtest.log"],
   "fields": { "type": "staging" }
 }
]
}

Edit the supervisor.conf with this in the end

[program:logstash-forwarder]
command=sudo /opt/logstash-forwarder/bin/logstash-forwarder -config=/opt/logstash-forwarder/logstash-forwarder.conf
stdout_logfile=/var/log/logstash-forwarder.log
stderr_logfile=/var/log/logstash-forwarder.err
autorestart=true
autostart=true
user=root
directory=/opt/logstash-forwarder 

How to monitor your vms ? use zabbix

If you want to monitor your vms at one central place.
Use Zabbix

1) Bring up a zabbix server

https://www.digitalocean.com/community/tutorials/how-to-install-zabbix-on-ubuntu-configure-it-to-monitor-multiple-vps-servers

2)  install zabbix-agents on your vms and edit the (hostname, server, serveractive) fields in config files

    rpm -Uvh http://repo.zabbix.com/zabbix/2.2/rhel/7/x86_64/zabbix-release-2.2-1.el7.noarch.rpm
     yum install zabbix zabbix-agent
     vim /etc/zabbix/zabbix_agentd.conf 

3) configure the agents to run everytime the vm comes up

     systenctl start zabbix-agent.service
     systemctl start zabbix-agent.service
     systemctl status zabbix-agent.service
     ps -ax|grep zabbix

4) add your hosts to zabbix servers  now you can add some active checks(to push data) if you want else zabbix server will keep polling agent

5) Note:- if you want to automate the hostname and do not want to manually go edit it .. use HostnameItem = system.host   instead of Hostname

Image and openstack : how to get a centos7 image and change its credentials?

So I wanted  to create vms with centos7 on it so the steps I thought i would take was

$ nova boot --flavor FLAVOR_ID --image IMAGE_ID --key-name KEY_NAME \
  --user-data USER_DATA_FILE --security-groups SEC_GROUP_NAME --meta KEY=VALUE \
  INSTANCE_NAME

Now it seemed logical to look for IMAGE_ID in

$ nova image-list  

$glance  image-list

But the catch is centos image was not there

you can find it here :-

http://cloud.centos.org/centos/7/images/

$glance image-create imageName --location IMAGE_URL

Now go ahead get the image-id and create a vm as specified above.

Next thing you will run into is credentials. How do i log into the box ( lets say if you can not ssh for some reason into it)

Cloudinit comes to rescue a simple

mydata.file

$ nova boot --image ubuntu-cloudimage --flavor 1 --user-data mydata.file

mydata.file will look like this

#cloud-config
password: yourpwd
chpasswd: { expire: False }
ssh_pwauth: True

Now you can loginto your box with these credentials

 username:centos
 password:yourpwd