Can not write to web app directory - selinux issue

Is the content directory of some web app living somewhere that selinux is enforcing it policy? Perform the following to make directory writeable:

root@icinga2-p0 etc # semanage fcontext -a -t httpd_sys_rw_content_t "/etc/icingaweb2(/.*)?"


root@icinga2 etc # restorecon -Rv /etc/icingaweb2
restorecon reset /etc/icingaweb2 context system_u:object_r:etc_t:s0->system_u:object_r:httpd_sys_rw_content_t:s0
restorecon reset /etc/icingaweb2/setup.token context unconfined_u:object_r:etc_t:s0->unconfined_u:object_r:httpd_sys_rw_content_t:s0
restorecon reset /etc/icingaweb2/modules context system_u:object_r:etc_t:s0->system_u:object_r:httpd_sys_rw_content_t:s0
restorecon reset /etc/icingaweb2/modules/translation context system_u:object_r:etc_t:s0->system_u:object_r:httpd_sys_rw_content_t:s0
restorecon reset /etc/icingaweb2/modules/translation/config.ini context system_u:object_r:etc_t:s0->system_u:object_r:httpd_sys_rw_content_t:s0
restorecon reset /etc/icingaweb2/modules/setup context system_u:object_r:etc_t:s0->system_u:object_r:httpd_sys_rw_content_t:s0
restorecon reset /etc/icingaweb2/modules/setup/config.ini context system_u:object_r:etc_t:s0->system_u:object_r:httpd_sys_rw_content_t:s0

Update, compare kernel versions - reboot within 6hours period - CentOS

#!/bin/sh
host=`hostname -f`
date=`date`

# run updates to get latest software and kernel
yum -y update
maxdelay=$((6*60))
rdelay=$(($RANDOM%maxdelay))
sdelay=$(($rdelay*60))
kernel=`uname -r`
latest=`rpm -q kernel --queryformat '%{installtime} %{version}-%{release}.%{arch}\n' | sort -n -k1 | tail -1 | cut -d ' ' -f 2`
sdate=`date`

if [ "$kernel" = "$latest" ];
then
        echo "No need to reboot..."
        echo "You are running the latest kernel: $kernel"
else
        echo "There is a new kernel. You need a reboot..."
        echo "Current kernel: $kernel"
        echo "The latest kernel: $latest"
        echo "$sdate\nCurrent kernel: $kernel\nLatest kernel: $latest"
        echo "Sleeping for $sdelay seconds..."
        sleep $sdelay
        echo "Waking up and rebooting..."
        /sbin/reboot
fi

Install GRUB loader when Debian fails to boot

mount /dev/sdXY /mnt

mount --bind /dev /mnt/dev &&
mount --bind /dev/pts /mnt/dev/pts &&
mount --bind /proc /mnt/proc &&
mount --bind /sys /mnt/sys

chroot /mnt

grub-install /dev/sdX
grub-install --recheck /dev/sdX

Disable service start up at re/boot - Debian 6

The following is the chkconfig equivalents on Debian 6:

update-rc.d  <service> defaults
update-rc.d  <service> start 20 3 4 5
update-rc.d -f <service>  remove

Part of the man page:

UPDATE-RC.D(8)                                   sysv-rc                                   UPDATE-RC.D(8)

NAME
       update-rc.d - install and remove System-V style init script links

SYNOPSIS
       update-rc.d [-n] [-f] name remove

       update-rc.d [-n] name defaults [NN | SS KK]

       update-rc.d  [-n]  name  start|stop  NN  runlevel  [runlevel]...   .  start|stop NN runlevel [run‐
              level]...  . ...

       update-rc.d [-n] name disable|enable [ S|2|3|4|5 ]

DESCRIPTION
       update-rc.d updates the System V style init script links /etc/rcrunlevel.d/NNname whose target  is
       the script /etc/init.d/name.  These links are run by init when it changes runlevels; they are gen‐
       erally used to start and stop system services such as daemons.  runlevel is one of  the  runlevels
       supported  by  init,  namely, 0123456789S, and NN is the two-digit sequence number that determines
       where in the sequence init will run the scripts.

       This manpage documents only the usage and behaviour of update-rc.d.  For a discussion of the  Sys‐
       tem V style init script arrangements please see init(8) and the Debian Policy Manual.

CentOS 5 EOL - Yum repo issue

Now that CentOS 5 has gone end of life, there's no update repos left to use:

-bash-3.2# cat /etc/redhat-release
CentOS release 5.11 (Final)
-bash-3.2# yum update
Loaded plugins: downloadonly, fastestmirror
Determining fastest mirrors
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. Invalid release/
removing mirrorlist with no valid mirrors: /var/cache/yum/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base

All repo contents have been moved to http://vault.centos.org so to fix the problem, the other repos must be removed:

-bash-3.2# ls /etc/yum.repos.d/CentOS-
CentOS-Base.repo       CentOS-Media.repo      CentOS-Vault.repo
CentOS-Debuginfo.repo  CentOS-Sources.repo    CentOS-fasttrack.repo
-bash-3.2# ls /etc/yum.repos.d/CentOS-

Leaving only CentOS-Vault.repo:

-bash-3.2# yum update
Loaded plugins: downloadonly, fastestmirror
Determining fastest mirrors
Setting up Update Process
No Packages marked for Update
-bash-3.2#

Boot Linux OS into shell to change root password

Get access to the console. At the grub boot options edit boot line which has "kernel" string and append:

init=/bin/sh

Boot this and the Linux OS will boot to a shell. Mount / as rewrite:

mount -o remount,rw /

Change the root password:

passwd

Reboot Linux OS:

reboot

Fix Expired Certificate Issue on Ganeti cluster

When the certificate expires on a ganeti cluster you will get these error messages when starting the ganeti service:

(0) root@server ~
# gnt-instance list
Cannot communicate with the master daemon.
Is it running and listening for connections?
(1) root@server ~
# gnt-cluster getmaster
sever.gnt6.fog.priv
(0) root@server ~
# ls /var/lib/ganeti/ssconf_master_node
/var/lib/ganeti/ssconf_master_node
(0) root@server ~
# cat /var/lib/ganeti/ssconf_master_node
sever.gnt6.fog.priv
(0) root@server ~
# /etc/init.d/ganeti start
Starting Ganeti cluster:ganeti-noded...done.
ganeti-masterd...ERROR:root:RPC error in master_info from node X: Error 60: server certificate verification failed. CAfile: /var/lib/ganeti/server.pem CRLfile: none
ERROR:root:RPC error in master_info from node X: Error 60: server certificate verification failed. CAfile: /var/lib/ganeti/server.pem CRLfile: none
ERROR:root:RPC error in master_info from node Y: Error 60: server certificate verification failed. CAfile: /var/lib/ganeti/server.pem CRLfile: none
ERROR:root:RPC error in master_info from node Z: Error 60: server certificate verification failed. CAfile: /var/lib/ganeti/server.pem CRLfile: none
CRITICAL:root:Cluster inconsistent, most of the nodes didn't answer after multiple retries. Aborting startup
CRITICAL:root:Use the --no-voting option if you understand what effects it has on the cluster state
failed (exit code 1).
ganeti-rapi...done.
ganeti-confd...done.
(0) root@sever ~
# gnt-instance list
Cannot communicate with the master daemon.
Is it running and listening for connections?
(1) root@server ~

Checking the certificate, you see that it has expired. To solve this issue, on the current (but broken) master, create a new certificate:

# cp /var/lib/ganeti/server.pem ~/expired.server.pem
# openssl req -new -newkey rsa:1024 -days 1825 -nodes -x509 -keyout /var/lib/ganeti/server.pem -out /var/lib/ganeti/server.pem -batch &&
# chmod 0400 /var/lib/ganeti/server.pem
Then run this script to copy this new certificate to all nodes in the cluster:
#!/bin/sh
for i in X Y Z
do
       ssh $i "cp /var/lib/ganeti/server.pem ~/"
       scp newserver.pem $i:/var/lib/ganeti/server.pem
       ssh $i "chmod 0400 /var/lib/ganeti/server.pem"
       ssh $i "/etc/init.d/ganeti restart"
done
/etc/init.d/ganeti restart

Note it will restart the ganeti service on all the non-master nodes before restarting the service on the master node.