Monday, 15 December 2014

Uninstall a whole load of packages at once!

Using yum to uninstall all ruby packages that has the string rhscl in them:
$ yum list installed|grep ruby|grep rhscl|awk '{print $1}'| xargs yum -y remove 
Like-wise for ruby under foreman and scl:
$ yum list installed|grep ruby|grep foreman|awk '{print $1}'| xargs yum -y remove 
$ yum list installed|grep ruby|grep scl|awk '{print $1}'| xargs yum -y remove 

Friday, 12 December 2014

Install Repository Keys - Apt-get update - gpg --recv-keys

When running apt-get update, you get this error:
(0) root@debiansvr /var/tmp
# apt-get update
Get:1 http://packages.yourdomain wheezy Release.gpg [490 B]
Get:2 http://packages.yourdomain wheezy Release [3,489 B]
Get:3 http://ftp.uk.debian.org wheezy Release.gpg [1,655 B]
Ign http://packages.yourdomain wheezy Release
Get:4 http://ftp.uk.debian.org wheezy Release [168 kB]
Get:5 http://security.debian.org wheezy/updates Release.gpg [836 B]
Get:6 http://security.debian.org wheezy/updates Release [102 kB]
Get:7 http://ftp.uk.debian.org wheezy/main Sources [5,955 kB]
Get:8 http://security.debian.org wheezy/updates/main Sources [142 kB]
Get:9 http://packages.yourdomain wheezy/main Sources [6,892 B]
Get:10 http://packages.yourdomain wheezy/main i386 Packages [6,883 B]
Ign http://packages.yourdomain wheezy/main Translation-en_GB
Ign http://packages.yourdomain wheezy/main Translation-en
Get:11 http://security.debian.org wheezy/updates/contrib Sources [14 B]
Get:12 http://security.debian.org wheezy/updates/non-free Sources [14 B]
Get:13 http://security.debian.org wheezy/updates/main i386 Packages [228 kB]
Get:14 http://security.debian.org wheezy/updates/contrib i386 Packages [14 B]
Get:15 http://security.debian.org wheezy/updates/non-free i386 Packages [14 B]
Get:16 http://security.debian.org wheezy/updates/contrib Translation-en [14 B]
Get:17 http://security.debian.org wheezy/updates/main Translation-en [128 kB]
Get:18 http://security.debian.org wheezy/updates/non-free Translation-en [14 B]
Get:19 http://ftp.uk.debian.org wheezy/contrib Sources [47.8 kB]
Get:20 http://ftp.uk.debian.org wheezy/non-free Sources [93.4 kB]
Get:21 http://ftp.uk.debian.org wheezy/main i386 Packages [5,858 kB]
Get:22 http://ftp.uk.debian.org wheezy/contrib i386 Packages [42.3 kB]
Get:23 http://ftp.uk.debian.org wheezy/non-free i386 Packages [77.5 kB]
Get:24 http://ftp.uk.debian.org wheezy/contrib Translation-en [34.8 kB]
Get:25 http://ftp.uk.debian.org wheezy/main Translation-en [3,846 kB]
Get:26 http://ftp.uk.debian.org wheezy/non-free Translation-en [66.1 kB]
Fetched 16.8 MB in 15s (1,064 kB/s)
Reading package lists... Done
W: GPG error: http://packages.yourdomain wheezy Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 1639B68C40E3D96F
Check out which of your keys is missing:
(127) root@debiansvr /var/tmp
# apt-key list
/etc/apt/trusted.gpg
--------------------
pub   1024D/F42584E6 2008-04-06 [expired: 2012-05-15]
uid                  Lenny Stable Release Key 

pub   4096R/55BE302B 2009-01-27 [expired: 2012-12-31]
uid                  Debian Archive Automatic Signing Key (5.0/lenny) 

pub   2048R/6D849617 2009-01-24 [expired: 2013-01-23]
uid                  Debian-Volatile Archive Automatic Signing Key (5.0/lenny)

pub   1024D/DB782D34 2007-10-11
uid                  Matt Baker (Package Maintainer) 
sub   2048g/A5734914 2007-10-11

pub   1024D/7F1D2347 2006-08-23 [expired: 2011-10-20]
uid                  Index Data (Package Key) 

pub   1024D/773F7ECF 2009-03-10 [expires: 2015-03-27]
uid                  Nightly Build (ILRT autonomous builder) 
sub   2048g/DC9C4CFD 2009-03-10 [expires: 2015-03-27]

pub   1024D/DC2698A1 2006-02-23
uid                  steve.org.uk APT key (This key is only used to sign the APT repository at http://www.steve.org.uk/apt/) 
sub   2048g/0CB6CBA8 2006-02-23

pub   4096R/B98321F9 2010-08-07 [expires: 2017-08-05]
uid                  Squeeze Stable Release Key 

pub   4096R/473041FA 2010-08-27 [expires: 2018-03-05]
uid                  Debian Archive Automatic Signing Key (6.0/squeeze) 

pub   4096R/8D77295D 2009-07-12
uid                  Eric Evans 
uid                  Eric Evans 
uid                  Eric Evans 
uid                  Eric Evans 
sub   4096R/C47D63C0 2009-07-12

pub   2048R/2B5C1B00 2011-04-13
uid                  Sylvain Lebresne (pcmanus) 
sub   2048R/9CB2AA80 2011-04-13

pub   1024D/16BA136C 2005-08-21
uid                  Backports.org Archive Key 
sub   2048g/5B82CECE 2005-08-21

pub   4096R/46925553 2012-04-27 [expires: 2020-04-25]
uid                  Debian Archive Automatic Signing Key (7.0/wheezy) 

pub   4096R/65FFB764 2012-05-08 [expires: 2019-05-07]
uid                  Wheezy Stable Release Key 

pub   2048R/40E3D96F 2014-08-06
uid                  UOB package repo admin (Central software repo admin) 
sub   2048R/C92A16D8 2014-08-06

/etc/apt/trusted.gpg.d//debian-archive-jessie-stable.gpg
--------------------------------------------------------
pub   4096R/518E17E1 2013-08-17 [expires: 2021-08-15]
uid                  Jessie Stable Release Key 

/etc/apt/trusted.gpg.d//debian-archive-squeeze-automatic.gpg
------------------------------------------------------------
pub   4096R/473041FA 2010-08-27 [expires: 2018-03-05]
uid                  Debian Archive Automatic Signing Key (6.0/squeeze) 

/etc/apt/trusted.gpg.d//debian-archive-squeeze-stable.gpg
---------------------------------------------------------
pub   4096R/B98321F9 2010-08-07 [expires: 2017-08-05]
uid                  Squeeze Stable Release Key 

/etc/apt/trusted.gpg.d//debian-archive-wheezy-automatic.gpg
-----------------------------------------------------------
pub   4096R/46925553 2012-04-27 [expires: 2020-04-25]
uid                  Debian Archive Automatic Signing Key (7.0/wheezy) 

/etc/apt/trusted.gpg.d//debian-archive-wheezy-stable.gpg
--------------------------------------------------------
pub   4096R/65FFB764 2012-05-08 [expires: 2019-05-07]
uid                  Wheezy Stable Release Key 

(0) root@debiansvr /var/tmp
Download the Release and Release.pgp files of the repos. Get the keys and install them:
$ wget http://packages.yourdomain/debian/dists/wheezy/Release.gpg
$ wget http://packages.yourdomain/debian/dists/wheezy/Release
$ gpg Release.gpg
$ gpg --recv-keys 65FFB764
$ gpg --recv-keys 46925553
or just copy the /etc/apt/trusted.gpg file from a system that works over and run apt-get update ;)

Thursday, 11 December 2014

Making an LDAP client for user authentication - CentOS 6 and 5

Install the software:
yum install nss-pam-ldapd nscd pam_krb5 

-bash-3.2# authconfig --enablecache --enableshadow --passalgo=sha512 --disablenis --enableldap --ldapserver="ldap://YOUR_LDAP_SERVER" --ldapbasedn="dc=YOUR_DC1,dc=YOUR_DC2,dc=YOUR_DC3" --enablekrb5 --krb5kdc=YOUR_KERBEROS_DOMAIN --krb5adminserver=YOUR_KERBEROS_SERVER --krb5realm=YOUR_KERBEROS_REALM --enablelocauthorize --disablekrb5kdcdns --disablekrb5realmdns --disablewinbind --disablewins --disablesmartcard --enablemkhomedir --update

Then edit/create the nslcd.conf file:
# This is the configuration file for the LDAP nameservice
# switch library's nslcd daemon. It configures the mapping
# between NSS names (see /etc/nsswitch.conf) and LDAP
# information in the directory.
# See the manual page nslcd.conf(5) for more information.

uid nslcd
gid ldap

# The location at which the LDAP server(s) should be reachable.
uri ldap://YOUR_LDAP_SERVER/

# The search base that will be used for all queries.
base dc=YOUR_DC1,dc=YOUR_DC2,dc=YOUR_DC3

# The DN to bind with for normal lookups.
binddn uid=YOUR_ADMIN,ou=YOUR_OU,dc=YOUR_DC1,dc=YOUR_DC2,dc=YOUR_DC3

# The password
bindpw SECRETPASSWORDHERE

ssl no
tls_cacertdir /etc/openldap/cacerts

map passwd homeDirectory "/home/$uid"
Then check /etc/nsswitch.conf file, making sure the following entries are present:
passwd:     files ldap
shadow:     files ldap
group:      files ldap
Restart nslcd:
/etc/init.d/nslcd restart
nslcd is not available on CentOS 5, so we use sssd instead.
yum install sssd
yum remove nscd
Edit the sssd.conf file to:
[sssd]
domains = ADS.BRIS.AC.UK
services = nss,pam
config_file_version = 2
 
[nss]
filter_groups = root
filter_users = root
override_homedir = /home/%u
 
# A domain with identities provided by LDAP and authentication by Kerberos
[domain/YOUR_KERBEROS_DOMAIN]
enumerate = false
 
id_provider = ldap
auth_provider = krb5
chpass_provider = krb5
 
ldap_uri = ldap://YOUR_LDAP_SERVER/
ldap_search_base = dc=YOUR_DC1,dc=YOUR_DC2,dc=YOUR_DC3
#tls_reqcert = demand
ldap_tls_cacert = /etc/openldap/cacerts
ldap_default_bind_dn = uid=YOUR_ADMIN,ou=YOUR_OU,dc=YOUR_DC1,dc=YOUR_DC2,dc=YOUR_DC3
ldap_default_authtok_type = password
ldap_default_authtok = SECRETPASSWORDHERE
 
krb5_server = YOUR_KERBEROS_SERVER
krb5_realm = YOUR_KERBEROS_REALM
krb5_ccachedir = /tmp
krb5_ccname_template = FILE:%d/krb5cc_%U_XXXXXX
krb5_auth_timeout = 15
Run this authconfig command:
-bash-3.2$ authconfig --enablesssdauth --enablesssd --disableldap --update
Then check /etc/nsswitch.conf file, making sure the following entries are present:
passwd:     files sss
shadow:     files sss
group:      files sss
Restart sssd: Restart nslcd:
/etc/init.d/sssd restart
Make sure it starts up at boot time:
-bash-3.2# chkconfig --list
iscsi           0:off   1:off   2:off   3:on    4:on    5:on    6:off
iscsid          0:off   1:off   2:off   3:on    4:on    5:on    6:off
lvm2-monitor    0:off   1:on    2:on    3:on    4:on    5:on    6:off
mcstrans        0:off   1:off   2:on    3:on    4:on    5:on    6:off
messagebus      0:off   1:off   2:off   3:on    4:on    5:on    6:off
multipathd      0:off   1:off   2:off   3:off   4:off   5:off   6:off
netconsole      0:off   1:off   2:off   3:off   4:off   5:off   6:off
netfs           0:off   1:off   2:off   3:on    4:on    5:on    6:off
netplugd        0:off   1:off   2:off   3:off   4:off   5:off   6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off
puppet          0:off   1:off   2:off   3:off   4:off   5:off   6:off
rawdevices      0:off   1:off   2:off   3:on    4:on    5:on    6:off
rdisc           0:off   1:off   2:off   3:off   4:off   5:off   6:off
rsyslog         0:off   1:off   2:off   3:off   4:off   5:off   6:off
sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
sssd            0:off   1:off   2:off   3:on    4:on    5:on    6:off
-bash-3.2# chkconfig sssd on
-bash-3.2# chkconfig --list
iscsi           0:off   1:off   2:off   3:on    4:on    5:on    6:off
iscsid          0:off   1:off   2:off   3:on    4:on    5:on    6:off
lvm2-monitor    0:off   1:on    2:on    3:on    4:on    5:on    6:off
mcstrans        0:off   1:off   2:on    3:on    4:on    5:on    6:off
messagebus      0:off   1:off   2:off   3:on    4:on    5:on    6:off
multipathd      0:off   1:off   2:off   3:off   4:off   5:off   6:off
netconsole      0:off   1:off   2:off   3:off   4:off   5:off   6:off
netfs           0:off   1:off   2:off   3:on    4:on    5:on    6:off
netplugd        0:off   1:off   2:off   3:off   4:off   5:off   6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off
puppet          0:off   1:off   2:off   3:off   4:off   5:off   6:off
rawdevices      0:off   1:off   2:off   3:on    4:on    5:on    6:off
rdisc           0:off   1:off   2:off   3:off   4:off   5:off   6:off
rsyslog         0:off   1:off   2:off   3:off   4:off   5:off   6:off
sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
sssd            0:off   1:off   2:on    3:on    4:on    5:on    6:off

Wednesday, 10 December 2014

Foreman broken! Passenger - Error writing to temporary file

Nobody has touched it but overnight it broke! Now we just get this error from passenger:
Web application could not be started
Error writing to temporary file
Application root
/usr/share/foreman
Environment (value of RAILS_ENV, RACK_ENV, WSGI_ENV and PASSENGER_ENV)
production
Ruby interpreter command
/usr/bin/ruby193-ruby
User and groups
uid=497(foreman) gid=498(foreman) groups=498(foreman),52(puppet)
Environment variables
[cut]

Ulimits

    Unknown

System memory usage

                 total       used       free     shared    buffers     cached
    Mem:         32057      31710        347          3        715      29020
    -/+ buffers/cache:       1973      30084
    Swap:         4047          0       4047

Powered by Phusion Passenger, mod_rails / mod_rack for Apache and Nginx.

I spent nearly all day looking at this but with no avail. Does anyone have any ideas?

Tuesday, 9 December 2014

Ganeti Xen VM - Creating and Growing Disk

Installing a Ganeti VM

(1) root@node03 /srv/ganeti/os/centos
#  /usr/local/projects/deploy/ganeti/bin/mk-gnt-instance-hail.sh  -h fsxz.isys -m 4096 -c 2 -d 66 -o centos+5
Tue Dec  9 15:54:28 2014  - INFO: Selected nodes for instance fsxz.isys via iallocator hail: node05.gnt6.fog.priv, node04.gnt6.fog.priv
Tue Dec  9 15:54:30 2014 * creating instance disks...
Tue Dec  9 15:54:40 2014 adding instance fsxz.isys to cluster config
Tue Dec  9 15:54:42 2014 * checking mirrors status
Tue Dec  9 15:54:43 2014  - INFO: - device disk/0:  0.70% done, 18m 48s remaining (estimated)
Tue Dec  9 15:54:43 2014  - INFO: - device disk/1:  1.90% done, 1m 53s remaining (estimated)
Tue Dec  9 15:54:43 2014 * pausing disk sync to install instance OS
Tue Dec  9 15:54:44 2014 * running the instance OS create scripts...

Growing Disks

Growing a VM disk involves 2 stages:
Grow the block device
Resize the file system

Ganeti can grow the block device for you using the gnt-instance grow-disk command:

$ gnt-instance grow-disk test.mydomain.com 0 100G
The instance will not see the new disk size until it's rebooted.

# Reboot the instance $ sudo gnt-instance reboot test.mydomain.com Resize the file system After the VM is rebooted, log in to it (ssh) and resize the partion

sudo resize2fs /dev/xvdX
A Script to generate a MAC address from the IP address:
#!/bin/bash

echo -n 'Virtual MAC 02:00'
printf ':%02X' ${1//./ }
echo

Monday, 8 December 2014

Install Grub Boot option - after installing new distro

Install new distro on spare partition or disk. Do not install grub! Boot up original distro (Debian 7 here).
root@sonsmon:/home/sont# update-grub
Generating grub.cfg ...
Found background image: /usr/share/images/desktop-base/desktop-grub.png
Found linux image: /boot/vmlinuz-3.2.0-4-amd64
Found initrd image: /boot/initrd.img-3.2.0-4-amd64
Found CentOS release 5.11 (Final) on /dev/sda2
done

root@sonsmon:/home/sont# cd /boot/grub
root@sonsmon:/boot/grub# vi grub.cfg
### BEGIN /etc/grub.d/30_os-prober ###
menuentry "CentOS release 5.11 (Final) (on /dev/sda2)" --class gnu-linux --class gnu --class os {
        insmod part_msdos
        insmod ext2
        set root='(hd0,msdos2)'
        search --no-floppy --fs-uuid --set=root d87e04c9-0742-46d7-8ad7-39835b144cc8
        multiboot /boot/xen.gz-2.6.16.400.el5
        linux /boot/vmlinuz-2.6.18-400.el5xen root=/dev/sda2
        initrd /boot/initrd-2.6.18-400.el5xen.img
}
### END /etc/grub.d/30_os-prober ###
Create a new boot initrd image loaded with drivers:
[root@localhost ~]# cd /boot
[root@localhost boot]# mkinitrd -v --omit-scsi-modules --preload=xenblk --with=xennet --builtin=virtio_pci --builtin=virtio_blk --builtin=xen_vbd initrd-xen-para-son.img 2.6.18-400.el5xen