Set Timezone - Solaris 11

To get the current set timezone:

svccfg -s timezone:default listprop timezone/localtime

To set the timezone:

# svccfg -s timezone:default setprop timezone/localtime = Europe/London
# svcadm refresh timezone

Breaking a ZFS mirror - Solaris 11

So you've mirrored your system disk wrongly! This is how you would break it and fix the problem before re-mirroring:

root@solaris11server:~$ zpool status
  pool: rpool
 state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
        pool will no longer be accessible on older software versions.
  scan: resilvered 108G in 20m20s with 0 errors on Fri Dec  2 10:16:44 2016

config:

        NAME                         STATE     READ WRITE CKSUM
        rpool                        ONLINE       0     0     0
          mirror-0                   ONLINE       0     0     0
            c0t5000CCA0166ED0ECd0s0  ONLINE       0     0     0
            c0t5000CCA0166F8C50d0    ONLINE       0     0     0

errors: No known data errors

Added mirror as disk not slice. Turned disk into an EFI/GPT tabled disk. Using format and selecting the disk gives this messsage:

selecting c0t5000CCA0166F8C50d0
[disk formatted]
/dev/dsk/c0t5000CCA0166F8C50d0s0 is part of active ZFS pool rpool. Please see zpool(1M).
Reading the primary EFI GPT label failed.  Using backup label.
Use the 'backup' command to restore the primary label.

Do we use the back command? Don't know. We must break the mirror first though - using zpool detach:

root@solaris11server:~# zpool detach rpool c0t5000CCA0166F8C50d0
root@solaris11server:~# zpool status
  pool: rpool
 state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
        pool will no longer be accessible on older software versions.
  scan: resilvered 108G in 20m20s with 0 errors on Fri Dec  2 10:16:44 2016

config:

        NAME                       STATE     READ WRITE CKSUM
        rpool                      ONLINE       0     0     0
          c0t5000CCA0166ED0ECd0s0  ONLINE       0     0     0

errors: No known data errors

Now look at the partition map on our good disk:

root@solaris11server:~# prtvtoc /dev/rdsk/c0t5000CCA0166ED0ECd0s0
* /dev/rdsk/c0t5000CCA0166ED0ECd0s0 (volume "solaris") partition map
*
* Dimensions:
*     512 bytes/sector
*     625 sectors/track
*      20 tracks/cylinder
*   12500 sectors/cylinder
*   46875 cylinders
*   46873 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
* Unallocated space:
*       First     Sector    Last
*       Sector     Count    Sector
*           0     12500     12499
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      2    00      12500 585900000 585912499
       2      5    01          0 585912500 585912499

And compare it with our bad disk:

root@solaris11server:~# prtvtoc /dev/rdsk/c0t5000CCA0166F8C50d0s0
* /dev/rdsk/c0t5000CCA0166F8C50d0s0 partition map
*
* Dimensions:
*     512 bytes/sector
* 585937500 sectors
* 585937433 accessible sectors
*
* Flags:
*   1: unmountable
*  10: read-only
*
* Unallocated space:
*       First     Sector    Last
*       Sector     Count    Sector
*          34       222       255
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      4    00        256 585920827 585921082
       8     11    00  585921083     16384 585937466

So let us label our bad disk with the proper SMI label - not the EFI one:

root@solaris11server:~# format -e /dev/rdsk/c0t5000CCA0166F8C50d0
selecting /dev/rdsk/c0t5000CCA0166F8C50d0
[disk formatted]


FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        inquiry    - show disk ID
        scsi       - independent SCSI mode selects
        cache      - enable, disable or query SCSI disk cache
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> label
[0] SMI Label
[1] EFI Label
Specify Label type[1]: 0
Auto configuration via format.dat[no]?
Auto configuration via generic SCSI-2[no]?
format> p


PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        ! - execute , then return
        quit
partition> p
Current partition table (default):
Total disk cylinders available: 46873 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0       root    wm       0 -    20      128.17MB    (21/0/0)       262500
  1       swap    wu      21 -    41      128.17MB    (21/0/0)       262500
  2     backup    wu       0 - 46872      279.38GB    (46873/0/0) 585912500
  3 unassigned    wm       0                0         (0/0/0)             0
  4 unassigned    wm       0                0         (0/0/0)             0
  5 unassigned    wm       0                0         (0/0/0)             0
  6        usr    wm      42 - 46872      279.13GB    (46831/0/0) 585387500
  7 unassigned    wm       0                0         (0/0/0)             0

partition> q

Now, let's copy the partition table of our good disk over to our bad one:

root@solaris11server:~# prtvtoc /dev/rdsk/c0t5000CCA0166ED0ECd0s0 | fmthard -s - /dev/rdsk/c0t5000CCA0166F8C50d0s2
fmthard:  New volume table of contents now in place.
root@solaris11server:~# prtvtoc /dev/rdsk/c0t5000CCA0166F8C50d0s0
* /dev/rdsk/c0t5000CCA0166F8C50d0s0 partition map
*
* Dimensions:
*     512 bytes/sector
*     625 sectors/track
*      20 tracks/cylinder
*   12500 sectors/cylinder
*   46875 cylinders
*   46873 accessible cylinders
*
* Flags:
*   1: unmountable
*  10: read-only
*
* Unallocated space:
*       First     Sector    Last
*       Sector     Count    Sector
*           0     12500     12499
*
*                          First     Sector    Last
* Partition  Tag  Flags    Sector     Count    Sector  Mount Directory
       0      2    00      12500 585900000 585912499
       2      5    01          0 585912500 585912499

All is now good, so let's attach (mirror) our system disks:

root@solaris11server:~# zpool attach rpool c0t5000CCA0166ED0ECd0s0 c0t5000CCA0166F8C50d0s0
Make sure to wait until resilver is done before rebooting.
root@solaris11server:~# zpool status
  pool: rpool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function in a degraded state.
action: Wait for the resilver to complete.
        Run 'zpool status -v' to see device specific details.
  scan: resilver in progress since Wed Dec 14 10:13:36 2016
    1.52G scanned out of 108G at 58.9M/s, 30m52s to go
    1.52G resilvered, 1.41% done
config:

        NAME                         STATE     READ WRITE CKSUM
        rpool                        DEGRADED     0     0     0
          mirror-0                   DEGRADED     0     0     0
            c0t5000CCA0166ED0ECd0s0  ONLINE       0     0     0
            c0t5000CCA0166F8C50d0s0  DEGRADED     0     0     0  (resilvering)

errors: No known data errors

After the resilvering has finished, our system is mirrored correctly! :)

GitLab CE - repo web page not updating

So you've just created a GitLab repo and push the git working directory contents to it but the page does not change from the default instructions. To fix this log on to the gitlab server and perform the following command as root:

# gitlab-rake cache:clear

This is on the following GitLab CE:

GitLab 8.13.3

GitLab Shell 3.6.6

GitLab Workhorse 0.8.5

GitLab API v3

Git 2.7.4

Ruby 2.3.1p112

Rails 4.2.7.1

PostgreSQL 9.2.18

Dirty CoW kernel check - CentOS

So there's been a load of work due to the Dirty CoW vulnerability... you need to find out if there's been a kernel update so that you can reboot your CentOS P/VM... Here's a oneliner:

if [ "`rpm -q kernel --queryformat '%{installtime} %{version}-%{release}.%{arch}\n' | \
sort -n -k1 | tail -1 | cut -d ' ' -f 2`" = "`uname -r`" ]; \
then echo "You are running the latest kernel" && uname -r; \
else echo "There is a new kernel. You need a reboot" && echo "Current kernel: " && uname -r &&  \
echo "The latest kernel: " && rpm -q kernel --queryformat '%{version}-%{release}.%{arch}\n' | sort -n -k1 | tail -1;  fi

This is what it looks like on CentOS 6:

# if [ "`rpm -q kernel --queryformat '%{installtime} %{version}-%{release}.%{arch}\n' |sort -n -k1 | tail -1 | cut -d ' ' -f 2`" = "`uname -r`" ]; then echo "You are running the latest kernel" && uname -r; else echo "There is a new kernel. You need a reboot" && echo "Current kernel: " && uname -r && echo "The latest kernel: " && rpm -q kernel --queryformat '%{version}-%{release}.%{arch}\n' | sort -n -k1 | tail -1;  fi
There is a new kernel. You need a reboot
Current kernel:
2.6.32-642.4.2.el6.x86_64
The latest kernel:
2.6.32-642.6.2.el6.x86_64

The table below shows which kernel you should be running to fix the Dirty CoW vulnerability:

Distro	Kernel version
CentOS 5	2.6.32-642.3.1.el6.x86_64
CentOS 6	2.6.32-642.6.2.el6.x86_64
CentOS 7	3.10.0-327.36.3.el7.x86_64
Debian 7	3.2.82-1
Debian 8	3.16.36-1+deb8u2

Postfix MTA service not working! CentOS 6

I was having problems with keeping the Postfix MTA configured and running with Puppet. Each time Puppet ran it detected that it wasn't running and attempted to start it with no avail. The error when looking at the service was this:

# service postfix status
master dead but pid file exists

But removing the pid file didn't not help:

# locate postfix|grep pid
/var/spool/postfix/pid
/var/spool/postfix/pid/master.pid
[root@webtest ~]# rm /var/spool/postfix/pid/master.pid
rm: remove regular file `/var/spool/postfix/pid/master.pid'? y
[root@webtest ~]# service postfix status
master dead but subsys locked

So looking at the logs this was seen:

# tail  /var/log/maillog
Oct 30 19:44:06 webtest postfix/master[8005]: fatal: bind 127.0.0.1 port 25: Address already in use
Oct 30 20:09:49 webtest postfix/postfix-script[10053]: starting the Postfix mail system
Oct 30 20:09:49 webtest postfix/master[10054]: fatal: bind 127.0.0.1 port 25: Address already in use
Oct 30 20:10:04 webtest postfix/postfix-script[10602]: starting the Postfix mail system
Oct 30 20:10:04 webtest postfix/master[10603]: fatal: bind 127.0.0.1 port 25: Address already in use
Oct 30 20:10:53 webtest postfix/postfix-script[11037]: starting the Postfix mail system

The problem looks like another MTA was running hogging port 25. A quick ps for sendmail revealed nothing, but there's another agent that comes with CentOS 6:

[root@webtest ~]# ps -ef|grep send
root     12448  9780  0 20:16 pts/0    00:00:00 grep send
[root@webtest ~]# ps -ef|grep exim
root     12109  9780  0 20:22 pts/0    00:00:00 grep exim
exim     57456     1  0 Jul07 ?        00:00:00 /usr/sbin/exim -bd -q1h

[root@webtest ~]# service exim stop
Shutting down exim:                                        [  OK  ]
[root@webtest ~]# chkconfig exim off

Now a Puppet run should install and run Postfix without a problem:

# puppet agent -t
Notice: Local environment: 'production' doesn't match server specified node environment 'websites', switching agent to 'websites'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for website.domain.com
Info: Applying configuration version '1477858442'
Notice: /Stage[main]/postfixmta/Service[postfix]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/postfixmta/Service[postfix]: Unscheduling refresh on Service[postfix]
Notice: Applied catalog in 1.67 seconds
# puppet agent -t
Notice: Local environment: 'production' doesn't match server specified node environment 'websites', switching agent to 'websites'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for website.domain.com
Info: Applying configuration version '1477858442'
Notice: Applied catalog in 1.46 seconds

Fix Analytics not displaying in OpsCenter for LDOM - Solaris 11

root@solaris-ldom:~# svcs scn-agent
STATE          STIME    FMRI
maintenance    Sep_23   svc:/application/management/common-agent-container-1:scn-agent
root@solaris-ldom:~# svcs -xv
svc:/application/management/common-agent-container-1:scn-agent (Cacao, a common Java container for JDMK/JMX based management solution)
State: maintenance since Fri Sep 23 22:36:14 2016
Reason: Restarting too quickly.
  See: http://support.oracle.com/msg/SMF-8000-L5
  See: man -M /usr/share/man -s 1M cacaoadm
  See: man -M /usr/share/man -s 5 cacao
  See: /var/svc/log/application-management-common-agent-container-1:scn-agent.log
Impact: This service is not running.
root@solaris-ldom:~# cat /var/svc/log/application-management-common-agent-container-1:scn-agent.log
[ Mar 24 09:57:57 Disabled. ]
[ Mar 24 09:57:57 Rereading configuration. ]
[ Mar 24 09:58:01 Enabled. ]

-cut-

[ Sep 23 22:36:12 Stopping because all processes in service exited. ]
[ Sep 23 22:36:13 Executing stop method ("/usr/lib/cacao/lib/tools/scripts/cacao_smf stop scn-agent"). ]
[ Sep 23 22:36:14 Method "stop" exited with status 0. ]
[ Sep 23 22:36:14 Restarting too quickly, changing state to maintenance. ]
root@solaris-ldom:~# svcadm disable svc:/application/management/common-agent-container-1:scn-agent
root@solaris-ldom:~# svcs scn-agent
STATE          STIME    FMRI
disabled       11:55:48 svc:/application/management/common-agent-container-1:scn-agent
root@solaris-ldom:~# svcs -xv
root@solaris-ldom:~# svcadm enable svc:/application/management/common-agent-container-1:scn-agent
root@solaris-ldom:~# svcs -xv
svc:/application/management/common-agent-container-1:scn-agent (Cacao, a common Java container for JDMK/JMX based management solution)
State: offline* transitioning to online since Mon Sep 26 11:56:09 2016
Reason: Start method is running.
  See: http://support.oracle.com/msg/SMF-8000-C4
  See: man -M /usr/share/man -s 1M cacaoadm
  See: man -M /usr/share/man -s 5 cacao
  See: /var/svc/log/application-management-common-agent-container-1:scn-agent.log
Impact: This service is not running.
root@solaris-ldom:~# tail /var/svc/log/application-management-common-agent-container-1:scn-agent.log
[ Sep 23 22:31:50 Executing start method ("/usr/lib/cacao/lib/tools/scripts/cacao_smf start scn-agent"). ]
[ Sep 23 22:33:13 Method "start" exited with status 0. ]
[ Sep 23 22:36:12 Stopping because all processes in service exited. ]
[ Sep 23 22:36:13 Executing stop method ("/usr/lib/cacao/lib/tools/scripts/cacao_smf stop scn-agent"). ]
[ Sep 23 22:36:14 Method "stop" exited with status 0. ]
[ Sep 23 22:36:14 Restarting too quickly, changing state to maintenance. ]
[ Sep 26 11:55:48 Leaving maintenance because disable requested. ]
[ Sep 26 11:55:48 Disabled. ]
[ Sep 26 11:56:09 Enabled. ]
[ Sep 26 11:56:09 Executing start method ("/usr/lib/cacao/lib/tools/scripts/cacao_smf start scn-agent"). ]
root@solaris-ldom:/var/adm# svcs scn-agent
STATE          STIME    FMRI
online         11:57:12 svc:/application/management/common-agent-container-1:scn-agent
root@solaris-ldom:/var/adm#

SELinux and sending mail via HTTPD - CentOS 7

Use this SELinux command to give the Apache process to use sendmail:

#sudo setsebool -p httpd_can_sendmail 1

Monitor DNS lookup

# tcpdump -i eth0 port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:48:15.324300 IP client.63533 > dns-server.domain: 17904+ PTR? 1.0.0.127.in-addr.arpa. (40)
10:48:15.324848 IP dns-server.domain > client.63533: 17904* 1/1/2 PTR localhost. (121)
10:48:15.325137 IP client.50547 > dns-server.domain: 49520+ AAAA? localhost. (27)
10:48:15.325293 IP client.43181 > dns-server.domain: 25134+ PTR? xx.x.xxx.xxx.in-addr.arpa. (43)
10:48:15.325643 IP dns-server.domain > client.50547: 49520* 1/1/1 AAAA ::1 (85)
10:48:15.325903 IP dns-server.domain > client.43181: 25134* 1/3/6 PTR dns-server. (268)
10:48:19.565837 IP client.25663 > dns-server.domain: 43756+ AAAA? client. (42)
10:48:19.566389 IP dns-server.domain > client.25663: 43756 NXDomain* 0/1/0 (93)
10:48:19.566497 IP client.64053 > dns-server.domain: 55198+ AAAA? client. (40)
10:48:19.567026 IP dns-server.domain > client.64053: 55198* 0/1/0 (91)
10:48:19.567086 IP client.49399 > dns-server.domain: 1076+ AAAA? client. (37)
10:48:19.567600 IP dns-server.domain > client.49399: 1076 NXDomain* 0/1/0 (88)
10:48:19.567656 IP client.24922 > dns-server.domain: 50409+ AAAA? client. (26)
10:48:19.568080 IP dns-server.domain > client.24922: 50409 NXDomain 0/1/0 (101)
10:53:15.248429 IP client.17122 > dns-server.domain: 45962+ PTR? 1.0.0.127.in-addr.arpa. (40)
10:53:15.248968 IP dns-server.domain > client.17122: 45962* 1/1/2 PTR localhost. (121)
10:53:15.249332 IP client.17597 > dns-server.domain: 59594+ AAAA? localhost. (27)
10:53:15.249819 IP dns-server.domain > client.17597: 59594* 1/1/1 AAAA ::1 (85)
10:53:19.590980 IP client.52610 > dns-server.domain: 39707+ AAAA? client. (42)
10:53:19.591525 IP dns-server.domain > client.52610: 39707 NXDomain* 0/1/0 (93)
10:53:19.591683 IP client.32529 > dns-server.domain: 23733+ AAAA? client. (40)
10:53:19.592180 IP dns-server.domain > client.32529: 23733* 0/1/0 (91)

Perl code for sending messages via Gmail - authentication error

Here's a comprehensive blog post to do this! http://robertmaldon.blogspot.co.uk/2006/10/sending-email-through-google-smtp-from.html Here's the chunk I used (after using CPAN to install Net::SMTP::SSL*) - gmail3.pl:

#!/usr/bin/perl -w

use Net::SMTP::SSL;

sub send_mail {
my $to = $_[0];
my $subject = $_[1];
my $body = $_[2];

my $from = 'johnny@gmail.com';
my $password = 'MySecretGmailPassword';

my $smtp;

if (not $smtp = Net::SMTP::SSL->new('smtp.gmail.com',
                            Port => 465,
                            Debug => 1)) {
   die "Could not connect to server\n";
}

$smtp->auth($from, $password)
   || die "Authentication failed!\n";

$smtp->mail($from . "\n");
my @recepients = split(/,/, $to);
foreach my $recp (@recepients) {
    $smtp->to($recp . "\n");
}
$smtp->data();
$smtp->datasend("From: " . $from . "\n");
$smtp->datasend("To: " . $to . "\n");
$smtp->datasend("Subject: " . $subject . "\n");
$smtp->datasend("\n");
$smtp->datasend($body . "\n");
$smtp->dataend();
$smtp->quit;
}

# Send away!
&send_mail('johnny@mywork.com', 'Server just blew up', 'Some more detail');

On execution, you might get this authentication error:

*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 together with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at ./gmail3.pl line 15.
Net::SMTP::SSL>>> Net::SMTP::SSL(1.03)
Net::SMTP::SSL>>>   IO::Socket::SSL(1.94)
Net::SMTP::SSL>>>     IO::Socket::IP(0.21)
Net::SMTP::SSL>>>       IO::Socket(1.34)
Net::SMTP::SSL>>>         IO::Handle(1.33)
Net::SMTP::SSL>>>           Exporter(5.68)
Net::SMTP::SSL>>>   Net::Cmd(3.10)
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 220 smtp.gmail.com ESMTP gg10sm6569316wjd.4 - gsmtp
Net::SMTP::SSL=GLOB(0x17f7a78)>>> EHLO localhost.localdomain
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-smtp.gmail.com at your service, [2001:630:e4:8102:21d:d8ff:feb7:1dee]
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-SIZE 35882577
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-8BITMIME
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-ENHANCEDSTATUSCODES
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-PIPELINING
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-CHUNKING
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250 SMTPUTF8
Net::SMTP::SSL=GLOB(0x17f7a78)>>> AUTH LOGIN
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 334 VXNlcm5hbWU6
Net::SMTP::SSL=GLOB(0x17f7a78)<<< (decoded) Username:
Net::SMTP::SSL=GLOB(0x17f7a78)>>> (decoded) johnny@gmail.com
Net::SMTP::SSL=GLOB(0x17f7a78)>>> Y2xlYXJkZWZAZ21haWwuY29t
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 334 UGFzc3dvcmQ6
Net::SMTP::SSL=GLOB(0x17f7a78)<<< (decoded) Password:
Net::SMTP::SSL=GLOB(0x17f7a78)>>> (decoded) MySecretGmailPassword
Net::SMTP::SSL=GLOB(0x17f7a78)>>> YXBwbGVncmVlbg==
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 534-5.7.14  Please log in via your web browser and
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 534-5.7.14 then try again.
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 534-5.7.14  Learn more at
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 534 5.7.14  https://support.google.com/mail/answer/78754 gg10sm6569316wjd.4 - gsmtp
Authentication failed!

This is simply solved by login on to the Gmail account (via a browser) and changing the password. Remember to insert the new password into the script and away we go:

*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 together with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at ./gmail3.pl line 15.
Net::SMTP::SSL>>> Net::SMTP::SSL(1.03)
Net::SMTP::SSL>>>   IO::Socket::SSL(1.94)
Net::SMTP::SSL>>>     IO::Socket::IP(0.21)
Net::SMTP::SSL>>>       IO::Socket(1.34)
Net::SMTP::SSL>>>         IO::Handle(1.33)
Net::SMTP::SSL>>>           Exporter(5.68)
Net::SMTP::SSL>>>   Net::Cmd(3.10)
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 220 smtp.gmail.com ESMTP uo4sm6542414wjc.36 - gsmtp
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> EHLO localhost.localdomain
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-smtp.gmail.com at your service, [2001:630:e4:8102:21d:d8ff:feb7:1dee]
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-SIZE 35882577
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-8BITMIME
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-ENHANCEDSTATUSCODES
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-PIPELINING
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-CHUNKING
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250 SMTPUTF8
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> AUTH LOGIN
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 334 VXNlcm5hbWU6
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< (decoded) Username:
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> (decoded) johnny@gmail.com
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> Y2xlYXJkZWZAZ21haWwuY29t
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 334 UGFzc3dvcmQ6
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< (decoded) Password:
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> (decoded) myNewSecretPassword
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> QXBwbGVHcjMzbg==
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 235 2.7.0 Accepted
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> MAIL FROM:
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250 2.1.0 OK uo4sm6542414wjc.36 - gsmtp
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> RCPT TO:
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250 2.1.5 OK uo4sm6542414wjc.36 - gsmtp
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> DATA
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 354  Go ahead uo4sm6542414wjc.36 - gsmtp
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> From: johnny@gmail.com
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> To: johnny@mywork.com
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> Subject: Server just blew up
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> Some more detail
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> .
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250 2.0.0 OK 1470992894 uo4sm6542414wjc.36 - gsmtp
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> QUIT
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 221 2.0.0 closing connection uo4sm6542414wjc.36 - gsmtp

Installing Perl Modules with CPAN:

# perl -MCPAN -e 'shell'
Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.9800)
Enter 'h' for help.

cpan[1]> install Net::SMTP::SSL
Reading '/root/.cpan/Metadata'
  Database was generated on Thu, 11 Aug 2016 00:53:43 GMT
Running install for module 'Net::SMTP::SSL'
Running make for R/RJ/RJBS/Net-SMTP-SSL-1.03.tar.gz
Fetching with HTTP::Tiny:
http://cpan.netbet.org/authors/id/R/RJ/RJBS/Net-SMTP-SSL-1.03.tar.gz
Checksum for /root/.cpan/sources/authors/id/R/RJ/RJBS/Net-SMTP-SSL-1.03.tar.gz ok
Scanning cache /root/.cpan/build for sizes
............................................................................DONE

  CPAN.pm: Building R/RJ/RJBS/Net-SMTP-SSL-1.03.tar.gz

Checking if your kit is complete...
Looks good
Writing Makefile for Net::SMTP::SSL
Writing MYMETA.yml and MYMETA.json
cp lib/Net/SMTP/SSL.pm blib/lib/Net/SMTP/SSL.pm
Manifying blib/man3/Net::SMTP::SSL.3pm
  RJBS/Net-SMTP-SSL-1.03.tar.gz
  /bin/make -- OK
'YAML' not installed, will not store persistent state
Running make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/test.t .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.07 cusr  0.00 csys =  0.08 CPU)
Result: PASS
  RJBS/Net-SMTP-SSL-1.03.tar.gz
  /bin/make test -- OK
Running make install
Manifying blib/man3/Net::SMTP::SSL.3pm
Installing /usr/local/share/perl5/Net/SMTP/SSL.pm
Installing /usr/local/share/man/man3/Net::SMTP::SSL.3pm
Appending installation info to /usr/lib64/perl5/perllocal.pod
  RJBS/Net-SMTP-SSL-1.03.tar.gz
  /bin/make install  -- OK

cpan[2]> Terminal does not support GetHistory.
Lockfile removed.

Move files within all subdirectories - Windows CMD

So you are in a directory which have subdirectories contain files you want to move into the parent. Problem with the CMD on Windows (DOS) is that it can not do * globs for directories. This command will do:

F:\RICKRI~1>forfiles /s /m *.epub /c "cmd /c move @path %CD%"

Rsyslog and SELinux issue - CentOS 7

So you've configured rsyslog (server) to put logs in a certain directory but nothing is getting logged. Well looking at the logs there's permissions issues with error messages such as:

Jul 18 14:21:31 pmoney dbus-daemon: dbus[1149]: avc:  received policyload notice (seqno=4)
Jul 18 14:21:31 pmoney dbus[1149]: avc:  received policyload notice (seqno=4)
Jul 18 14:21:31 pmoney dbus[1149]: [system] Reloaded configuration
Jul 18 14:21:31 pmoney dbus-daemon: dbus[1149]: [system] Reloaded configuration
Jul 18 14:21:56 pmoney rsyslogd-3000: Could not open dynamic file '/var/log/remote/idrac155/stockwood.pink.priv/common.log' [state -3000] - discarding message
Jul 18 14:22:00 pmoney rsyslogd-3000: Could not open dynamic file '/var/log/remote/idrac155/woodstock.pink.priv/common.log' [state -3000] - discarding message

And even with the SELinux commands, there are still errors:

# semanage fcontext -a -t syslogd_var_lib_t "/var/log/remote(/.*)?"
# restorecon -R -v /var/log/remote

[root@pmoney remote]# tail /var/log/messages
Jul 18 14:21:31 pmoney dbus-daemon: dbus[1149]: avc:  received policyload notice (seqno=4)
Jul 18 14:21:31 pmoney dbus[1149]: avc:  received policyload notice (seqno=4)
Jul 18 14:21:31 pmoney dbus[1149]: [system] Reloaded configuration
Jul 18 14:21:31 pmoney dbus-daemon: dbus[1149]: [system] Reloaded configuration
Jul 18 14:21:56 pmoney rsyslogd-3000: Could not open dynamic file '/var/log/remote/idrac155/stockwood.pink.priv/common.log' [state -3000] - discarding message
Jul 18 14:22:00 pmoney rsyslogd-3000: Could not open dynamic file '/var/log/remote/idrac155/woodstock.pink.priv/common.log' [state -3000] - discarding message
Jul 18 14:22:28 pmoney dbus-daemon: dbus[1149]: avc:  received policyload notice (seqno=5)
Jul 18 14:22:28 pmoney dbus[1149]: avc:  received policyload notice (seqno=5)

Jul 18 14:40:54 pmoney rsyslogd-3000: Could not open dynamic file '/var/log/remote/idrac151/helo.pink.priv/common.log' [state -3000] - discarding message
Jul 18 14:41:07 pmoney rsyslogd-3000: Could not open dynamic file '/var/log/remote/idrac151/helo.pink.priv/common.log' [state -3000] - discarding message
Jul 18 14:41:18 pmoney rsyslogd-3000: Could not open dynamic file '/var/log/remote/idrac151/helo.pink.priv/common.log' [state -3000] - discarding message

Then perform the following on the individual directories themselves:

# restorecon -v /var/log/remote/idrac155
# semanage fcontext -a -t var_log_t "/var/log/remote/idrac155"
# restorecon -v /var/log/remote/idrac155
restorecon reset /var/log/remote/idrac155 context system_u:object_r:syslogd_var_lib_t:s0->system_u:object_r:var_log_t:s0

# semanage fcontext -a -t var_log_t "/var/log/remote/idrac151"
# restorecon -v /var/log/remote/idrac151
restorecon reset /var/log/remote/idrac151 context system_u:object_r:syslogd_var_lib_t:s0->system_u:object_r:var_log_t:s0

W: There is no public key available for the following key IDs error - Debian 7

When using apt-get update, you get this error:

# apt-get update
Ign cdrom://[Debian GNU/Linux 7.6.0 _Wheezy_ - Official i386 DVD Binary-1 20140712-13:02] wheezy Release.gpg
Ign cdrom://[Debian GNU/Linux 7.6.0 _Wheezy_ - Official i386 DVD Binary-1 20140712-13:02] wheezy Release
Ign cdrom://[Debian GNU/Linux 7.6.0 _Wheezy_ - Official i386 DVD Binary-1 20140712-13:02] wheezy/contrib i386 Packages/DiffIndex
Ign cdrom://[Debian GNU/Linux 7.6.0 _Wheezy_ - Official i386 DVD Binary-1 20140712-13:02] wheezy/main i386 Packages/DiffIndex
Ign cdrom://[Debian GNU/Linux 7.6.0 _Wheezy_ - Official i386 DVD Binary-1 20140712-13:02] wheezy/contrib Translation-en_GB
Ign cdrom://[Debian GNU/Linux 7.6.0 _Wheezy_ - Official i386 DVD Binary-1 20140712-13:02] wheezy/main Translation-en_GB
Hit http://security.debian.org wheezy/updates Release.gpg
Hit http://security.debian.org wheezy/updates Release
Hit http://security.debian.org wheezy/updates/main Sources
Hit http://security.debian.org wheezy/updates/contrib Sources
Hit http://security.debian.org wheezy/updates/main i386 Packages
Hit http://security.debian.org wheezy/updates/contrib i386 Packages
Hit http://security.debian.org wheezy/updates/contrib Translation-en
Hit http://security.debian.org wheezy/updates/main Translation-en
Reading package lists... Done 
W: There is no public key available for the following key IDs:
9D6D8F6BC857C906

Pull down the missing key, using:

# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 9D6D8F6BC857C906
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /tmp/tmp.bzDMKZFH54 --trustdb-name /etc/apt//trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyring /etc/apt/trusted.gpg.d//debian-archive-squeeze-automatic.gpg --keyring /etc/apt/trusted.gpg.d//debian-archive-squeeze-stable.gpg --keyring /etc/apt/trusted.gpg.d//debian-archive-wheezy-automatic.gpg --keyring /etc/apt/trusted.gpg.d//debian-archive-wheezy-stable.gpg --keyserver keyserver.ubuntu.com --recv-keys 9D6D8F6BC857C906
gpg: requesting key C857C906 from hkp server keyserver.ubuntu.com
gpg: key C857C906: public key "Debian Security Archive Automatic Signing Key (8/jessie) " imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
root@sonts-Mac-mini:~# apt-get update
Ign cdrom://[Debian GNU/Linux 7.6.0 _Wheezy_ - Official i386 DVD Binary-1 20140712-13:02] wheezy Release.gpg
Ign cdrom://[Debian GNU/Linux 7.6.0 _Wheezy_ - Official i386 DVD Binary-1 20140712-13:02] wheezy Release
Ign cdrom://[Debian GNU/Linux 7.6.0 _Wheezy_ - Official i386 DVD Binary-1 20140712-13:02] wheezy/contrib i386 Packages/DiffIndex
Ign cdrom://[Debian GNU/Linux 7.6.0 _Wheezy_ - Official i386 DVD Binary-1 20140712-13:02] wheezy/main i386 Packages/DiffIndex
Ign cdrom://[Debian GNU/Linux 7.6.0 _Wheezy_ - Official i386 DVD Binary-1 20140712-13:02] wheezy/contrib Translation-en_GB
Ign cdrom://[Debian GNU/Linux 7.6.0 _Wheezy_ - Official i386 DVD Binary-1 20140712-13:02] wheezy/main Translation-en_GB
Hit http://security.debian.org wheezy/updates Release.gpg
Hit http://security.debian.org wheezy/updates Release   
Hit http://security.debian.org wheezy/updates/main Sources
Hit http://security.debian.org wheezy/updates/contrib Sources
Hit http://security.debian.org wheezy/updates/main i386 Packages
Hit http://security.debian.org wheezy/updates/contrib i386 Packages
Hit http://security.debian.org wheezy/updates/contrib Translation-en
Hit http://security.debian.org wheezy/updates/main Translation-en
Reading package lists... Done 

SELinux issue with Apache certificates

So you are setting up Apache SSL (mod_ssl) and installing the certificates in /etc/pki/tls/certs and have configured your ssl.conf file to point at them. Starting or restarting Apache then give you these errors:

# tail /var/log/httpd/error_log
[Wed Jun 29 16:07:55.097464 2016] [auth_digest:notice] [pid 6159] AH01757: generating secret for digest authentication ...
[Wed Jun 29 16:07:55.098146 2016] [lbmethod_heartbeat:notice] [pid 6159] AH02282: No slotmem from mod_heartmonitor
[Wed Jun 29 16:07:55.098646 2016] [ssl:emerg] [pid 6159] AH02311: Fatal error initialising mod_ssl, exiting. See /etc/httpd/logs/ssl_error_log for more information
[Wed Jun 29 16:08:26.608731 2016] [core:notice] [pid 16249] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Wed Jun 29 16:08:26.609618 2016] [suexec:notice] [pid 16249] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Jun 29 16:08:26.610045 2016] [ssl:emerg] [pid 16249] AH02311: Fatal error initialising mod_ssl, exiting. See /etc/httpd/logs/ssl_error_log for more information

# tail /var/log/httpd/ssl_error_log
[Wed Jun 29 16:07:55.098595 2016] [ssl:emerg] [pid 6159] AH01895: Unable to configure verify locations for client authentication
[Wed Jun 29 16:07:55.098617 2016] [ssl:emerg] [pid 6159] SSL Library Error: error:0200100D:system library:fopen:Permission denied (fopen('/etc/pki/tls/certs/QuoVadisEVRootCertificate.crt','r'))
[Wed Jun 29 16:07:55.098626 2016] [ssl:emerg] [pid 6159] SSL Library Error: error:2006D002:BIO routines:BIO_new_file:system lib
[Wed Jun 29 16:07:55.098634 2016] [ssl:emerg] [pid 6159] SSL Library Error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
[Wed Jun 29 16:08:26.610008 2016] [ssl:emerg] [pid 16249] AH01895: Unable to configure verify locations for client authentication
[Wed Jun 29 16:08:26.610024 2016] [ssl:emerg] [pid 16249] SSL Library Error: error:0200100D:system library:fopen:Permission denied (fopen('/etc/pki/tls/certs/QuoVadisEVRootCertificate.crt','r'))
[Wed Jun 29 16:08:26.610032 2016] [ssl:emerg] [pid 16249] SSL Library Error: error:2006D002:BIO routines:BIO_new_file:system lib
[Wed Jun 29 16:08:26.610041 2016] [ssl:emerg] [pid 16249] SSL Library Error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib

Then running this command will fix the problem:

# restorecon -RvF /etc/ssl/certs/
restorecon reset /etc/pki/tls/certs/QuoVadisEVIntermediateCertificate.crt context unconfined_u:object_r:user_home_t:s0->system_u:object_r:cert_t:s0
restorecon reset /etc/pki/tls/certs/localhost.crt context unconfined_u:object_r:cert_t:s0->system_u:object_r:cert_t:s0
restorecon reset /etc/pki/tls/certs/QuoVadisEVRootCertificate.crt context unconfined_u:object_r:user_home_t:s0->system_u:object_r:cert_t:s0
restorecon reset /etc/pki/tls/certs/redcapbrtc.crt context unconfined_u:object_r:cert_t:s0->system_u:object_r:cert_t:s0

GitLab CE update broke service!

A recent GitLab CE update left the service down. A restart of the service resulted in this error when trying to login:

Could not authenticate you from Ldapmain because "Pg::undefinedtable: error: relation "u2f registrations" does not exist line 5: where a.attrelid = '"u2f registrations"'::reg... ^ : select a.attname, format type(a.atttypid, a.atttypmod), pg get expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod from pg attribute a left join pg attrdef d on a.attrelid = d.adrelid and a.attnum = d.adnum where a.attrelid = '"u2f registrations"'::regclass and a.attnum > 0 and not a.attisdropped order by a.attnum ".

A bit strange and worrying that maybe the database was corrupted and/or missing. But after an initial google for "gitlab u2f registrations" revealed that is was 2 factor authentication - which we are not using here. So first thing was to find out how to disable this system-wide - apparently this was the command: gitlab-rake gitlab:two_factor:disable_for_all_users, but running it resulted in the following:

[root@gitlab gitlab]# gitlab-rake gitlab:two_factor:disable_for_all_users
rake aborted!
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "u2f_registrations" does not exist
LINE 1: ...STINCT "users"."id") FROM "users" LEFT OUTER JOIN u2f_regist...
                                                             ^
: SELECT DISTINCT COUNT(DISTINCT "users"."id") FROM "users" LEFT OUTER JOIN u2f_registrations AS u2f ON u2f.user_id = users.id WHERE (u2f.id IS NOT NULL OR otp_required_for_login = 't')
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `async_exec'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract_adapter.rb:472:in `block in log'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract_adapter.rb:466:in `log'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:264:in `execute_simple_calculation'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:221:in `perform_calculation'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:127:in `calculate'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:42:in `count'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/two_factor.rake:6:in `block (3 levels) in '
PG::UndefinedTable: ERROR:  relation "u2f_registrations" does not exist
LINE 1: ...STINCT "users"."id") FROM "users" LEFT OUTER JOIN u2f_regist...
                                                             ^
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `async_exec'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract_adapter.rb:472:in `block in log'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract_adapter.rb:466:in `log'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:264:in `execute_simple_calculation'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:221:in `perform_calculation'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:127:in `calculate'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:42:in `count'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/two_factor.rake:6:in `block (3 levels) in '
Tasks: TOP => gitlab:two_factor:disable_for_all_users
(See full trace by running task with --trace)

So obviously that was not the problem. So let's take a snapshot of this gitlab VM and run a reconfigure:

[root@gitlab gitlab]# gitlab-ctl reconfigure
Starting Chef Client, version 12.6.0
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
  - package (0.0.0)
  - gitlab (0.0.1)
  - runit (0.14.2)
Compiling Cookbooks...
Recipe: gitlab::default
  * directory[/etc/gitlab] action create (up to date)
/sbin/init: unrecognized option '--version'
-.mount                                                                                                         loaded active mounted   /
  Converging 282 resources
  * directory[/etc/gitlab] action nothing (skipped due to action :nothing)
  * directory[/var/opt/gitlab] action create (up to date)
  * directory[/opt/gitlab/embedded/etc] action create (up to date)
  * template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date)
Recipe: gitlab::web-server
  * group[gitlab-www] action create (up to date)
  * user[gitlab-www] action create (up to date)
Recipe: gitlab::users
  * directory[/var/opt/gitlab] action create (up to date)
  * group[git] action create (up to date)
  * user[git] action create (up to date)
  * template[/var/opt/gitlab/.gitconfig] action create (up to date)
Recipe: gitlab::gitlab-shell
  * directory[/var/opt/gitlab/git-data/repositories] action create (up to date)
  * directory[/var/opt/gitlab/git-data] action create (up to date)
  * directory[/var/opt/gitlab/.ssh] action create (up to date)
  * file[/var/opt/gitlab/.ssh/authorized_keys] action create (up to date)
  * directory[/opt/gitlab/embedded/service/gitlab-shell/hooks/] action create (up to date)
  * file[/opt/gitlab/embedded/service/gitlab-shell/authorized_keys.lock] action create (up to date)
  * execute[chcon --recursive --type ssh_home_t /var/opt/gitlab/.ssh] action runid: --context (-Z) works only on an SELinux-enabled kernel
 (skipped due to only_if)
  * directory[/var/log/gitlab/gitlab-shell/] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-shell] action create (up to date)
  * template[/var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-shell/config.yml] action create (up to date)
  * template[/var/opt/gitlab/gitlab-shell/gitlab_shell_secret] action create (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret] action create (up to date)
Recipe: gitlab::gitlab-rails
  * directory[/var/log/gitlab] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-rails/shared] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-rails/shared/artifacts] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-rails/shared/lfs-objects] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-rails/uploads] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-ci/builds] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-rails/shared/pages] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-rails/etc] action create (up to date)
  * directory[/opt/gitlab/etc/gitlab-rails] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-rails/working] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-rails/tmp] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-rails/upgrade-status] action create (up to date)
  * directory[/var/log/gitlab/gitlab-rails] action create (up to date)
  * directory[/var/opt/gitlab/backups] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-rails] action create (up to date)
  * directory[/var/opt/gitlab/gitlab-ci] action create (up to date)
  * template[/opt/gitlab/etc/gitlab-rails/gitlab-rails-rc] action create (up to date)
  * template[/var/opt/gitlab/gitlab-rails/etc/secret] action create (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/.secret] action create (up to date)
  * template[/var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/config/database.yml] action create (up to date)
  * template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml] action create (up to date)
  * template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/config/resque.yml] action create (up to date)
  * template[/var/opt/gitlab/gitlab-rails/etc/aws.yml] action delete (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/config/aws.yml] action delete (up to date)
  * template[/var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action delete (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/config/initializers/smtp_settings.rb] action delete (up to date)
  * template[/var/opt/gitlab/gitlab-rails/etc/relative_url.rb] action delete (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/config/initializers/relative_url.rb] action delete (up to date)
  * template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml] action create (up to date)
  * template[/var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] action create (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/config/initializers/rack_attack.rb] action create (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret] action create (up to date)
  * directory[/opt/gitlab/etc/gitlab-rails/env] action create (up to date)
  * file[/opt/gitlab/etc/gitlab-rails/env/HOME] action create (up to date)
  * file[/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV] action create (up to date)
  * file[/opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS] action create (up to date)
  * file[/opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE] action create (up to date)
  * file[/opt/gitlab/etc/gitlab-rails/env/PATH] action create (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/tmp] action create (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/public/uploads] action create (up to date)
  * link[/opt/gitlab/embedded/service/gitlab-rails/log] action create (up to date)
  * link[/var/log/gitlab/gitlab-rails/sidekiq.log] action create (skipped due to not_if)
  * file[/opt/gitlab/embedded/service/gitlab-rails/db/schema.rb] action create (up to date)
  * remote_file[/var/opt/gitlab/gitlab-rails/VERSION] action create (up to date)
  * remote_file[/var/opt/gitlab/gitlab-rails/REVISION] action create (up to date)
  * file[/var/opt/gitlab/gitlab-rails/RUBY_VERSION] action create (up to date)
  * execute[chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public] action run
    - execute chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public
  * execute[clear the gitlab-rails cache] action nothing (skipped due to action :nothing)
  * bash[generate assets] action nothing (skipped due to action :nothing)
  * file[/var/opt/gitlab/gitlab-rails/config.ru] action delete (up to date)
Recipe: gitlab::gitlab-ci-proxying
  * template[/var/opt/gitlab/nginx/conf/gitlab-ci-http.conf] action delete (up to date)
Recipe: gitlab::selinux
  * execute[semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-7.2.0-ssh-keygen.pp] action runDisabled
 (skipped due to not_if)
Recipe: gitlab::add_trusted_certs
  * directory[/etc/gitlab/ssl/trusted-certs] action create (up to date)
  * directory[/opt/gitlab/embedded/ssl/certs] action create (up to date)
  * file[/opt/gitlab/embedded/ssl/certs/README] action create (up to date)
  * ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] action run

  * Moving existing certificates found in /opt/gitlab/embedded/ssl/certs
ERROR: /opt/gitlab/embedded/ssl/certs/pecacert.pem: OpenSSL error: nested asn1 error!


    ================================================================================
    Error executing action `run` on resource 'ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs]'
    ================================================================================

    RuntimeError
    ------------
    ERROR: Not a certificate: /opt/gitlab/embedded/ssl/certs/pecacert.pem / /opt/gitlab/embedded/ssl/certs/pecacert.pem

    Cookbook Trace:
    ---------------
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/helper.rb:514:in `raise_msg'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/helper.rb:460:in `block in move_existing_certificates'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/helper.rb:453:in `glob'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/helper.rb:453:in `move_existing_certificates'
    /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/add_trusted_certs.rb:31:in `block (2 levels) in from_file'

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/add_trusted_certs.rb

     28: ruby_block "Move existing certs and link to #{ssl_certs_dir}" do
     29:   block do
     30:     puts "\n\n  * Moving existing certificates found in #{ssl_certs_dir}\n"
     31:     cert_helper.move_existing_certificates
     32:     puts "\n  * Symlinking existing certificates found in #{trusted_certs_dir}\n"
     33:     cert_helper.link_certificates
     34:   end
     35:   only_if { cert_helper.new_certificate_added? }
     36:   notifies :restart, "service[unicorn]"
     37: end

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/add_trusted_certs.rb:28:in `from_file'

    ruby_block("Move existing certs and link to /opt/gitlab/embedded/ssl/certs") do
      action [:run]
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      block_name "Move existing certs and link to /opt/gitlab/embedded/ssl/certs"
      declared_type :ruby_block
      cookbook_name "gitlab"
      recipe_name "add_trusted_certs"
      block #
      only_if { #code block }
    end


Running handlers:
Running handlers complete
Chef Client failed. 1 resources updated in 16 seconds

So the problem is that re-configure (an upgrade performs a reconfigure) failed leaving the gitlab service down... So the problem was moving the certificates. GitLab was now fixed by copying the certificates by hand into the required location and (re)moving the source directory:

[root@gitlab ~]# cd /opt/gitlab/embedded/ssl/certs/
[root@gitlab certs]# ls -la
total 256
drwxr-xr-x. 2 root root     55 Jun 24 10:03 .
drwxr-xr-x. 5 root root     76 Jun 23 03:45 ..
-rw-r--r--  1 root root 252499 Jun 22 11:05 cacert.pem
-rw-r--r--  1 root root    800 Apr  4 19:47 pecacert.pem
-rw-r--r--  1 root root    151 Jun 23 03:46 README
[root@gitlab certs]# cat README
This directory is managed by omnibus-gitlab.
 Any file placed in this directory will be ignored
. Place certificates in /etc/gitlab/ssl/trusted-certs.

[root@gitlab certs]# cp /opt/gitlab/embedded/ssl/certs/* /etc/gitlab/ssl/trusted-certs

[root@gitlab ssl]# mv certs certs.son
[root@gitlab ssl]# gitlab-ctl reconfigure
Starting Chef Client, version 12.6.0
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
  - runit (0.14.2)
  - package (0.0.0)
  - gitlab (0.0.1)
Compiling Cookbooks...
Recipe: gitlab::default
  * directory[/etc/gitlab] action create (up to date)
/sbin/init: unrecognized option '--version'


[successful reconfigure detail snipped]


Recipe: gitlab::add_trusted_certs
  * directory[/etc/gitlab/ssl/trusted-certs] action create (up to date)
  * directory[/opt/gitlab/embedded/ssl/certs] action create
    - create new directory /opt/gitlab/embedded/ssl/certs
    - change mode from '' to '0755'
  * file[/opt/gitlab/embedded/ssl/certs/README] action create
    - create new file /opt/gitlab/embedded/ssl/certs/README
    - update content in file /opt/gitlab/embedded/ssl/certs/README from none to e09a2d
    --- /opt/gitlab/embedded/ssl/certs/README   2016-06-24 10:14:17.275761125 +0100
    +++ /opt/gitlab/embedded/ssl/certs/.README20160624-29376-1q3bw7j    2016-06-24 10:14:17.275761125 +0100
    @@ -1 +1,4 @@
    +This directory is managed by omnibus-gitlab.
    + Any file placed in this directory will be ignored
    +. Place certificates in /etc/gitlab/ssl/trusted-certs.
    - change mode from '' to '0644'
  * ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] action run

  * Moving existing certificates found in /opt/gitlab/embedded/ssl/certs

  * Symlinking existing certificates found in /etc/gitlab/ssl/trusted-certs

 Skipping /etc/gitlab/ssl/trusted-certs/cacert.pem.

 Skipping /etc/gitlab/ssl/trusted-certs/pecacert.pem.

 Skipping /etc/gitlab/ssl/trusted-certs/README.


[successful reconfigure detail snipped]


    - execute "bash"  "/tmp/chef-script20160624-29376-v5yl6q"
Recipe: gitlab::gitlab-rails
  * execute[clear the gitlab-rails cache] action run
    - execute /opt/gitlab/bin/gitlab-rake cache:clear
Recipe: gitlab::unicorn
  * service[unicorn] action restart
    - restart service service[unicorn]
Recipe: gitlab::sidekiq
  * service[sidekiq] action restart
    - restart service service[sidekiq]
Recipe: gitlab::unicorn
  * service[unicorn] action restart
    - restart service service[unicorn]
Recipe: gitlab::gitlab-workhorse
  * service[gitlab-workhorse] action restart
    - restart service service[gitlab-workhorse]
Recipe: gitlab::nginx
  * service[nginx] action restart
    - restart service service[nginx]

Running handlers:
Running handlers complete
Chef Client finished, 22/294 resources updated in 43 seconds
gitlab Reconfigured!

Service is up and running and login is just fine!

Shrinking filesystems with LVM - CentOS

So you have run out of space on one of your LVM volumes, but there's another volume with too much space that's not being used. You need to shrink one and grow the other! First thing first! DO NOT SHRINK A (LVM) VOLUME WITHOUT resizing the FILESYSTEM first! Your data (filesystem) will be screwed! So let's have a go. Here will have a full volume /filetek/sth/sthdbs, and we have a volume being used as a mount point with too much space /filetek. We need to shrink /filetek and expand /filetek/sth/sthdbs.

root@server /
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/volg0-root
                      969M  526M  394M  58% /
/dev/sda1             190M   85M   96M  47% /boot
/dev/mapper/volg0-home
                      4.7G  173M  4.3G   4% /home
/dev/mapper/volg0-usr
                      3.8G 1018M  2.6G  28% /usr
/dev/mapper/volg0-var
                      3.8G  470M  3.2G  13% /var
/dev/mapper/volg1-filetek
                      197G  9.4G  178G   6% /filetek
/dev/mapper/volg1-filetek--tmp
                       50G   52M   47G   1% /filetek/tmp
/dev/mapper/volg1-filetek--jrnl1
                       50G  885M   46G   2% /filetek/jrnl1
/dev/mapper/volg1-filetek--sthdbs
                      1.0T  970G  1.7G 100% /filetek/sth/sthdbs
/dev/mapper/volg1-filetek2
                      197G  1.6G  186G   1% /filetek2
/dev/mapper/volg1-filetek2--tmp2
                       50G   52M   47G   1% /filetek2/tmp2
/dev/mapper/volg1-filetek--jrnl2
                       50G  885M   46G   2% /filetek2/jrnl2

Let's umount all the volumes involved:

root@server ~
$ umount /filetek/tmp /filetek/jrnl1 /filetek/sth/sthdbs
root@server ~
$ umount /filetek

Let's resize (shrink) /filetek to 32GB by fsck'ing it first:

root@server ~
$ fsck -f /dev/mapper/volg1-filetek
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
filetek: 2119/13107200 files (2.1% non-contiguous), 3307507/52428800 blocks

Using resize2fs:

root@server ~
$ resize2fs /dev/mapper/volg1-filetek 32G
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/mapper/volg1-filetek to 8388608 (4k) blocks.
The filesystem on /dev/mapper/volg1-filetek is now 8388608 blocks long.

Now we can safely reduce the LVM volume size with the lvreduce command:

root@server ~
$ lvreduce -L 40G /dev/mapper/volg1-filetek
  WARNING: Reducing active logical volume to 40.00 GiB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce filetek? [y/n]: y
  Size of logical volume volg1/filetek changed from 200.00 GiB (51200 extents) to 40.00 GiB (10240 extents).
  Logical volume filetek successfully resized

All, very good so let's expand our volume that has run out of space:

root@server ~
$ lvextend --size +200G /dev/mapper/volg1-filetek--sthdbs
  Size of logical volume volg1/filetek-sthdbs changed from 1.02 TiB (266240 extents) to 1.21 TiB (317440 extents).
  Logical volume filetek-sthdbs successfully resized

Now we need to grow the filesystem, but first let's fsck (e2fsck) it:

root@server ~
$ e2fsck -f /dev/mapper/volg1-filetek--sthdbs
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
filetek-sthdbs: 1170/68157440 files (21.9% non-contiguous), 258555686/272629760 blocks

root@server ~
$ resize2fs /dev/mapper/volg1-filetek--sthdbs
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/mapper/volg1-filetek--sthdbs to 325058560 (4k) blocks.
The filesystem on /dev/mapper/volg1-filetek--sthdbs is now 325058560 blocks long.

Let's re-mount both filesystems to file all is good!

root@server ~
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/volg0-root
                      969M  526M  394M  58% /
/dev/sda1             190M   85M   96M  47% /boot
/dev/mapper/volg0-home
                      4.7G  173M  4.3G   4% /home
/dev/mapper/volg0-usr
                      3.8G 1018M  2.6G  28% /usr
/dev/mapper/volg0-var
                      3.8G  470M  3.2G  13% /var
/dev/mapper/volg1-filetek2
                       32G  1.6G   29G   6% /filetek2
/dev/mapper/volg1-filetek--tmp
                       50G   52M   47G   1% /filetek/tmp
/dev/mapper/volg1-filetek--jrnl1
                       50G  885M   46G   2% /filetek/jrnl1
/dev/mapper/volg1-filetek2--tmp2
                       50G   52M   47G   1% /filetek2/tmp2
/dev/mapper/volg1-filetek--jrnl2
                       50G  885M   46G   2% /filetek2/jrnl2
/dev/mapper/volg1-filetek
                       32G  9.4G   21G  32% /filetek
/dev/mapper/volg1-filetek--sthdbs
                      1.2T  970G  189G  84% /filetek/sth/sthdbs

Resizing Filesystems with LVM - CentOS 7

See how much space is available on physical disk:

[root@centos ~]# pvs
  PV         VG    Fmt  Attr PSize   PFree
  /dev/sda2  volg0 lvm2 a--  199.00g 21.00g

See what sizes your logical volumes are:
[root@centos ~]# lvs
  LV   VG    Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home volg0 -wi-ao---- 16.00g
  opt  volg0 -wi-ao---- 32.00g
  root volg0 -wi-ao---- 32.00g
  swap volg0 -wi-ao---- 66.00g
  var  volg0 -wi-ao---- 32.00g

See what size your filesystems are:

[root@centos ~]# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/volg0-root   32G  1.2G   29G   4% /
devtmpfs                 32G     0   32G   0% /dev
tmpfs                    32G     0   32G   0% /dev/shm
tmpfs                    32G  9.0M   32G   1% /run
tmpfs                    32G     0   32G   0% /sys/fs/cgroup
/dev/sda1               976M  129M  796M  14% /boot
/dev/mapper/volg0-home   16G   45M   15G   1% /home
/dev/mapper/volg0-var    32G  172M   30G   1% /var
/dev/mapper/volg0-opt    32G   49M   30G   1% /opt
tmpfs                   6.3G     0  6.3G   0% /run/user/0

Resize your logical volume - in this case /var adding 10G to current 32G = 42G:

[root@centos ~]# lvresize -L 42G /dev/mapper/volg0-var
  Size of logical volume volg0/var changed from 32.00 GiB (8192 extents) to 42.00 GiB (10752 extents).
  Logical volume var successfully resized.

Underlining volume is expanded, but not filesystem on top:

[root@centos ~]# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/volg0-root   32G  1.2G   29G   4% /
devtmpfs                 32G     0   32G   0% /dev
tmpfs                    32G     0   32G   0% /dev/shm
tmpfs                    32G  9.0M   32G   1% /run
tmpfs                    32G     0   32G   0% /sys/fs/cgroup
/dev/sda1               976M  129M  796M  14% /boot
/dev/mapper/volg0-home   16G   45M   15G   1% /home
/dev/mapper/volg0-var    32G  172M   30G   1% /var
/dev/mapper/volg0-opt    32G   49M   30G   1% /opt
tmpfs                   6.3G     0  6.3G   0% /run/user/0

So resize filesystem on top:

[root@centos ~]# resize2fs /dev/mapper/volg0-var
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/mapper/volg0-var is mounted on /var; on-line resizing required
old_desc_blocks = 4, new_desc_blocks = 6
The filesystem on /dev/mapper/volg0-var is now 11010048 blocks long.

See now filesystem is 42GB in size:

[root@centos ~]# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/volg0-root   32G  1.2G   29G   4% /
devtmpfs                 32G     0   32G   0% /dev
tmpfs                    32G     0   32G   0% /dev/shm
tmpfs                    32G  9.0M   32G   1% /run
tmpfs                    32G     0   32G   0% /sys/fs/cgroup
/dev/sda1               976M  129M  796M  14% /boot
/dev/mapper/volg0-home   16G   45M   15G   1% /home
/dev/mapper/volg0-var    42G  172M   40G   1% /var
/dev/mapper/volg0-opt    32G   49M   30G   1% /opt
tmpfs                   6.3G     0  6.3G   0% /run/user/0

Job done:

[root@centos ~]#

Stop logging F5 probes - Apache 2

The log format for Apache to log real IP of requests on the F5 (make sure X-Forwarding is enabled) is:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{X-Forwarded-For}i\"" combined

To tell Apache not to log the probes from the F5 (which is needed for loadbalancing) is to have these directives:

SetEnvIf Remote_Addr IP_range_or_specific_IP dontlog

So IP_range_or_specific_IP is the regex in quotes, e.g "192\.168\.1\." for range and "192\.168\.1\.42" for IP. You will also need this setting on the statement that does the logging:

CustomLog "logs/access_log" combined env=!dontlog

Restart Apache with the graceful command.

Error: /sbin/mingetty: hvc0: cannot open tty: No such device or address - CentOS 6

I get this annoying error message on some of my servers which fills up the /var/log/messages file. My solution for this is to edit the /etc/sysconfig/init file, removing it on this line:

# What ttys should gettys be started on?
ACTIVE_CONSOLES="/dev/tty[1-6] /dev/hvc0"

You will need to reboot to make the messages go away! Restarting init using 'telinit u' is not enough! Or you can just create the device file using mknod!

# mknod -m 600 /dev/hvc0 c 5 1

Resizing VM system disk - LVM Partition 1 does not start on physical sector boundary - CentOS 7

So you have a virtual machine and you've just expanded your virtual disk, log on to your CentOS Linux system and performed a fdisk -l to find this error:

# fdisk -l

Disk /dev/sda: 236.2 GB, 236223201280 bytes, 461373440 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: dos
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1   209715199   104857599+  ee  GPT
Partition 1 does not start on physical sector boundary.

Disk /dev/mapper/centos-root: 34.4 GB, 34359738368 bytes, 67108864 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/centos-swap: 4294 MB, 4294967296 bytes, 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/centos-tmp: 4294 MB, 4294967296 bytes, 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/centos-var: 34.4 GB, 34359738368 bytes, 67108864 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/centos-opt: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/centos-home: 7507 MB, 7507804160 bytes, 14663680 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Knowing that you have an /dev/sda2 and /dev/sda3 and want to create an /dev/sda4 but how? Install parted, and use this to fix the disk you've just expanded:

# parted
GNU Parted 3.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Error: The backup GPT table is not at the end of the disk, as it should be.  This might mean that another operating system believes the disk is smaller.  Fix, by
moving the backup to the end (and removing the old backup)?
Fix/Ignore/Cancel? Fix
Warning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 251658240 blocks) or continue with the
current setting?
Fix/Ignore? Fix
Model: Msft Virtual Disk (scsi)
Disk /dev/sda: 236GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size   File system  Name                  Flags
 1      1049kB  538MB   537MB  ext4
 2      538MB   1075MB  537MB  fat16        EFI System Partition  boot
 3      1075MB  107GB   106GB                                     lvm

(parted) q

Now you can see all three partitions and so will be able to create the fourth:

# fdisk /dev/sda
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p

Disk /dev/sda: 236.2 GB, 236223201280 bytes, 461373440 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt


#         Start          End    Size  Type            Name
 1         2048      1050623    512M  Microsoft basic
 2      1050624      2099199    512M  EFI System      EFI System Partition
 3      2099200    209713151     99G  Linux LVM

Command (m for help): n
Partition number (4-128, default 4):
First sector (34-461373406, default 209713152):
Last sector, +sectors or +size{K,M,G,T,P} (209713152-461373406, default 461373406):
Created partition 4


Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

Now you have created the forth partition, you can see it with the fdisk -l command:

[root@troi st14988]# fdisk -l
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/sda: 236.2 GB, 236223201280 bytes, 461373440 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk label type: gpt


#         Start          End    Size  Type            Name
 1         2048      1050623    512M  Microsoft basic
 2      1050624      2099199    512M  EFI System      EFI System Partition
 3      2099200    209713151     99G  Linux LVM
 4    209713152    461373406    120G  Linux filesyste

Disk /dev/mapper/centos-root: 34.4 GB, 34359738368 bytes, 67108864 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/centos-swap: 4294 MB, 4294967296 bytes, 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/centos-tmp: 4294 MB, 4294967296 bytes, 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/centos-var: 34.4 GB, 34359738368 bytes, 67108864 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/centos-opt: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/centos-home: 7507 MB, 7507804160 bytes, 14663680 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Once you have rebooted for the new partition table to take effect, you can add the new partition to the LVM volume group to extend the logical volumes you need to grow/resize:

# vgs
  VG     #PV #LV #SN Attr   VSize  VFree
  centos   1   6   0 wz--n- 99.00g 4.00m

# vgextend centos /dev/sda4
  Physical volume "/dev/sda4" successfully created
  Volume group "centos" successfully extended

# vgs
  VG     #PV #LV #SN Attr   VSize   VFree
  centos   2   6   0 wz--n- 218.99g 120.00g

Now that you have more space, we can extend any of our logical volumes we want:

# lvs
  LV   VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home centos -wi-ao----  6.99g
  opt  centos -wi-ao---- 20.00g
  root centos -wi-ao---- 32.00g
  swap centos -wi-ao----  4.00g
  tmp  centos -wi-ao----  4.00g
  var  centos -wi-ao---- 32.00g

# lvextend -L +100G centos/opt
  Size of logical volume centos/opt changed from 20.00 GiB (5120 extents) to 120.00 GiB (30720 extents).
  Logical volume opt successfully resized.

# lvs
  LV   VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home centos -wi-ao----   6.99g
  opt  centos -wi-ao---- 120.00g
  root centos -wi-ao----  32.00g
  swap centos -wi-ao----   4.00g
  tmp  centos -wi-ao----   4.00g
  var  centos -wi-ao----  32.00g

# lvextend centos/home -l+100%FREE
  Size of logical volume centos/home changed from 6.99 GiB (1790 extents) to 26.99 GiB (6910 extents).
  Logical volume home successfully resized.

# lvs
  LV   VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home centos -wi-ao----  26.99g
  opt  centos -wi-ao---- 120.00g
  root centos -wi-ao----  32.00g
  swap centos -wi-ao----   4.00g
  tmp  centos -wi-ao----   4.00g
  var  centos -wi-ao----  32.00g

MongoDB won't start - ERROR: dbpath (/data/db) does not exist.

For some reason - due to a bug probably - mongodb (or mongod) database won't start on default install because it is referencing the running direction /data/db to create a lock file. This error occurs even if you set the dbpath in /etc/mongod.conf to somewhere else (default installation says: /var/lib/mongodb). To fix this, simply create a simlink or the directory /data/db itself!

# service mongod start
Redirecting to /bin/systemctl start  mongod.service
Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.
# systemctl status mongod.service
● mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2016-03-03 09:50:01 GMT; 6s ago
  Process: 21502 ExecStart=/usr/bin/mongod $OPTIONS run (code=exited, status=45)

Mar 03 09:50:01 p0 systemd[1]: Starting High-performance, schema-free document-oriented database...
Mar 03 09:50:01 p0 mongod[21502]: about to fork child process, waiting until server is ready for connections.
Mar 03 09:50:01 p0 mongod[21502]: forked process: 21504
Mar 03 09:50:01 p0 mongod[21502]: ERROR: child process failed, exited with error number 45
Mar 03 09:50:01 p0 systemd[1]: mongod.service: control process exited, code=exited status=45
Mar 03 09:50:01 p0 systemd[1]: Failed to start High-performance, schema-free document-oriented database.
Mar 03 09:50:01 p0 systemd[1]: Unit mongod.service entered failed state.
Mar 03 09:50:01 p0 systemd[1]: mongod.service failed.

# mongod --repair
2016-03-03T09:55:49.905+0000 [initandlisten] MongoDB starting : pid=21520 port=27017 dbpath=/data/db 64-bit host=comp-sscm-p0
2016-03-03T09:55:49.906+0000 [initandlisten] db version v2.6.11
2016-03-03T09:55:49.906+0000 [initandlisten] git version: nogitversion
2016-03-03T09:55:49.906+0000 [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-03-03T09:55:49.906+0000 [initandlisten] build info: Linux buildvm-08.phx2.fedoraproject.org 4.1.4-100.fc21.x86_64 #1 SMP Tue Aug 4 03:25:05 UTC 2015 x86_64 BOOST_LIB_VERSION=1_53
2016-03-03T09:55:49.906+0000 [initandlisten] allocator: tcmalloc
2016-03-03T09:55:49.906+0000 [initandlisten] options: { repair: true }
2016-03-03T09:55:49.906+0000 [initandlisten] exception in initAndListen: 10296
*********************************************************************
 ERROR: dbpath (/data/db) does not exist.
 Create this directory or give existing directory in --dbpath.
 See http://dochub.mongodb.org/core/startingandstoppingmongo
*********************************************************************
, terminating
2016-03-03T09:55:49.906+0000 [initandlisten] dbexit:
2016-03-03T09:55:49.906+0000 [initandlisten] shutdown: going to close listening sockets...
2016-03-03T09:55:49.906+0000 [initandlisten] shutdown: going to flush diaglog...
2016-03-03T09:55:49.906+0000 [initandlisten] shutdown: going to close sockets...
2016-03-03T09:55:49.906+0000 [initandlisten] shutdown: waiting for fs preallocator...
2016-03-03T09:55:49.906+0000 [initandlisten] shutdown: closing all files...
2016-03-03T09:55:49.906+0000 [initandlisten] closeAllFiles() finished
2016-03-03T09:55:49.906+0000 [initandlisten] dbexit: really exiting now

# mkdir /data
# ln -s /var/lib/mongodb /data/db

or

# mkdir -p /data/db
# chown mongodb:mongodb /data/db

# service mongod start
Redirecting to /bin/systemctl start  mongod.service
# systemctl status mongod.service
● mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-03-03 09:58:15 GMT; 55s ago
  Process: 21863 ExecStart=/usr/bin/mongod $OPTIONS run (code=exited, status=0/SUCCESS)
 Main PID: 21865 (mongod)
   CGroup: /system.slice/mongod.service
           └─21865 /usr/bin/mongod --quiet -f /etc/mongod.conf run

Mar 03 09:58:15 p0 systemd[1]: Starting High-performance, schema-free document-oriented database...
Mar 03 09:58:15 p0 mongod[21863]: about to fork child process, waiting until server is ready for connections.
Mar 03 09:58:15 p0 mongod[21863]: forked process: 21865
Mar 03 09:58:15 p0 systemd[1]: Started High-performance, schema-free document-oriented database.
Mar 03 09:58:58 p0 systemd[1]: Started High-performance, schema-free document-oriented database.

Add entry in /etc/nsswitch - Solaris 11

To add DNS to /etc/nsswitch.conf

svccfg -s name-service/switch setprop config/ipnodes = astring: '("files dns")'
svccfg -s name-service/switch setprop config/host = astring: '("files dns")'
svccfg -s name-service/switch listprop config
svcadm refresh name-service/switch

Regex with awk

To use a regular expression in awk use the operator ~ and //, e.g/:

cat home-dir.txt | awk '{if ($3 !~ /^[0-9]/) print $0;}'

to print any line that does not have a number in column 3

RedHat 7 - Software failed to update

If you have issues with yum updating your OS or packages, with this error:

Error downloading packages:
  lsscsi-0.27-3.el7.x86_64: [Errno 256] No more mirrors to try.

Mount the DVD-ROM with the RedHat EL 7 source disk:

# mount /dev/cdrom /cdrom

Make sure your yum repo has this file: cdrom.repo:

[CDROM]
name=CDROM
baseurl=file:///cdrom
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

Make sure you are joined to the RHN with a valid subscription:

And if you have this error: rhn_register fails with "ERROR: can not find RHNS CA file:/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT", fix this using the instructions here: https://access.redhat.com/solutions/26435?tour=6 correcting these lines in /etc/sysconfig/rhn/up2date:

serverURL=https://xmlrpc.rhn.redhat.com/XMLRPC
sslCACert=/usr/share/rhn/RHNS-CA-CERT

Install .net Framework 3.5 on Windows Server 2012

Start up a PowerShell as Administrator and run this:

> Install-WindowsFeature -name NET-Framework-Core -source J:\sources\sxs

Where J is your install media drive letter.