Notes of a Systems Administrator

Friday, 4 November 2016

GitLab CE - repo web page not updating

So you've just created a GitLab repo and push the git working directory contents to it but the page does not change from the default instructions. To fix this log on to the gitlab server and perform the following command as root:

# gitlab-rake cache:clear

This is on the following GitLab CE:

GitLab 8.13.3

GitLab Shell 3.6.6

GitLab Workhorse 0.8.5

GitLab API v3

Git 2.7.4

Ruby 2.3.1p112

Rails 4.2.7.1

PostgreSQL 9.2.18

Tuesday, 1 November 2016

Dirty CoW kernel check - CentOS

So there's been a load of work due to the Dirty CoW vulnerability... you need to find out if there's been a kernel update so that you can reboot your CentOS P/VM... Here's a oneliner:

if [ "`rpm -q kernel --queryformat '%{installtime} %{version}-%{release}.%{arch}\n' | \
sort -n -k1 | tail -1 | cut -d ' ' -f 2`" = "`uname -r`" ]; \
then echo "You are running the latest kernel" && uname -r; \
else echo "There is a new kernel. You need a reboot" && echo "Current kernel: " && uname -r &&  \
echo "The latest kernel: " && rpm -q kernel --queryformat '%{version}-%{release}.%{arch}\n' | sort -n -k1 | tail -1;  fi

This is what it looks like on CentOS 6:

# if [ "`rpm -q kernel --queryformat '%{installtime} %{version}-%{release}.%{arch}\n' |sort -n -k1 | tail -1 | cut -d ' ' -f 2`" = "`uname -r`" ]; then echo "You are running the latest kernel" && uname -r; else echo "There is a new kernel. You need a reboot" && echo "Current kernel: " && uname -r && echo "The latest kernel: " && rpm -q kernel --queryformat '%{version}-%{release}.%{arch}\n' | sort -n -k1 | tail -1;  fi
There is a new kernel. You need a reboot
Current kernel:
2.6.32-642.4.2.el6.x86_64
The latest kernel:
2.6.32-642.6.2.el6.x86_64

The table below shows which kernel you should be running to fix the Dirty CoW vulnerability:

Distro	Kernel version
CentOS 5	2.6.32-642.3.1.el6.x86_64
CentOS 6	2.6.32-642.6.2.el6.x86_64
CentOS 7	3.10.0-327.36.3.el7.x86_64
Debian 7	3.2.82-1
Debian 8	3.16.36-1+deb8u2

Friday, 28 October 2016

Postfix MTA service not working! CentOS 6

I was having problems with keeping the Postfix MTA configured and running with Puppet. Each time Puppet ran it detected that it wasn't running and attempted to start it with no avail. The error when looking at the service was this:

# service postfix status
master dead but pid file exists

But removing the pid file didn't not help:

# locate postfix|grep pid
/var/spool/postfix/pid
/var/spool/postfix/pid/master.pid
[root@webtest ~]# rm /var/spool/postfix/pid/master.pid
rm: remove regular file `/var/spool/postfix/pid/master.pid'? y
[root@webtest ~]# service postfix status
master dead but subsys locked

So looking at the logs this was seen:

# tail  /var/log/maillog
Oct 30 19:44:06 webtest postfix/master[8005]: fatal: bind 127.0.0.1 port 25: Address already in use
Oct 30 20:09:49 webtest postfix/postfix-script[10053]: starting the Postfix mail system
Oct 30 20:09:49 webtest postfix/master[10054]: fatal: bind 127.0.0.1 port 25: Address already in use
Oct 30 20:10:04 webtest postfix/postfix-script[10602]: starting the Postfix mail system
Oct 30 20:10:04 webtest postfix/master[10603]: fatal: bind 127.0.0.1 port 25: Address already in use
Oct 30 20:10:53 webtest postfix/postfix-script[11037]: starting the Postfix mail system

The problem looks like another MTA was running hogging port 25. A quick ps for sendmail revealed nothing, but there's another agent that comes with CentOS 6:

[root@webtest ~]# ps -ef|grep send
root     12448  9780  0 20:16 pts/0    00:00:00 grep send
[root@webtest ~]# ps -ef|grep exim
root     12109  9780  0 20:22 pts/0    00:00:00 grep exim
exim     57456     1  0 Jul07 ?        00:00:00 /usr/sbin/exim -bd -q1h

[root@webtest ~]# service exim stop
Shutting down exim:                                        [  OK  ]
[root@webtest ~]# chkconfig exim off

Now a Puppet run should install and run Postfix without a problem:

# puppet agent -t
Notice: Local environment: 'production' doesn't match server specified node environment 'websites', switching agent to 'websites'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for website.domain.com
Info: Applying configuration version '1477858442'
Notice: /Stage[main]/postfixmta/Service[postfix]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/postfixmta/Service[postfix]: Unscheduling refresh on Service[postfix]
Notice: Applied catalog in 1.67 seconds
# puppet agent -t
Notice: Local environment: 'production' doesn't match server specified node environment 'websites', switching agent to 'websites'.
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for website.domain.com
Info: Applying configuration version '1477858442'
Notice: Applied catalog in 1.46 seconds

Monday, 3 October 2016

Fix Analytics not displaying in OpsCenter for LDOM - Solaris 11

root@solaris-ldom:~# svcs scn-agent
STATE          STIME    FMRI
maintenance    Sep_23   svc:/application/management/common-agent-container-1:scn-agent
root@solaris-ldom:~# svcs -xv
svc:/application/management/common-agent-container-1:scn-agent (Cacao, a common Java container for JDMK/JMX based management solution)
State: maintenance since Fri Sep 23 22:36:14 2016
Reason: Restarting too quickly.
  See: http://support.oracle.com/msg/SMF-8000-L5
  See: man -M /usr/share/man -s 1M cacaoadm
  See: man -M /usr/share/man -s 5 cacao
  See: /var/svc/log/application-management-common-agent-container-1:scn-agent.log
Impact: This service is not running.
root@solaris-ldom:~# cat /var/svc/log/application-management-common-agent-container-1:scn-agent.log
[ Mar 24 09:57:57 Disabled. ]
[ Mar 24 09:57:57 Rereading configuration. ]
[ Mar 24 09:58:01 Enabled. ]

-cut-

[ Sep 23 22:36:12 Stopping because all processes in service exited. ]
[ Sep 23 22:36:13 Executing stop method ("/usr/lib/cacao/lib/tools/scripts/cacao_smf stop scn-agent"). ]
[ Sep 23 22:36:14 Method "stop" exited with status 0. ]
[ Sep 23 22:36:14 Restarting too quickly, changing state to maintenance. ]
root@solaris-ldom:~# svcadm disable svc:/application/management/common-agent-container-1:scn-agent
root@solaris-ldom:~# svcs scn-agent
STATE          STIME    FMRI
disabled       11:55:48 svc:/application/management/common-agent-container-1:scn-agent
root@solaris-ldom:~# svcs -xv
root@solaris-ldom:~# svcadm enable svc:/application/management/common-agent-container-1:scn-agent
root@solaris-ldom:~# svcs -xv
svc:/application/management/common-agent-container-1:scn-agent (Cacao, a common Java container for JDMK/JMX based management solution)
State: offline* transitioning to online since Mon Sep 26 11:56:09 2016
Reason: Start method is running.
  See: http://support.oracle.com/msg/SMF-8000-C4
  See: man -M /usr/share/man -s 1M cacaoadm
  See: man -M /usr/share/man -s 5 cacao
  See: /var/svc/log/application-management-common-agent-container-1:scn-agent.log
Impact: This service is not running.
root@solaris-ldom:~# tail /var/svc/log/application-management-common-agent-container-1:scn-agent.log
[ Sep 23 22:31:50 Executing start method ("/usr/lib/cacao/lib/tools/scripts/cacao_smf start scn-agent"). ]
[ Sep 23 22:33:13 Method "start" exited with status 0. ]
[ Sep 23 22:36:12 Stopping because all processes in service exited. ]
[ Sep 23 22:36:13 Executing stop method ("/usr/lib/cacao/lib/tools/scripts/cacao_smf stop scn-agent"). ]
[ Sep 23 22:36:14 Method "stop" exited with status 0. ]
[ Sep 23 22:36:14 Restarting too quickly, changing state to maintenance. ]
[ Sep 26 11:55:48 Leaving maintenance because disable requested. ]
[ Sep 26 11:55:48 Disabled. ]
[ Sep 26 11:56:09 Enabled. ]
[ Sep 26 11:56:09 Executing start method ("/usr/lib/cacao/lib/tools/scripts/cacao_smf start scn-agent"). ]
root@solaris-ldom:/var/adm# svcs scn-agent
STATE          STIME    FMRI
online         11:57:12 svc:/application/management/common-agent-container-1:scn-agent
root@solaris-ldom:/var/adm#

Monday, 5 September 2016

SELinux and sending mail via HTTPD - CentOS 7

Use this SELinux command to give the Apache process to use sendmail:

#sudo setsebool -p httpd_can_sendmail 1

Thursday, 1 September 2016

Monitor DNS lookup

# tcpdump -i eth0 port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:48:15.324300 IP client.63533 > dns-server.domain: 17904+ PTR? 1.0.0.127.in-addr.arpa. (40)
10:48:15.324848 IP dns-server.domain > client.63533: 17904* 1/1/2 PTR localhost. (121)
10:48:15.325137 IP client.50547 > dns-server.domain: 49520+ AAAA? localhost. (27)
10:48:15.325293 IP client.43181 > dns-server.domain: 25134+ PTR? xx.x.xxx.xxx.in-addr.arpa. (43)
10:48:15.325643 IP dns-server.domain > client.50547: 49520* 1/1/1 AAAA ::1 (85)
10:48:15.325903 IP dns-server.domain > client.43181: 25134* 1/3/6 PTR dns-server. (268)
10:48:19.565837 IP client.25663 > dns-server.domain: 43756+ AAAA? client. (42)
10:48:19.566389 IP dns-server.domain > client.25663: 43756 NXDomain* 0/1/0 (93)
10:48:19.566497 IP client.64053 > dns-server.domain: 55198+ AAAA? client. (40)
10:48:19.567026 IP dns-server.domain > client.64053: 55198* 0/1/0 (91)
10:48:19.567086 IP client.49399 > dns-server.domain: 1076+ AAAA? client. (37)
10:48:19.567600 IP dns-server.domain > client.49399: 1076 NXDomain* 0/1/0 (88)
10:48:19.567656 IP client.24922 > dns-server.domain: 50409+ AAAA? client. (26)
10:48:19.568080 IP dns-server.domain > client.24922: 50409 NXDomain 0/1/0 (101)
10:53:15.248429 IP client.17122 > dns-server.domain: 45962+ PTR? 1.0.0.127.in-addr.arpa. (40)
10:53:15.248968 IP dns-server.domain > client.17122: 45962* 1/1/2 PTR localhost. (121)
10:53:15.249332 IP client.17597 > dns-server.domain: 59594+ AAAA? localhost. (27)
10:53:15.249819 IP dns-server.domain > client.17597: 59594* 1/1/1 AAAA ::1 (85)
10:53:19.590980 IP client.52610 > dns-server.domain: 39707+ AAAA? client. (42)
10:53:19.591525 IP dns-server.domain > client.52610: 39707 NXDomain* 0/1/0 (93)
10:53:19.591683 IP client.32529 > dns-server.domain: 23733+ AAAA? client. (40)
10:53:19.592180 IP dns-server.domain > client.32529: 23733* 0/1/0 (91)

Friday, 12 August 2016

Perl code for sending messages via Gmail - authentication error

Here's a comprehensive blog post to do this! http://robertmaldon.blogspot.co.uk/2006/10/sending-email-through-google-smtp-from.html Here's the chunk I used (after using CPAN to install Net::SMTP::SSL*) - gmail3.pl:

#!/usr/bin/perl -w

use Net::SMTP::SSL;

sub send_mail {
my $to = $_[0];
my $subject = $_[1];
my $body = $_[2];

my $from = 'johnny@gmail.com';
my $password = 'MySecretGmailPassword';

my $smtp;

if (not $smtp = Net::SMTP::SSL->new('smtp.gmail.com',
                            Port => 465,
                            Debug => 1)) {
   die "Could not connect to server\n";
}

$smtp->auth($from, $password)
   || die "Authentication failed!\n";

$smtp->mail($from . "\n");
my @recepients = split(/,/, $to);
foreach my $recp (@recepients) {
    $smtp->to($recp . "\n");
}
$smtp->data();
$smtp->datasend("From: " . $from . "\n");
$smtp->datasend("To: " . $to . "\n");
$smtp->datasend("Subject: " . $subject . "\n");
$smtp->datasend("\n");
$smtp->datasend($body . "\n");
$smtp->dataend();
$smtp->quit;
}

# Send away!
&send_mail('johnny@mywork.com', 'Server just blew up', 'Some more detail');

On execution, you might get this authentication error:

*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 together with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at ./gmail3.pl line 15.
Net::SMTP::SSL>>> Net::SMTP::SSL(1.03)
Net::SMTP::SSL>>>   IO::Socket::SSL(1.94)
Net::SMTP::SSL>>>     IO::Socket::IP(0.21)
Net::SMTP::SSL>>>       IO::Socket(1.34)
Net::SMTP::SSL>>>         IO::Handle(1.33)
Net::SMTP::SSL>>>           Exporter(5.68)
Net::SMTP::SSL>>>   Net::Cmd(3.10)
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 220 smtp.gmail.com ESMTP gg10sm6569316wjd.4 - gsmtp
Net::SMTP::SSL=GLOB(0x17f7a78)>>> EHLO localhost.localdomain
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-smtp.gmail.com at your service, [2001:630:e4:8102:21d:d8ff:feb7:1dee]
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-SIZE 35882577
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-8BITMIME
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-ENHANCEDSTATUSCODES
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-PIPELINING
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250-CHUNKING
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 250 SMTPUTF8
Net::SMTP::SSL=GLOB(0x17f7a78)>>> AUTH LOGIN
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 334 VXNlcm5hbWU6
Net::SMTP::SSL=GLOB(0x17f7a78)<<< (decoded) Username:
Net::SMTP::SSL=GLOB(0x17f7a78)>>> (decoded) johnny@gmail.com
Net::SMTP::SSL=GLOB(0x17f7a78)>>> Y2xlYXJkZWZAZ21haWwuY29t
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 334 UGFzc3dvcmQ6
Net::SMTP::SSL=GLOB(0x17f7a78)<<< (decoded) Password:
Net::SMTP::SSL=GLOB(0x17f7a78)>>> (decoded) MySecretGmailPassword
Net::SMTP::SSL=GLOB(0x17f7a78)>>> YXBwbGVncmVlbg==
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 534-5.7.14  Please log in via your web browser and
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 534-5.7.14 then try again.
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 534-5.7.14  Learn more at
Net::SMTP::SSL=GLOB(0x17f7a78)<<< 534 5.7.14  https://support.google.com/mail/answer/78754 gg10sm6569316wjd.4 - gsmtp
Authentication failed!

This is simply solved by login on to the Gmail account (via a browser) and changing the password. Remember to insert the new password into the script and away we go:

*******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 together with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at ./gmail3.pl line 15.
Net::SMTP::SSL>>> Net::SMTP::SSL(1.03)
Net::SMTP::SSL>>>   IO::Socket::SSL(1.94)
Net::SMTP::SSL>>>     IO::Socket::IP(0.21)
Net::SMTP::SSL>>>       IO::Socket(1.34)
Net::SMTP::SSL>>>         IO::Handle(1.33)
Net::SMTP::SSL>>>           Exporter(5.68)
Net::SMTP::SSL>>>   Net::Cmd(3.10)
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 220 smtp.gmail.com ESMTP uo4sm6542414wjc.36 - gsmtp
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> EHLO localhost.localdomain
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-smtp.gmail.com at your service, [2001:630:e4:8102:21d:d8ff:feb7:1dee]
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-SIZE 35882577
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-8BITMIME
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-ENHANCEDSTATUSCODES
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-PIPELINING
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250-CHUNKING
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250 SMTPUTF8
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> AUTH LOGIN
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 334 VXNlcm5hbWU6
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< (decoded) Username:
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> (decoded) johnny@gmail.com
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> Y2xlYXJkZWZAZ21haWwuY29t
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 334 UGFzc3dvcmQ6
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< (decoded) Password:
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> (decoded) myNewSecretPassword
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> QXBwbGVHcjMzbg==
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 235 2.7.0 Accepted
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> MAIL FROM:
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250 2.1.0 OK uo4sm6542414wjc.36 - gsmtp
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> RCPT TO:
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250 2.1.5 OK uo4sm6542414wjc.36 - gsmtp
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> DATA
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 354  Go ahead uo4sm6542414wjc.36 - gsmtp
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> From: johnny@gmail.com
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> To: johnny@mywork.com
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> Subject: Server just blew up
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> Some more detail
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> .
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 250 2.0.0 OK 1470992894 uo4sm6542414wjc.36 - gsmtp
Net::SMTP::SSL=GLOB(0x2ac9a78)>>> QUIT
Net::SMTP::SSL=GLOB(0x2ac9a78)<<< 221 2.0.0 closing connection uo4sm6542414wjc.36 - gsmtp

Installing Perl Modules with CPAN:

# perl -MCPAN -e 'shell'
Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.9800)
Enter 'h' for help.

cpan[1]> install Net::SMTP::SSL
Reading '/root/.cpan/Metadata'
  Database was generated on Thu, 11 Aug 2016 00:53:43 GMT
Running install for module 'Net::SMTP::SSL'
Running make for R/RJ/RJBS/Net-SMTP-SSL-1.03.tar.gz
Fetching with HTTP::Tiny:
http://cpan.netbet.org/authors/id/R/RJ/RJBS/Net-SMTP-SSL-1.03.tar.gz
Checksum for /root/.cpan/sources/authors/id/R/RJ/RJBS/Net-SMTP-SSL-1.03.tar.gz ok
Scanning cache /root/.cpan/build for sizes
............................................................................DONE

  CPAN.pm: Building R/RJ/RJBS/Net-SMTP-SSL-1.03.tar.gz

Checking if your kit is complete...
Looks good
Writing Makefile for Net::SMTP::SSL
Writing MYMETA.yml and MYMETA.json
cp lib/Net/SMTP/SSL.pm blib/lib/Net/SMTP/SSL.pm
Manifying blib/man3/Net::SMTP::SSL.3pm
  RJBS/Net-SMTP-SSL-1.03.tar.gz
  /bin/make -- OK
'YAML' not installed, will not store persistent state
Running make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/test.t .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.07 cusr  0.00 csys =  0.08 CPU)
Result: PASS
  RJBS/Net-SMTP-SSL-1.03.tar.gz
  /bin/make test -- OK
Running make install
Manifying blib/man3/Net::SMTP::SSL.3pm
Installing /usr/local/share/perl5/Net/SMTP/SSL.pm
Installing /usr/local/share/man/man3/Net::SMTP::SSL.3pm
Appending installation info to /usr/lib64/perl5/perllocal.pod
  RJBS/Net-SMTP-SSL-1.03.tar.gz
  /bin/make install  -- OK

cpan[2]> Terminal does not support GetHistory.
Lockfile removed.