A recent GitLab CE update left the service down. A restart of the service resulted in this error when trying to login:
Could not authenticate you from Ldapmain because "Pg::undefinedtable: error: relation "u2f registrations" does not exist line 5: where a.attrelid = '"u2f registrations"'::reg... ^ : select a.attname, format type(a.atttypid, a.atttypmod), pg get expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod from pg attribute a left join pg attrdef d on a.attrelid = d.adrelid and a.attnum = d.adnum where a.attrelid = '"u2f registrations"'::regclass and a.attnum > 0 and not a.attisdropped order by a.attnum ".
A bit strange and worrying that maybe the database was corrupted and/or missing. But after an initial google for "gitlab u2f registrations" revealed that is was 2 factor authentication - which we are not using here. So first thing was to find out how to disable this system-wide - apparently this was the command: gitlab-rake gitlab:two_factor:disable_for_all_users, but running it resulted in the following:
[root@gitlab gitlab]# gitlab-rake gitlab:two_factor:disable_for_all_users
rake aborted!
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "u2f_registrations" does not exist
LINE 1: ...STINCT "users"."id") FROM "users" LEFT OUTER JOIN u2f_regist...
^
: SELECT DISTINCT COUNT(DISTINCT "users"."id") FROM "users" LEFT OUTER JOIN u2f_registrations AS u2f ON u2f.user_id = users.id WHERE (u2f.id IS NOT NULL OR otp_required_for_login = 't')
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `async_exec'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract_adapter.rb:472:in `block in log'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract_adapter.rb:466:in `log'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:264:in `execute_simple_calculation'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:221:in `perform_calculation'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:127:in `calculate'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:42:in `count'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/two_factor.rake:6:in `block (3 levels) in '
PG::UndefinedTable: ERROR: relation "u2f_registrations" does not exist
LINE 1: ...STINCT "users"."id") FROM "users" LEFT OUTER JOIN u2f_regist...
^
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `async_exec'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract_adapter.rb:472:in `block in log'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.2.6/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract_adapter.rb:466:in `log'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:264:in `execute_simple_calculation'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:221:in `perform_calculation'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:127:in `calculate'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/relation/calculations.rb:42:in `count'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/two_factor.rake:6:in `block (3 levels) in '
Tasks: TOP => gitlab:two_factor:disable_for_all_users
(See full trace by running task with --trace)
So obviously that was not the problem. So let's take a snapshot of this gitlab VM and run a reconfigure:
[root@gitlab gitlab]# gitlab-ctl reconfigure
Starting Chef Client, version 12.6.0
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
- package (0.0.0)
- gitlab (0.0.1)
- runit (0.14.2)
Compiling Cookbooks...
Recipe: gitlab::default
* directory[/etc/gitlab] action create (up to date)
/sbin/init: unrecognized option '--version'
-.mount loaded active mounted /
Converging 282 resources
* directory[/etc/gitlab] action nothing (skipped due to action :nothing)
* directory[/var/opt/gitlab] action create (up to date)
* directory[/opt/gitlab/embedded/etc] action create (up to date)
* template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date)
Recipe: gitlab::web-server
* group[gitlab-www] action create (up to date)
* user[gitlab-www] action create (up to date)
Recipe: gitlab::users
* directory[/var/opt/gitlab] action create (up to date)
* group[git] action create (up to date)
* user[git] action create (up to date)
* template[/var/opt/gitlab/.gitconfig] action create (up to date)
Recipe: gitlab::gitlab-shell
* directory[/var/opt/gitlab/git-data/repositories] action create (up to date)
* directory[/var/opt/gitlab/git-data] action create (up to date)
* directory[/var/opt/gitlab/.ssh] action create (up to date)
* file[/var/opt/gitlab/.ssh/authorized_keys] action create (up to date)
* directory[/opt/gitlab/embedded/service/gitlab-shell/hooks/] action create (up to date)
* file[/opt/gitlab/embedded/service/gitlab-shell/authorized_keys.lock] action create (up to date)
* execute[chcon --recursive --type ssh_home_t /var/opt/gitlab/.ssh] action runid: --context (-Z) works only on an SELinux-enabled kernel
(skipped due to only_if)
* directory[/var/log/gitlab/gitlab-shell/] action create (up to date)
* directory[/var/opt/gitlab/gitlab-shell] action create (up to date)
* template[/var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-shell/config.yml] action create (up to date)
* template[/var/opt/gitlab/gitlab-shell/gitlab_shell_secret] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret] action create (up to date)
Recipe: gitlab::gitlab-rails
* directory[/var/log/gitlab] action create (up to date)
* directory[/var/opt/gitlab/gitlab-rails/shared] action create (up to date)
* directory[/var/opt/gitlab/gitlab-rails/shared/artifacts] action create (up to date)
* directory[/var/opt/gitlab/gitlab-rails/shared/lfs-objects] action create (up to date)
* directory[/var/opt/gitlab/gitlab-rails/uploads] action create (up to date)
* directory[/var/opt/gitlab/gitlab-ci/builds] action create (up to date)
* directory[/var/opt/gitlab/gitlab-rails/shared/pages] action create (up to date)
* directory[/var/opt/gitlab/gitlab-rails/etc] action create (up to date)
* directory[/opt/gitlab/etc/gitlab-rails] action create (up to date)
* directory[/var/opt/gitlab/gitlab-rails/working] action create (up to date)
* directory[/var/opt/gitlab/gitlab-rails/tmp] action create (up to date)
* directory[/var/opt/gitlab/gitlab-rails/upgrade-status] action create (up to date)
* directory[/var/log/gitlab/gitlab-rails] action create (up to date)
* directory[/var/opt/gitlab/backups] action create (up to date)
* directory[/var/opt/gitlab/gitlab-rails] action create (up to date)
* directory[/var/opt/gitlab/gitlab-ci] action create (up to date)
* template[/opt/gitlab/etc/gitlab-rails/gitlab-rails-rc] action create (up to date)
* template[/var/opt/gitlab/gitlab-rails/etc/secret] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/.secret] action create (up to date)
* template[/var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/config/database.yml] action create (up to date)
* template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml] action create (up to date)
* template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/config/resque.yml] action create (up to date)
* template[/var/opt/gitlab/gitlab-rails/etc/aws.yml] action delete (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/config/aws.yml] action delete (up to date)
* template[/var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action delete (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/config/initializers/smtp_settings.rb] action delete (up to date)
* template[/var/opt/gitlab/gitlab-rails/etc/relative_url.rb] action delete (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/config/initializers/relative_url.rb] action delete (up to date)
* template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml] action create (up to date)
* template[/var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/config/initializers/rack_attack.rb] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret] action create (up to date)
* directory[/opt/gitlab/etc/gitlab-rails/env] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/HOME] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/PATH] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/tmp] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/public/uploads] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/log] action create (up to date)
* link[/var/log/gitlab/gitlab-rails/sidekiq.log] action create (skipped due to not_if)
* file[/opt/gitlab/embedded/service/gitlab-rails/db/schema.rb] action create (up to date)
* remote_file[/var/opt/gitlab/gitlab-rails/VERSION] action create (up to date)
* remote_file[/var/opt/gitlab/gitlab-rails/REVISION] action create (up to date)
* file[/var/opt/gitlab/gitlab-rails/RUBY_VERSION] action create (up to date)
* execute[chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public] action run
- execute chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public
* execute[clear the gitlab-rails cache] action nothing (skipped due to action :nothing)
* bash[generate assets] action nothing (skipped due to action :nothing)
* file[/var/opt/gitlab/gitlab-rails/config.ru] action delete (up to date)
Recipe: gitlab::gitlab-ci-proxying
* template[/var/opt/gitlab/nginx/conf/gitlab-ci-http.conf] action delete (up to date)
Recipe: gitlab::selinux
* execute[semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-7.2.0-ssh-keygen.pp] action runDisabled
(skipped due to not_if)
Recipe: gitlab::add_trusted_certs
* directory[/etc/gitlab/ssl/trusted-certs] action create (up to date)
* directory[/opt/gitlab/embedded/ssl/certs] action create (up to date)
* file[/opt/gitlab/embedded/ssl/certs/README] action create (up to date)
* ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] action run
* Moving existing certificates found in /opt/gitlab/embedded/ssl/certs
ERROR: /opt/gitlab/embedded/ssl/certs/pecacert.pem: OpenSSL error: nested asn1 error!
================================================================================
Error executing action `run` on resource 'ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs]'
================================================================================
RuntimeError
------------
ERROR: Not a certificate: /opt/gitlab/embedded/ssl/certs/pecacert.pem / /opt/gitlab/embedded/ssl/certs/pecacert.pem
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/helper.rb:514:in `raise_msg'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/helper.rb:460:in `block in move_existing_certificates'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/helper.rb:453:in `glob'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/helper.rb:453:in `move_existing_certificates'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/add_trusted_certs.rb:31:in `block (2 levels) in from_file'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/add_trusted_certs.rb
28: ruby_block "Move existing certs and link to #{ssl_certs_dir}" do
29: block do
30: puts "\n\n * Moving existing certificates found in #{ssl_certs_dir}\n"
31: cert_helper.move_existing_certificates
32: puts "\n * Symlinking existing certificates found in #{trusted_certs_dir}\n"
33: cert_helper.link_certificates
34: end
35: only_if { cert_helper.new_certificate_added? }
36: notifies :restart, "service[unicorn]"
37: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/add_trusted_certs.rb:28:in `from_file'
ruby_block("Move existing certs and link to /opt/gitlab/embedded/ssl/certs") do
action [:run]
retries 0
retry_delay 2
default_guard_interpreter :default
block_name "Move existing certs and link to /opt/gitlab/embedded/ssl/certs"
declared_type :ruby_block
cookbook_name "gitlab"
recipe_name "add_trusted_certs"
block #
only_if { #code block }
end
Running handlers:
Running handlers complete
Chef Client failed. 1 resources updated in 16 seconds
So the problem is that re-configure (an upgrade performs a reconfigure) failed leaving the gitlab service down... So the problem was moving the certificates. GitLab was now fixed by copying the certificates by hand into the required location and (re)moving the source directory:
[root@gitlab ~]# cd /opt/gitlab/embedded/ssl/certs/
[root@gitlab certs]# ls -la
total 256
drwxr-xr-x. 2 root root 55 Jun 24 10:03 .
drwxr-xr-x. 5 root root 76 Jun 23 03:45 ..
-rw-r--r-- 1 root root 252499 Jun 22 11:05 cacert.pem
-rw-r--r-- 1 root root 800 Apr 4 19:47 pecacert.pem
-rw-r--r-- 1 root root 151 Jun 23 03:46 README
[root@gitlab certs]# cat README
This directory is managed by omnibus-gitlab.
Any file placed in this directory will be ignored
. Place certificates in /etc/gitlab/ssl/trusted-certs.
[root@gitlab certs]# cp /opt/gitlab/embedded/ssl/certs/* /etc/gitlab/ssl/trusted-certs
[root@gitlab ssl]# mv certs certs.son
[root@gitlab ssl]# gitlab-ctl reconfigure
Starting Chef Client, version 12.6.0
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
- runit (0.14.2)
- package (0.0.0)
- gitlab (0.0.1)
Compiling Cookbooks...
Recipe: gitlab::default
* directory[/etc/gitlab] action create (up to date)
/sbin/init: unrecognized option '--version'
[successful reconfigure detail snipped]
Recipe: gitlab::add_trusted_certs
* directory[/etc/gitlab/ssl/trusted-certs] action create (up to date)
* directory[/opt/gitlab/embedded/ssl/certs] action create
- create new directory /opt/gitlab/embedded/ssl/certs
- change mode from '' to '0755'
* file[/opt/gitlab/embedded/ssl/certs/README] action create
- create new file /opt/gitlab/embedded/ssl/certs/README
- update content in file /opt/gitlab/embedded/ssl/certs/README from none to e09a2d
--- /opt/gitlab/embedded/ssl/certs/README 2016-06-24 10:14:17.275761125 +0100
+++ /opt/gitlab/embedded/ssl/certs/.README20160624-29376-1q3bw7j 2016-06-24 10:14:17.275761125 +0100
@@ -1 +1,4 @@
+This directory is managed by omnibus-gitlab.
+ Any file placed in this directory will be ignored
+. Place certificates in /etc/gitlab/ssl/trusted-certs.
- change mode from '' to '0644'
* ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] action run
* Moving existing certificates found in /opt/gitlab/embedded/ssl/certs
* Symlinking existing certificates found in /etc/gitlab/ssl/trusted-certs
Skipping /etc/gitlab/ssl/trusted-certs/cacert.pem.
Skipping /etc/gitlab/ssl/trusted-certs/pecacert.pem.
Skipping /etc/gitlab/ssl/trusted-certs/README.
[successful reconfigure detail snipped]
- execute "bash" "/tmp/chef-script20160624-29376-v5yl6q"
Recipe: gitlab::gitlab-rails
* execute[clear the gitlab-rails cache] action run
- execute /opt/gitlab/bin/gitlab-rake cache:clear
Recipe: gitlab::unicorn
* service[unicorn] action restart
- restart service service[unicorn]
Recipe: gitlab::sidekiq
* service[sidekiq] action restart
- restart service service[sidekiq]
Recipe: gitlab::unicorn
* service[unicorn] action restart
- restart service service[unicorn]
Recipe: gitlab::gitlab-workhorse
* service[gitlab-workhorse] action restart
- restart service service[gitlab-workhorse]
Recipe: gitlab::nginx
* service[nginx] action restart
- restart service service[nginx]
Running handlers:
Running handlers complete
Chef Client finished, 22/294 resources updated in 43 seconds
gitlab Reconfigured!
Service is up and running and login is just fine!
