Gitlab 跨版本升级记录

需求

由于公司需要使用Rancher的pipeline，但是在绑定公司Gitlab的时候报错，后查看是由于Gitlab 的版本太低导致，所以需要升级Gitlab，并且原Gitlab服务器配置较低，正好升级下配置。
目前Gitlab中rom组实在是太大了，Git备份总共198G，Rom组占了170G，所以打算将rom组剥离出来，给rom组一个单独的gitlab，其它组共用一个

环境说明

原Gitlab : 8.10.2 –> 新Gitlab : 11.x

操作系统：Centos7

主机类型：虚拟机

CPU: 8核

内存：16G

由于升级版本差距较大，直接在原服务器升级风险较大，并且还需要给rom组单独一套gitlab，所以新准备一台服务器重新部署升级，给rom组以外的研发人员使用，原gitlab不作升级给rom组使用。

注意事项

gitlab不能跨多个版本升级，并且不同版本的gitlab导入gitlab-gake的备份数据会报以下错误，所以只能安装与旧服务器同版本的gitlab之后在按顺序升级

# gitlab-rake gitlab:backup:restore
Unpacking backup ... done
GitLab version mismatch:
    Your current GitLab version (11.6.5) differs from the GitLab version in the backup!
    Please switch to the following version and try again:
    version: 8.10.2

Hint: git checkout v8.10.2

gitlab的gitlab-rake命令不会备份任何的配置文件，需要用户自行配置到少要备份以下文件
- /etc/gitlab/gitlab.rb
- /etc/gitlab/gitlab-secrets.json
  正式安装升级

1. 安装gitlab:8.10.2

Gitlab rpm包下载地址：https://packages.gitlab.com/gitlab/gitlab-ce?page=116

国内推荐用这个：https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/

要用到的包提前下载好备用，还有下载完的包一定要检查MD5，血和泪的教训

我的升级顺序为： 8.10.2 --> 8.17.8 --> 9.5.5 --> 10.8.5 --> 11.7.0
我用到的包：
gitlab-ce-8.17.8-ce.0.el7.x86_64.rpm
gitlab-ce-9.5.5-ce.0.el7.x86_64.rpm
gitlab-ce-10.8.5-ce.0.el7.x86_64.rpm
gitlab-ce-11.7.0-ce.0.el7.x86_64.rpm

安装

1	rpm -ivh gitlab-ce-8.10.2-ce.0.el7.x86_64.rpm

参数解释：

-i, --install                    install package(s) 安装包
-U, --upgrade=<packagefile>+     upgrade package(s) 更新包，后面会用到
-h, --hash                       print hash marks as package installs (good with -v) 

### 2. 备份原数据

最好停掉gitlab，防止有人提交代码
    
```shell
# gitlab-ctl stop unicorn
# gitlab-ctl stop nginx

备份数据

1	# gitlab-rake gitlab:backup:create

3. 还原数据

3.1 将gitlab数据备份后，拷贝到新主机的/var/opt/gitlab/backups 目录，此目录为gitlab默认的备份目录

198G备份文件，漫长的等待…

BACKUP=timestamp_of_backup 如果备份目录有多个备份文件可指定时间戳
force=yes 不询问authorized_keys文件是否应该重新生成，并设定“yes”表示将删除数据库表，启用“写入authorized_keys文件”设置，并更新LDAP提供程序。

# ls /var/opt/gitlab/backups/
1547761060_gitlab_backup.tar
# time gitlab-rake gitlab:backup:restore BACKUP=1547761060 force=yes

...中间输出内容略过，其中time命令是为了看执行时长的。...

real	106m46.133s
user	1m14.546s
sys	25m5.829s

将原服务器的配置文件拷贝到新主机，并更新配置

需要注意拷贝过来的配置文件中的data目录是否与现在的目录的一致

1
2
3

# scp 10.60.0.120:/etc/gitlab/gitlab.rb /etc/gitlab/gitlab.rb
# scp 10.60.0.120:/etc/gitlab/gitlab-secrets.json  /etc/gitlab/gitlab-secrets.json
# gitlab-ctl reconfigure

4. 测试

访问git地址验证数据是否正常恢复，然后删除rom组的数据，需要注意虽然在页面删除了数据，但是gitlab可能出于安全考虑并没有将数据从机器上删除，需要手动删除。

1	# rm -rf /var/opt/gitlab/git-data/repositories/rom

重新备份一下数据

# gitlab-rake gitlab:backup:create
// 备份完后在查看一下备份文件的大小，美滋滋
# du -sh /home/backups/1548406242_gitlab_backup.tar
24G	/home/backups/1548406242_gitlab_backup.tar

4. 升级

我们现在使用的是8.10.2版本，从官网看8.x最高版本为8.17.8，版本差距也有点大，所以先拍了个虚拟机快照，然后直接升级。

# rpm -Uvh gitlab-ce-8.17.8-ce.0.el7.x86_64.rpm

...内容太多略过，只截取了最后一点内容...
Running handlers:
Running handlers complete
Chef Client finished, 58/326 resources updated in 02 minutes 03 seconds
gitlab Reconfigured!
Restarting previously running GitLab services
ok: run: gitlab-workhorse: (pid 28090) 2s
ok: run: logrotate: (pid 28116) 0s
ok: run: nginx: (pid 28102) 1s
ok: run: postgresql: (pid 27858) 101s
ok: run: redis: (pid 27844) 102s
ok: run: sidekiq: (pid 28030) 29s
ok: run: unicorn: (pid 28126) 0s

Upgrade complete! If your GitLab server is misbehaving try running

   sudo gitlab-ctl restart

before anything else. If you need to roll back to the previous version you can
use the database backup made during the upgrade (scroll up for the filename).

测试了一下没啥问题，看来不用怕了，直接走起

# rpm -Uvh gitlab-ce-9.5.5-ce.0.el7.x86_64.rpm
# rpm -Uvh gitlab-ce-10.8.5-ce.0.el7.x86_64.rpm
# rpm -Uvh gitlab-ce-11.7.0-ce.0.el7.x86_64.rpm   
warning: gitlab-ce-11.7.0-ce.0.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID f27eab47: NOKEY
Preparing...                          ################################# [100%]
* git_data_dir has been deprecated since 8.10 and removed in 11.0. Use git_data_dirs instead.
* mattermost['support_email'] has been deprecated since 10.2 and removed in 11.0.
Deprecations found. Please correct them and try again.
error: %pre(gitlab-ce-11.7.0-ce.0.el7.x86_64) scriptlet failed, exit status 1
error: gitlab-ce-11.7.0-ce.0.el7.x86_64: install failed
error: gitlab-ce-10.8.5-ce.0.el7.x86_64: erase skipped

万万没想到最后一步出错了，看意思是gitlab.rb配置文件中的‘git_data_dir’ 被替换成了‘git_data_dirs’
[‘support_email’] 已经在10.2版本中弃用，11.0版本已经移除了，提示的很明显，该删的删，该移除的移除，之后记得要重新更新配置的命令然后在重新升级

1 2	# gitlab-ctl reconfigure # rpm -Uvh gitlab-ce-11.7.0-ce.0.el7.x86_64.rpm

Ok 升级完成，测试下访问，推送，拉代码没问题就万事大吉啦。

FQA

下面的错误是因为postgresql目录权限不对，有两种解决方法，推荐方法2：

将目录修改成正确的权限，可查看原服务器对应目录的权限，改成一样的即可

可以使用gitlab-ctl reconfigure 恢复

Unpacking backup ...
done
Before restoring the database we recommend removing all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.

Do you want to continue (yes/no)? Do you want to continue (yes/no)? yes
Removing all tables. Press `Ctrl-C` within 5 seconds to abort
Cleaning the database ...
rake aborted!
PG::ConnectionBad: FATAL:  could not open relation mapping file "global/pg_filenode.map": Permission denied
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/postgresql_adapter.rb:651:in `connect'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/postgresql_adapter.rb:242:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/postgresql_adapter.rb:44:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/postgresql_adapter.rb:44:in `postgresql_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:438:in `new_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:448:in `checkout_new_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:422:in `acquire_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:in `block in checkout'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:in `checkout'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in `block in connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in `connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:571:in `retrieve_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.7/lib/active_record/connection_handling.rb:87:in `connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:27:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:50:in `block (3 levels) in <top (required)>'
Tasks: TOP => gitlab:db:drop_tables
(See full trace by running task with --trace)

这个错误是因为疏忽大意，下包没检查md5，下载的包是有问题的导致安装时出现问题。

# rpm -Uvh gitlab-ce-9.5.5-ce.0.el7.x86_64.rpm
warning: gitlab-ce-9.5.5-ce.0.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID f27eab47: NOKEY
Preparing...                          ################################# [100%]
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
done
Dumping repositories ...
[SKIPPED]
Dumping uploads ...
[SKIPPED]
Dumping builds ...
[SKIPPED]
Dumping artifacts ...
[SKIPPED]
Dumping pages ...
[SKIPPED]
Dumping lfs objects ...
[SKIPPED]
Dumping container registry images ...
[DISABLED]
Creating backup archive: 1548481271_2019_01_26_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
done
Deleting old backups ... skipping
Updating / installing...
   1:gitlab-ce-9.5.5-ce.0.el7         ################################# [ 50%]
error: unpacking of archive failed on file /opt/gitlab/embedded/lib/libicui18n.so.57.1;5c4bf2b4: cpio: read failed - No such file or directory
error: gitlab-ce-9.5.5-ce.0.el7.x86_64: install failed
error: gitlab-ce-8.17.8-ce.0.el7.x86_64: erase skipped

gitlab 11.7 默认开启CI/CD 需要手动关闭：

点击Admin area(一个小扳手) -> settings -> CI/CD ->Continuous Integration and Deployment 点击Exand ->将Default to Auto DevOps pipeline for all projects和Enable shared runners for new projects勾去掉，我在这的时候保存时报500的错误，错误日志如下：

Started PATCH "/admin/application_settings" for 127.0.0.1 at 2019-01-29 21:02:10 -0500
Processing by Admin::ApplicationSettingsController#update as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "application_setting"=>{"auto_devops_enabled"=>"0", "auto_devops_domain"=>"", "shared_runners_enabled"=>"1", "shared_runners_text"=>"", "max_artifacts_size"=>"100", "default_artifacts_expire_in"=>"0", "archive_builds_in_human_readable"=>"", "protected_ci_variables"=>"[FILTERED]"}}
Completed 500 Internal Server Error in 76ms (ActiveRecord: 25.1ms)

OpenSSL::Cipher::CipherError ():

lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:55:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:42:in `ensure_token'
app/models/concerns/token_authenticatable.rb:38:in `block in add_authentication_token_field'
app/services/application_settings/update_service.rb:18:in `execute'
app/controllers/admin/application_settings_controller.rb:40:in `update'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:417:in `set_locale'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Started GET "/-/metrics" for 127.0.0.1 at 2019-01-29 21:02:11 -0500
Processing by MetricsController#index as HTML
Completed 200 OK in 2ms (Views: 0.6ms | ActiveRecord: 0.0ms)

解决方案如下：

# sudo gitlab-rails c:
-------------------------------------------------------------------------------------
 GitLab:       11.7.0 (1d9280e)
 GitLab Shell: 8.4.4
 postgresql:   9.6.11
-------------------------------------------------------------------------------------
Loading production environment (Rails 5.0.7.1)
irb(main):001:0> settings = ApplicationSetting.last
irb(main):001:0> settings.update_column(:runners_registration_token_encrypted, nil)

以下问题没有遇到，但是转过来备用吧，忘了出处了。。。

如果升级后出现500的错误有可能是数据库升级异常导致的，可尝试使用以下方法解决

1 2	# gitlab-rake db:migrate:status // 查看数据库升级状态up 为正常，如果有down的说明有问题 # gitlab-rake db:migrate // 数据库关系升级

升级后出现以下错误：Error executing action run on resource ‘ruby_block[directory resource: /var/opt/gitlab/git-data/repositories]’
错误信息：

resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
- package (0.0.0)
- runit (0.14.2)
- gitlab (0.0.1)
Installing Cookbook Gems:
Compiling Cookbooks...
Recipe: gitlab::default
* directory[/etc/gitlab] action create (up to date)
/sbin/init: Unbekannte Option »--version«
-.mount                                                                                                 loaded active mounted   /
Converging 284 resources
* directory[/etc/gitlab] action create (up to date)
* directory[Create /var/opt/gitlab] action create (up to date)
* directory[/opt/gitlab/embedded/etc] action create (up to date)
* template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date)
Recipe: gitlab::web-server
* group[gitlab-www] action create (up to date)
* user[gitlab-www] action create (up to date)
Recipe: gitlab::users
* directory[/var/opt/gitlab] action create (up to date)
* group[git] action create (up to date)
* user[git] action create (up to date)
* template[/var/opt/gitlab/.gitconfig] action create (up to date)
Recipe: gitlab::gitlab-shell
* ruby_block[directory resource: /var/opt/gitlab/git-data] action run

  ================================================================================
  Error executing action `run` on resource 'ruby_block[directory resource: /var/opt/gitlab/git-data]'
  ================================================================================

Mixlib::ShellOut::ShellCommandFailed

解决方法：

1	chmod 2770 /var/opt/gitlab/git-data/repositories

导入后可能会出现以下报错，可尝试清除缓存

报错如下：

1	No repository The repository for this project does not exist. This means you can not push code until you create an empty repository or import existing one.

清除缓存的命令如下：

1	# gitlab-rake cache:clear

Gitlab 跨版本升级记录

Gitlab 跨版本升级记录

需求

环境说明

注意事项

正式安装升级

1. 安装gitlab:8.10.2

3. 还原数据

4. 测试

4. 升级

FQA

参考文档