I have recently been appointed the new gitlab admin for our local installation, but I have limited experience with gitlab. The previous admin unfortunately chose to move on to greener pastures.
Our installation is as follows: gitlab-ee-13.10.3-ee.0.el7.x86_64 running on CentOS Linux release 7.9.2009 (Core)
Unfortunately I have inherited a broken gitlab and i cannot seem to figure out how to resolve our issues.
our gitlab webpages only returns a 502 Whoops, GitLab is taking too much time to respond. error
gitlab-rails console
fails with the error message:
Traceback (most recent call last):
56: from bin/rails:4:in `<main>'
55: from bin/rails:4:in `require'
54: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/commands.rb:18:in `<top (required)>'
53: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/command.rb:46:in `invoke'
52: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/command/base.rb:69:in `perform'
51: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
50: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
49: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
48: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/commands/console/console_command.rb:101:in `perform'
47: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/command/actions.rb:15:in `require_application_and_environment!'
46: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/command/actions.rb:28:in `require_environment!'
45: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/application.rb:339:in `require_environment!'
44: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
43: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
42: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `block in require'
41: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:324:in `require'
40: from /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
39: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/application.rb:363:in `initialize!'
38: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/initializable.rb:60:in `run_initializers'
37: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:205:in `tsort_each'
36: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:226:in `tsort_each'
35: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `each_strongly_connected_component'
34: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `call'
33: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:347:in `each'
32: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:349:in `block in each_strongly_connected_component'
31: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:415:in `each_strongly_connected_component_from'
30: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:415:in `call'
29: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/initializable.rb:50:in `tsort_each_child'
28: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/initializable.rb:50:in `each'
27: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
26: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:431:in `each_strongly_connected_component_from'
25: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
24: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
23: from /opt/gitlab/embedded/lib/ruby/2.7.0/tsort.rb:228:in `block in tsort_each'
22: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/initializable.rb:61:in `block in run_initializers'
21: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/initializable.rb:32:in `run'
20: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/initializable.rb:32:in `instance_exec'
19: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/engine.rb:624:in `block in <class:Engine>'
18: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/engine.rb:624:in `each'
17: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/engine.rb:625:in `block (2 levels) in <class:Engine>'
16: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/engine.rb:665:in `load_config_initializer'
15: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/notifications.rb:182:in `instrument'
14: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/railties-6.0.3.6/lib/rails/engine.rb:666:in `block in load_config_initializer'
13: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:318:in `load'
12: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:291:in `load_dependency'
11: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:318:in `block in load'
10: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/dependencies.rb:318:in `load'
9: from /opt/gitlab/embedded/service/gitlab-rails/config/initializers/0_inject_feature_flags.rb:7:in `<top (required)>'
8: from /opt/gitlab/embedded/service/gitlab-rails/lib/feature.rb:137:in `register_definitions'
7: from /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:102:in `reload!'
6: from /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:145:in `load_all!'
5: from /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:145:in `each_with_object'
4: from /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:145:in `each'
3: from /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:146:in `block in load_all!'
2: from /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:161:in `load_all_from_path!'
1: from /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:161:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:165:in `block in load_all_from_path!': Feature flag 'usage_data_api' is already defined in '/opt/gitlab/embedded/service/gitlab-rails/config/feature_flags/development/usage_data_api.yml' (Feature::InvalidFeatureFlagError)
and gitlab-ctl reconfigure fails with:
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 70) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20210430-15489-165kk34" ----
STDOUT: rake aborted!
Feature::InvalidFeatureFlagError: Feature flag 'usage_data_api' is already defined in '/opt/gitlab/embedded/service/gitlab-rails/config/feature_flags/development/usage_data_api.yml'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:165:in `block in load_all_from_path!'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:161:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:161:in `load_all_from_path!'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:146:in `block in load_all!'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:145:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:145:in `each_with_object'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:145:in `load_all!'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:102:in `reload!'
/opt/gitlab/embedded/service/gitlab-rails/lib/feature.rb:137:in `register_definitions'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/0_inject_feature_flags.rb:7:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20210430-15489-165kk34" ----
Ran "bash" "/tmp/chef-script20210430-15489-165kk34" returned 1
Ive looked around on the net, but i am unable to find anyone else with this problem. Ive also tried to remove the offending usage_data_api.yml file, but then the commands fails with a new error message(unable to load --json file)
There are articles on the net describing how to enable disable features using the browser, but since our webpages are all returning 502 i cannot use that option.
That being said, i dont even know if the error message as describe above is the root cause for our webpages not working, or if that is something else.
Any and all help appriciated.
regards
kenneth