Gerrit replication hangs indefinitely

user_1778101 0
Viewed 687

I'm having trouble with Gerrit replication plugin. I'm trying to replicate repository to Gitlab over HTTPS. Most important configuration:

etc/replication.config

[gerrit]
        replicateOnStartup = true

[remote "gitlab-mirror"]
        url = https://<name.surname>:<password>@gitlab.domain/<Name.Surname>/${name}.git
        push = +refs/heads/*:refs/heads/*
        push = +refs/tags/*:refs/tags/*
        mirror = true
        projects = hello-world
        rescheduleDelay = 15

Repository on Gitlab side does exists under: https://gitlab.domain/<Name.Surname>/hello-world

I even cloned repository from Gerrit, add another remote to gitlab called mirror and pushed to it without hassle:

git clone ssh://admin@gerrit.domain:29418/hello-world
git remote add mirror https://<name.surname>:<password>@gitlab.domain/<Name.Surname>/hello-world.git
git push -u mirror --all

I'm scheduling replication as follows:

ssh -p 29418 gerrit.domain replication start

Which produce following log:

gerrit      | [2020-03-23 22:01:40,019 +0000] 6c533415 [sshd-SshDaemon[33060020](port=22)-nio2-thread-1] admin a/1000000 LOGIN FROM 172.64.1.1
gerrit      | [2020-03-23 22:01:40,071 +0000] 6c533415 [SSH replication start  (admin)] admin a/1000000 replication.start 7ms 1ms 0
gerrit      | [2020-03-23 22:01:40,102 +0000] 6c533415 [sshd-SshDaemon[33060020](port=22)-nio2-thread-2] admin a/1000000 LOGOUT

But then when process take place I got following Stack Trace:

gerrit      | [2020-03-23 22:02:04,660] [ReplicateTo-gitlab-mirror-1] ERROR com.googlesource.gerrit.plugins.replication.ReplicationTasksStorage : Error while deleting task d44f53430eda0b204ca13da6aab17c2173531c94
gerrit      | java.nio.file.NoSuchFileException: /srv/gerrit/data/replication/ref-updates/running/d44f53430eda0b204ca13da6aab17c2173531c94
gerrit      |   at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
gerrit      |   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
gerrit      |   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
gerrit      |   at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
gerrit      |   at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
gerrit      |   at java.nio.file.Files.delete(Files.java:1126)
gerrit      |   at com.googlesource.gerrit.plugins.replication.ReplicationTasksStorage$Task.finish(ReplicationTasksStorage.java:232)
gerrit      |   at com.googlesource.gerrit.plugins.replication.ReplicationTasksStorage.finish(ReplicationTasksStorage.java:130)
gerrit      |   at com.googlesource.gerrit.plugins.replication.Destination.notifyFinished(Destination.java:574)
gerrit      |   at com.googlesource.gerrit.plugins.replication.PushOne.runPushOperation(PushOne.java:413)
gerrit      |   at com.googlesource.gerrit.plugins.replication.PushOne.lambda$run$0(PushOne.java:300)
gerrit      |   at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:182)
gerrit      |   at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:170)
gerrit      |   at com.google.gerrit.server.git.PerThreadRequestScope$Propagator.lambda$scope$0(PerThreadRequestScope.java:70)
gerrit      |   at com.googlesource.gerrit.plugins.replication.PushOne.run(PushOne.java:303)
gerrit      |   at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:87)
gerrit      |   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
gerrit      |   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
gerrit      |   at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
gerrit      |   at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
gerrit      |   at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:610)
gerrit      |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
gerrit      |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
gerrit      |   at java.lang.Thread.run(Thread.java:748)

This is how data directory for replication looks like (I think whole time):

gerrit:/srv/gerrit$ find data/replication/
data/replication/
data/replication/ref-updates
data/replication/ref-updates/running
data/replication/ref-updates/building
data/replication/ref-updates/waiting
data/replication/ref-updates/waiting/50d5b9f61203cdd9223f21c21de7174f58a89bd3
data/replication/ref-updates/waiting/d44f53430eda0b204ca13da6aab17c2173531c94

Yep Gerrit tries to delete task from running (I have no idea why?) but task is in waiting. Gitlab repository does not get changes which is biggest problem.

I also tried queue replication event as follows, but that blocks indefinitely, until CTRL+C:

ssh -p 29418 gerrit.domain replication start --wait

Any idea what I'm missing or what more I could look for?

gerrit
0 Answers
Related