Resolving Issues on GoHarbor/Harbor (CNCF)

In this post I try to help you to get along with certain harbor issues.

Harbor is an open source trusted cloud native registry project that stores, signs, and scans content. Harbor extends the open source Docker Distribution by adding the functionalities usually required by users such as security, identity and management. Having a registry closer to the build and run environment can improve the image transfer efficiency. Harbor supports replication of images between registries, and also offers advanced security features such as user management, access control and activity auditing.

Error Picture:

1. GC or Jobs are not executed either manual or automatically
2. Jobs are staying pending
3. You’ll find an error like this in the logs:

ERROR: worker.fetch - LOADING Redis is loading the dataset in memory

Solution:

To resolve this you need to flush the redis

Exec into the redis container:

$ docker ps |grep -i redis
$ docker exec -it <container_id_or_name> bash

Execute the following commands to flush the Redis

redis-cli FLUSHDB
redis-cli -n DB_NUMBER FLUSHDB
redis-cli -n DB_NUMBER FLUSHDB ASYNC
redis-cli FLUSHALL
redis-cli FLUSHALL ASYNC

Error Picture:

1. Replication can not be stopped/started and jobs are Pending
2. The result is, you cant delete a failing replication job.
3. You see “internal server error”

Solution:

Switch to the replication_execution table

\d replication_execution;

2. Update the status of every execution job to Succeed

update replication_execution set status = 'Succeed',total = '1', end_time = now()

3. Go to the UI and delete the former pending/failing replication job

Error Picture:

1. Your certificate needs to be replaced because it is about to expire.

Solution:

1. Replace the certificiate + private key in your harbor installation folder
   In my case this was /opt/harbor/ssl/xyz.crt.pem and /opt/harbor/ssl/xyz.key.pem
2. Replace both on the running instance of harbor (when installing it, it copies those files into another directory) so do the following:

# Copy the certificate
$ cp /opt/harbor/ssl/xyz.crt.pem /var/harbor/data/secret/cert/server.crt
# Copy the key
$ cp /opt/harbor/ssl/xyz.key.pem /var/harbor/data/secret/cert/server.key

3. Restart both nginx services

$ docker ps  |grep goharbor  |grep "nginx -g" | awk '{print $1}'
c1498j072f44
27f4lg4db7a6
$ docker restart c1498j072f44 27f4lg4db7a6

4. Check that your certificate and key are renewed.

Error Picture: (Update 2021–03–18)

1. The logs show the following error

ERROR: dead_pool_reaper.reap - MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk.

Solution

$ redis-cli config 
127.0.0.1:6379> set stop-writes-on-bgsave-error no

Error Picture: (Update 2021–11–15)

1. The Vulnurability database does not update

harbor.scanner-adapter/vulnerability-database-updated-at:05/7/20, 7:40 AM

Solution

Append the following lines to harbors docker-compose.yml

trivy-adapter:
[...]
  volumes:
    - type: bind
      read_only: true
      source: /var/run/docker.sock
      target: /var/run/docker.sock
[...]

Then run docker-compuse up -d and the trivy adapter should be recreated.

Then we need to manually trigger an update of the database using for example the debian:10 image.

$ docker exec -it trivy-adapter trivy image debian:10
2021-11-15T08:08:13.413Z INFO Need to update DB
2021-11-15T08:08:13.413Z INFO Downloading DB...
24.71 MiB / 24.71 MiB [--------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 11.25 MiB p/s 3s
2021-11-15T08:08:18.599Z INFO Detecting Debian vulnerabilities...

Now when checking harbor again you should also see that the database is updated:

harbor.scanner-adapter/vulnerability-database-updated-at:11/15/21, 7:40 AM

Sources

• https://github.com/goharbor/harbor
• https://www.cyberciti.biz/faq/how-to-flush-redis-cache-and-delete-everything-using-the-cli/
• https://chartio.com/resources/tutorials/how-to-list-databases-and-tables-in-postgresql-using-psql/
• https://stackoverflow.com/questions/19581059/misconf-redis-is-configured-to-save-rdb-snapshots