<div dir="ltr"><div>I don't think this is a gluster problem...</div><div><br></div>Each container is going to have its own notion of user ids, hence the mystery uid 1000 in the redis container. I suspect that if you exec into the gitlab container, it may be the one running as 1000 (guessing based on the file names). If you want to share volumes across containers, you're going to have to do something explicitly to make sure each of them (with their own uid/gid) can read/write the volume, for example by sharing the same gid across all containers.<div><br></div><div>I'm going to suggest not sharing the same volume across all 3 containers unless they need shared access to the data.</div><div><br></div><div>-John</div><div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 7, 2017 at 12:13 PM, Gaurav Chhabra <span dir="ltr"><<a href="mailto:varuag.chhabra@gmail.com" target="_blank">varuag.chhabra@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif"><span style="color:rgb(36,39,41);font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;font-size:15px">Hello,</span><br></div><div class="gmail_quote"><div dir="ltr"><p style="font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;margin:0px 0px 1em;padding:0px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:15px;line-height:inherit;vertical-align:baseline;clear:both;color:rgb(36,39,41)"><br></p><p style="font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;margin:0px 0px 1em;padding:0px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:15px;line-height:inherit;vertical-align:baseline;clear:both;color:rgb(36,39,41)">I am trying to setup GitLab, Redis and PostgreSQL containers in Kubernetes using Gluster for persistence. GlusterFS nodes are setup on machines (CentOS) external to Kubernetes cluster (running on RancherOS host). Issue is that when GitLab tries starting up, the login page doesn't load. It's a fresh setup and not something that stopped working now.<br></p><pre style="font-family:Consolas,Menlo,Monaco,"Lucida Console","Liberation Mono","DejaVu Sans Mono","Bitstream Vera Sans Mono","Courier New",monospace,sans-serif;margin-top:0px;margin-bottom:1em;padding:5px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:13px;line-height:inherit;vertical-align:baseline;width:auto;max-height:600px;overflow:auto;background-color:rgb(239,240,241);word-wrap:normal;color:rgb(36,39,41)"><code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:Consolas,Menlo,Monaco,"Lucida Console","Liberation Mono","DejaVu Sans Mono","Bitstream Vera Sans Mono","Courier New",monospace,sans-serif;vertical-align:baseline;white-space:inherit">root@gitlab-2797053212-ph4j8:/<wbr>var/log/gitlab/gitlab# tail -50 sidekiq.log
...
2017-09-07T11:53:03.099Z 547 TID-1fdf1k ERROR: Error fetching job: ERR Error running script (call to f_7b91ed9f4cba40689cea7172d1fd<wbr>3e08b2efd8c9): @user_script:7: @user_script: 7: -MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle<wbr>/ruby/2.3.0/gems/redis-3.3.3/<wbr>lib/redis/client.rb:121:in `call'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle<wbr>/ruby/2.3.0/gems/peek-redis-1.<wbr>2.0/lib/peek/views/redis.rb:9:<wbr>in `call'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle<wbr>/ruby/2.3.0/gems/redis-3.3.3/<wbr>lib/redis.rb:2399:in `block in _eval'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle<wbr>/ruby/2.3.0/gems/redis-3.3.3/<wbr>lib/redis.rb:58:in `block in synchronize'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /usr/lib/ruby/2.3.0/monitor.rb<wbr>:214:in `mon_synchronize'
2017-09-07T11:53:03.100Z 547 TID-1fdf1k ERROR: /home/git/gitlab/vendor/bundle<wbr>/ruby/2.3.0/gems/redis-3.3.3/<wbr>lib/redis.rb:58:in `synchronize'
...
</code></pre><p style="font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;margin:0px 0px 1em;padding:0px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:15px;line-height:inherit;vertical-align:baseline;clear:both;color:rgb(36,39,41)">So i checked for Redis container logs.</p><pre style="font-family:Consolas,Menlo,Monaco,"Lucida Console","Liberation Mono","DejaVu Sans Mono","Bitstream Vera Sans Mono","Courier New",monospace,sans-serif;margin-top:0px;margin-bottom:1em;padding:5px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:13px;line-height:inherit;vertical-align:baseline;width:auto;max-height:600px;overflow:auto;background-color:rgb(239,240,241);word-wrap:normal;color:rgb(36,39,41)"><code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:Consolas,Menlo,Monaco,"Lucida Console","Liberation Mono","DejaVu Sans Mono","Bitstream Vera Sans Mono","Courier New",monospace,sans-serif;vertical-align:baseline;white-space:inherit">[root@node-a ~]# docker logs -f 67d44f585705
...
...
[1] 07 Sep 14:43:48.140 # Background saving error
[1] 07 Sep 14:43:54.048 * 1 changes in 900 seconds. Saving...
[1] 07 Sep 14:43:54.048 * Background saving started by pid 2437
[2437] 07 Sep 14:43:54.053 # Failed opening .rdb for saving: Permission denied
...
</code></pre><p style="font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;margin:0px 0px 1em;padding:0px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:15px;line-height:inherit;vertical-align:baseline;clear:both;color:rgb(36,39,41)">Checked online for this issue and then noticed the following permissions and owner details <em style="margin:0px;padding:0px;border:0px;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline">inside</em>of Redis pod:</p><pre style="font-family:Consolas,Menlo,Monaco,"Lucida Console","Liberation Mono","DejaVu Sans Mono","Bitstream Vera Sans Mono","Courier New",monospace,sans-serif;margin-top:0px;margin-bottom:1em;padding:5px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:13px;line-height:inherit;vertical-align:baseline;width:auto;max-height:600px;overflow:auto;background-color:rgb(239,240,241);word-wrap:normal;color:rgb(36,39,41)"><code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:Consolas,Menlo,Monaco,"Lucida Console","Liberation Mono","DejaVu Sans Mono","Bitstream Vera Sans Mono","Courier New",monospace,sans-serif;vertical-align:baseline;white-space:inherit">[root@node-a ~]# docker exec -it 67d44f585705 bash
groups: cannot find name for group ID 2000
root@redis-2138096053-0mlx4:/# ls -ld /var/lib/redis/
drwxr-sr-x 12 1000 1000 8192 Sep 7 11:51 /var/lib/redis/
root@redis-2138096053-0mlx4:/#
root@redis-2138096053-0mlx4:/# ls -l /var/lib/redis/
total 22
drwxr-sr-x 2 1000 1000 6 Sep 6 10:37 backups
drwxr-sr-x 2 1000 1000 6 Sep 6 10:37 builds
drwxr-sr-x 2 redis redis 6 Sep 6 10:14 data
-rw-r--r-- 1 redis redis 13050 Sep 7 11:51 dump.rdb
-rwxr-xr-x 1 redis redis 21 Sep 5 11:00 index.html
drwxrws--- 2 1000 1000 6 Sep 6 10:37 repositories
drwxr-sr-x 5 1000 1000 55 Sep 6 10:37 shared
drwxr-sr-x 2 root root 8192 Sep 6 10:37 ssh
drwxr-sr-x 3 redis redis 70 Sep 7 10:20 tmp
drwx--S--- 2 1000 1000 6 Sep 6 10:37 uploads
root@redis-2138096053-0mlx4:/#
root@redis-2138096053-0mlx4:/# grep 1000 /etc/passwd
root@redis-2138096053-0mlx4:/#
</code></pre><p style="font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;margin:0px 0px 1em;padding:0px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:15px;line-height:inherit;vertical-align:baseline;clear:both;color:rgb(36,39,41)">Ran following and all looked fine.</p><pre style="font-family:Consolas,Menlo,Monaco,"Lucida Console","Liberation Mono","DejaVu Sans Mono","Bitstream Vera Sans Mono","Courier New",monospace,sans-serif;margin-top:0px;margin-bottom:1em;padding:5px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:13px;line-height:inherit;vertical-align:baseline;width:auto;max-height:600px;overflow:auto;background-color:rgb(239,240,241);word-wrap:normal;color:rgb(36,39,41)"><code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:Consolas,Menlo,Monaco,"Lucida Console","Liberation Mono","DejaVu Sans Mono","Bitstream Vera Sans Mono","Courier New",monospace,sans-serif;vertical-align:baseline;white-space:inherit">root@redis-2138096053-0mlx4:/# chown redis:redis -R /var/lib/redis/
</code></pre><p style="margin:0px 0px 1em;padding:0px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;line-height:inherit;vertical-align:baseline;clear:both;color:rgb(36,39,41)"><font face="arial, helvetica, sans-serif">However, when i deleted and ran the GitLab deployment YAML again, the permissions inside the Redis container <em style="margin:0px;padding:0px;border:0px;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;vertical-align:baseline">again</em> got skewed up. I am not sure whether Gluster is messing up with the Redis file/folders permissions but i can't think of any other reason </font></p><div class="gmail_default" style="display:inline"><font face="arial, helvetica, sans-serif">except for mount.</font></div><p></p><p style="font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;margin:0px 0px 1em;padding:0px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:15px;line-height:inherit;vertical-align:baseline;clear:both;color:rgb(36,39,41)">One thing i would like to highlight is that all the three containers are using the <em style="margin:0px;padding:0px;border:0px;font-variant:inherit;font-weight:inherit;font-stretch:inherit;font-size:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline">same</em> PVC</p><pre style="font-family:Consolas,Menlo,Monaco,"Lucida Console","Liberation Mono","DejaVu Sans Mono","Bitstream Vera Sans Mono","Courier New",monospace,sans-serif;margin-top:0px;margin-bottom:1em;padding:5px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:13px;line-height:inherit;vertical-align:baseline;width:auto;max-height:600px;overflow:auto;background-color:rgb(239,240,241);word-wrap:normal;color:rgb(36,39,41)"><code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:Consolas,Menlo,Monaco,"Lucida Console","Liberation Mono","DejaVu Sans Mono","Bitstream Vera Sans Mono","Courier New",monospace,sans-serif;vertical-align:baseline;white-space:inherit">- name: gluster-vol1
persistentVolumeClaim:
claimName: gluster-dyn-pvc
</code></pre><p style="font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;margin:0px 0px 1em;padding:0px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:15px;line-height:inherit;vertical-align:baseline;clear:both;color:rgb(36,39,41)">Above is common for all three. What differs is shown below:</p><pre style="font-family:Consolas,Menlo,Monaco,"Lucida Console","Liberation Mono","DejaVu Sans Mono","Bitstream Vera Sans Mono","Courier New",monospace,sans-serif;margin-top:0px;margin-bottom:1em;padding:5px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;font-size:13px;line-height:inherit;vertical-align:baseline;width:auto;max-height:600px;overflow:auto;background-color:rgb(239,240,241);word-wrap:normal;color:rgb(36,39,41)"><code style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:Consolas,Menlo,Monaco,"Lucida Console","Liberation Mono","DejaVu Sans Mono","Bitstream Vera Sans Mono","Courier New",monospace,sans-serif;vertical-align:baseline;white-space:inherit">a) postgresql-deployment.yaml
volumeMounts:
- name: gluster-vol1
mountPath: /var/lib/postgresql
b) redisio-deployment.yaml
volumeMounts:
- name: gluster-vol1
mountPath: /var/lib/redis
c) gitlab-deployment.yaml
volumeMounts:
- name: gluster-vol1
mountPath: /home/git/data
</code></pre><p style="margin:0px 0px 1em;padding:0px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;line-height:inherit;vertical-align:baseline;clear:both;color:rgb(36,39,41)"><font face="verdana, sans-serif">Any suggestion? Also, i</font></p><div class="gmail_default" style="font-family:verdana,sans-serif;display:inline"><font face="verdana, sans-serif"> guess this is not </font></div><font face="verdana, sans-serif">the right way to use the same <code style="margin:0px;padding:1px 5px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;line-height:inherit;vertical-align:baseline;background-color:rgb(239,240,241)">PVC</code>/<code style="margin:0px;padding:1px 5px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;line-height:inherit;vertical-align:baseline;background-color:rgb(239,240,241)">Storage Class</code> for all three containers<div class="gmail_default" style="font-family:verdana,sans-serif;display:inline"> because i just noticed that all contents reside in the same dir inside Gluster nodes.</div></font><p></p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-variant-numeric:inherit;font-stretch:inherit;line-height:inherit;vertical-align:baseline;clear:both;color:rgb(36,39,41)"><font face="verdana, sans-serif"></font></p><div class="gmail_default" style="font-family:verdana,sans-serif;display:inline"><font face="verdana, sans-serif">I know there are many things involved besides Gluster so this may not be _the_ right forum but amongst all, my gut feeling is that Gluster might be the reason for the permission issue.</font></div><p></p></div></div></div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>