<div dir="ltr">+Raghavendra/Nithya<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 6, 2017 at 7:41 PM, Jarsulic, Michael [CRI] <span dir="ltr"><<a href="mailto:mjarsulic@bsd.uchicago.edu" target="_blank">mjarsulic@bsd.uchicago.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
I am still working at recovering from a few failed OS hard drives on my gluster storage and have been removing, and re-adding bricks quite a bit. I noticed yesterday night that some of the directories are not visible when I access them through the client, but are still on the brick. For example:<br>
<br>
Client:<br>
<br>
# ls /scratch/dw<br>
Ethiopian_imputation HGDP Rolwaling Tibetan_Alignment<br>
<br>
Brick:<br>
<br>
# ls /data/brick1/scratch/dw<br>
1000GP_Phase3 Ethiopian_imputation HGDP Rolwaling SGDP Siberian_imputation Tibetan_Alignment mapata<br>
<br>
<br>
However, the directory is accessible on the client side (just not visible):<br>
<br>
# stat /scratch/dw/SGDP<br>
File: `/scratch/dw/SGDP'<br>
Size: 212992 Blocks: 416 IO Block: 131072 directory<br>
Device: 21h/33d Inode: 11986142482805280401 Links: 2<br>
Access: (0775/drwxrwxr-x) Uid: (339748621/dw) Gid: (339748621/dw)<br>
Access: 2017-06-02 16:00:02.398109000 -0500<br>
Modify: 2017-06-06 06:59:13.004947703 -0500<br>
Change: 2017-06-06 06:59:13.004947703 -0500<br>
<br>
<br>
The only place I see the directory mentioned in the log files are in the rebalance logs. The following piece may provide a clue as to what is going on:<br>
<br>
[2017-06-05 20:46:51.752726] E [MSGID: 109010] [dht-rebalance.c:2259:gf_<wbr>defrag_get_entry] 0-hpcscratch-dht: /dw/SGDP/HGDP00476_chr6.tped gfid not present<br>
[2017-06-05 20:46:51.752742] E [MSGID: 109010] [dht-rebalance.c:2259:gf_<wbr>defrag_get_entry] 0-hpcscratch-dht: /dw/SGDP/LP6005441-DNA_B08_<wbr>chr4.tmp gfid not present<br>
[2017-06-05 20:46:51.752773] E [MSGID: 109010] [dht-rebalance.c:2259:gf_<wbr>defrag_get_entry] 0-hpcscratch-dht: /dw/SGDP/LP6005441-DNA_B08.<wbr>geno.tmp gfid not present<br>
[2017-06-05 20:46:51.752789] E [MSGID: 109010] [dht-rebalance.c:2259:gf_<wbr>defrag_get_entry] 0-hpcscratch-dht: /dw/SGDP/LP6005443-DNA_D02_<wbr>chr4.out gfid not present<br>
<br>
This happened yesterday during a rebalance that failed. However, doing a rebalance fix-layout allowed my to clean up these errors and successfully complete a migration to a re-added brick.<br>
<br>
<br>
Here is the information for my storage cluster:<br>
<br>
# gluster volume info<br>
<br>
Volume Name: hpcscratch<br>
Type: Distribute<br>
Volume ID: 80b8eeed-1e72-45b9-8402-<wbr>e01ae0130105<br>
Status: Started<br>
Number of Bricks: 6<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: fs001-ib:/data/brick2/scratch<br>
Brick2: fs003-ib:/data/brick5/scratch<br>
Brick3: fs003-ib:/data/brick6/scratch<br>
Brick4: fs004-ib:/data/brick7/scratch<br>
Brick5: fs001-ib:/data/brick1/scratch<br>
Brick6: fs004-ib:/data/brick8/scratch<br>
Options Reconfigured:<br>
server.event-threads: 8<br>
performance.client-io-threads: on<br>
client.event-threads: 8<br>
performance.cache-size: 32MB<br>
performance.readdir-ahead: on<br>
diagnostics.client-log-level: INFO<br>
diagnostics.brick-log-level: INFO<br>
<br>
<br>
Mount points for the bricks:<br>
<br>
/dev/sdb on /data/brick2 type xfs (rw,noatime,nobarrier)<br>
/dev/sda on /data/brick1 type xfs (rw,noatime,nobarrier)<br>
<br>
<br>
Mount point on the client:<br>
<br>
10.xx.xx.xx:/hpcscratch on /scratch type fuse.glusterfs (rw,default_permissions,allow_<wbr>other,max_read=131072)<br>
<br>
<br>
My question is what are some of the possibilities for the root cause of this issue and what is the recommended way of recovering from it? Let me know if you need any more information.<br>
<br>
<br>
--<br>
Mike Jarsulic<br>
Sr. HPC Administrator<br>
Center for Research Informatics | University of Chicago<br>
773.702.2066<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div>