[Gluster-users] Problem with files showing zero links

[Joe Julian]

Response inline

On 01/28/2016 07:28 AM, Ronny Adsetts wrote:
> Hi all,
>
> Have an issue I'm having trouble explaining or getting to the bottom of. I have a two node, two brick replicated 75G volume containing ~4661 files:
>
> gotham:~# gluster volume status software
> Status of volume: software
> Gluster process                                         Port    Online  Pid
> ------------------------------------------------------------------------------
> Brick gotham.stor.graysofwestminster.co.uk:/data/gluste
> rfs/software/brick1/brick                               49152   Y       27296
> Brick metropolis.stor.graysofwestminster.co.uk:/data/gl
> usterfs/software/brick1/brick                           49152   Y       30335
> NFS Server on localhost                                 2049    Y       27309
> Self-heal Daemon on localhost                           N/A     Y       27316
> NFS Server on metropolis.stor.graysofwestminster.co.uk  2049    Y       30348
> Self-heal Daemon on metropolis.stor.graysofwestminster.
> co.uk                                                   N/A     Y       30355
>
> Task Status of Volume software
> ------------------------------------------------------------------------------
> There are no active volume tasks
>
>
> gotham:~# gluster volume heal software info
> Brick gotham.graysofwestminster.co.uk:/data/glusterfs/software/brick1/brick/
> Number of entries: 0
>
> Brick metropolis.graysofwestminster.co.uk:/data/glusterfs/software/brick1/brick/
> Number of entries: 0
>
>
> I have the volume mounted on both nodes at /stor/software. On one of the nodes, there are lots of files showing with a link count of 0 which is leading to the Windows software using the data having validation issues:
>
> metropolis:~# ls -ltr /stor/software/win_patches/ | head
> total 21585884
> -rwxr--r-- 0 ainet Domain Admins     27816 Jan  4  2006 nullpatch.exe
> -rwxrwxr-- 0 ronny Domain Admins 135477136 Oct 29  2006 W2KA_SP4_128_x86_ENU.exe
> -rwxrwxr-- 0 ronny Domain Admins    432568 Oct 29  2006 MSXML26SP3.exe
> -rwxrwxr-- 0 ronny Domain Admins   1070592 Oct 29  2006 msxml3sp7.msi
> -rwxrwxr-- 0 ronny Domain Admins    711160 Oct 29  2006 Windows2000-KB842773.EXE
> -rwxrwxr-- 0 ronny Domain Admins    760824 Oct 29  2006 Windows2000-KB842933.EXE
> -rwxrwxr-- 0 ronny Domain Admins   2585872 Oct 29  2006 WindowsInstaller-KB893803v2.exe
> -rwxrwxr-- 0 ronny Domain Admins    106240 Oct 29  2006 Windows-KB870669.exe
> -rwxrwxr-- 0 ronny Domain Admins   1260024 Oct 29  2006 Windows2000-KB908506.EXE
>
>
>  From the other node:
>
> gotham:~# ls -ltr /stor/software/win_patches/ | head
> total 21585884
> -rwxr--r-- 1 1066 1016     27816 Jan  4  2006 nullpatch.exe
> -rwxrwxr-- 1 1045 1016 135477136 Oct 29  2006 W2KA_SP4_128_x86_ENU.exe
> -rwxrwxr-- 1 1045 1016    432568 Oct 29  2006 MSXML26SP3.exe
> -rwxrwxr-- 1 1045 1016   1070592 Oct 29  2006 msxml3sp7.msi
> -rwxrwxr-- 1 1045 1016    711160 Oct 29  2006 Windows2000-KB842773.EXE
> -rwxrwxr-- 1 1045 1016    760824 Oct 29  2006 Windows2000-KB842933.EXE
> -rwxrwxr-- 1 1045 1016   2585872 Oct 29  2006 WindowsInstaller-KB893803v2.exe
> -rwxrwxr-- 1 1045 1016    106240 Oct 29  2006 Windows-KB870669.exe
> -rwxrwxr-- 1 1045 1016   1260024 Oct 29  2006 Windows2000-KB908506.EXE
>
>
> I'm seeing the following and lots more similar in the logs:
>
> [2016-01-28 15:13:15.099478] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-software-client-1: remote operation failed: No such file or directory. Path: /win_patches/SkypeSetup71732106.msi (3489f648-9fde-4bcc-b906-6ef88ffcf90f)

client-1 is brick2, metropolis. That error says the file is missing, but 
it also shows the gfid. Perhaps the gfid hardlink is missing (the 
.glusterfs tree).

Check the metadata for the file on both bricks:
    getfattr -m . -d -e hex 
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi

Ensure the gfid is the same on both. Check the number of links on both, 
there should be 2. With a gifd of 489f648-9fde-4bcc-b906-6ef88ffcf90f, 
that file should be hard linked to 
.glusterfs/48/9f/489f648-9fde-4bcc-b906-6ef88ffcf90f on both bricks.
> File does exist on both bricks:
>
> metropolis:~# ls -l /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
> -rwxr--r-- 1 ainet Domain Admins 44380160 Dec 31 11:41 /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
>
> gotham:~# ls -l /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
> -rwxr--r-- 2 1066 1016 44380160 Dec 31 11:41 /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
>
> Any pointers to get to the bottom of this would be greatly appreciated.

Look for log entries in the brick logs that corrospond to the timestamps 
on the client-side and see if there's some correlation.

>
> Thanks.
>
> Ronny
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160128/4e3f01e2/attachment.html>