[Gluster-users] I/O error for one folder within the mountpoint
Ravishankar N
ravishankar at redhat.com
Fri Jul 7 10:28:27 UTC 2017
On 07/07/2017 03:39 PM, Florian Leleu wrote:
>
> I guess you're right aboug gfid, I got that:
>
> [2017-07-07 07:35:15.197003] W [MSGID: 108008]
> [afr-self-heal-name.c:354:afr_selfheal_name_gfid_mismatch_check]
> 0-applicatif-replicate-0: GFID mismatch for
> <gfid:3fa785b5-4242-4816-a452-97da1a5e45c6>/snooper
> b9222041-72dd-43a3-b0ab-4169dbd9a87f on applicatif-client-1 and
> 60056f98-20f8-4949-a4ae-81cc1a139147 on applicatif-client-0
>
> Can you tell me how can I fix that ?If that helps I don't mind
> deleting the whole folder snooper, I have backup.
>
The steps listed in "Fixing Directory entry split-brain:" of
https://gluster.readthedocs.io/en/latest/Troubleshooting/split-brain/
should give you an idea. It is for files whose gfids mismatch but the
steps are similar for directories too.
If the contents of the snooper is same on all bricks , you could also
try directly deleting the directory from one of the bricks and
immediately doing an `ls snooper` from the mount to trigger heals to
recreate the entries.
Hope this helps
Ravi
>
> Thanks.
>
>
> Le 07/07/2017 à 11:54, Ravishankar N a écrit :
>> What does the mount log say when you get the EIO error on snooper?
>> Check if there is a gfid mismatch on snooper directory or the files
>> under it for all 3 bricks. In any case the mount log or the
>> glustershd.log of the 3 nodes for the gfids you listed below should
>> give you some idea on why the files aren't healed.
>> Thanks.
>>
>> On 07/07/2017 03:10 PM, Florian Leleu wrote:
>>>
>>> Hi Ravi,
>>>
>>> thanks for your answer, sure there you go:
>>>
>>> # gluster volume heal applicatif info
>>> Brick ipvr7.xxx:/mnt/gluster-applicatif/brick
>>> <gfid:e3b5ef36-a635-4e0e-bd97-d204a1f8e7ed>
>>> <gfid:f8030467-b7a3-4744-a945-ff0b532e9401>
>>> <gfid:def47b0b-b77e-4f0e-a402-b83c0f2d354b>
>>> <gfid:46f76502-b1d5-43af-8c42-3d833e86eb44>
>>> <gfid:d27a71d2-6d53-413d-b88c-33edea202cc2>
>>> <gfid:7e7f02b2-3f2d-41ff-9cad-cd3b5a1e506a>
>>> Status: Connected
>>> Number of entries: 6
>>>
>>> Brick ipvr8.xxx:/mnt/gluster-applicatif/brick
>>> <gfid:47ddf66f-a5e9-4490-8cd7-88e8b812cdbd>
>>> <gfid:8057d06e-5323-47ff-8168-d983c4a82475>
>>> <gfid:5b2ea4e4-ce84-4f07-bd66-5a0e17edb2b0>
>>> <gfid:baedf8a2-1a3f-4219-86a1-c19f51f08f4e>
>>> <gfid:8261c22c-e85a-4d0e-b057-196b744f3558>
>>> <gfid:842b30c1-6016-45bd-9685-6be76911bd98>
>>> <gfid:1fcaef0f-c97d-41e6-87cd-cd02f197bf38>
>>> <gfid:9d041c80-b7e4-4012-a097-3db5b09fe471>
>>> <gfid:ff48a14a-c1d5-45c6-a52a-b3e2402d0316>
>>> <gfid:01409b23-eff2-4bda-966e-ab6133784001>
>>> <gfid:c723e484-63fc-4267-b3f0-4090194370a0>
>>> <gfid:fb1339a8-803f-4e29-b0dc-244e6c4427ed>
>>> <gfid:056f3bba-6324-4cd8-b08d-bdf0fca44104>
>>> <gfid:a8f6d7e5-0ff2-4747-89f3-87592597adda>
>>> <gfid:3f6438a0-2712-4a09-9bff-d5a3027362b4>
>>> <gfid:392c8e2f-9da4-4af8-a387-bfdfea2f404e>
>>> <gfid:37e1edfd-9f58-4da3-8abe-819670c70906>
>>> <gfid:15b7cdb3-aae8-4ca5-b28c-e87a3e599c9b>
>>> <gfid:1d087e51-fb40-4606-8bb5-58936fb11a4c>
>>> <gfid:bb0352b9-4a5e-4075-9179-05c3a5766cf4>
>>> <gfid:40133fcf-a1fb-4d60-b169-e2355b66fb53>
>>> <gfid:00f75963-1b4a-4d75-9558-36b7d85bd30b>
>>> <gfid:2c0babdf-c828-475e-b2f5-0f44441fffdc>
>>> <gfid:bbeff672-43ef-48c9-a3a2-96264aa46152>
>>> <gfid:6c0969dd-bd30-4ba0-a7e5-ba4b3a972b9f>
>>> <gfid:4c81ea14-56f4-4b30-8fff-c088fe4b3dff>
>>> <gfid:1072cda3-53c9-4b95-992d-f102f6f87209>
>>> <gfid:2e8f9f29-78f9-4402-bc0c-e63af8cf77d6>
>>> <gfid:eeaa2765-44f4-4891-8502-5787b1310de2>
>>> Status: Connected
>>> Number of entries: 29
>>>
>>> Brick ipvr9.xxx:/mnt/gluster-applicatif/brick
>>> <gfid:e3b5ef36-a635-4e0e-bd97-d204a1f8e7ed>
>>> <gfid:f8030467-b7a3-4744-a945-ff0b532e9401>
>>> <gfid:def47b0b-b77e-4f0e-a402-b83c0f2d354b>
>>> <gfid:46f76502-b1d5-43af-8c42-3d833e86eb44>
>>> <gfid:d27a71d2-6d53-413d-b88c-33edea202cc2>
>>> <gfid:7e7f02b2-3f2d-41ff-9cad-cd3b5a1e506a>
>>> Status: Connected
>>> Number of entries: 6
>>>
>>>
>>> # gluster volume heal applicatif info split-brain
>>> Brick ipvr7.xxx:/mnt/gluster-applicatif/brick
>>> Status: Connected
>>> Number of entries in split-brain: 0
>>>
>>> Brick ipvr8.xxx:/mnt/gluster-applicatif/brick
>>> Status: Connected
>>> Number of entries in split-brain: 0
>>>
>>> Brick ipvr9.xxx:/mnt/gluster-applicatif/brick
>>> Status: Connected
>>> Number of entries in split-brain: 0
>>>
>>> Doesn't it seem odd that the first command give some different output ?
>>>
>>> Le 07/07/2017 à 11:31, Ravishankar N a écrit :
>>>> On 07/07/2017 01:23 PM, Florian Leleu wrote:
>>>>>
>>>>> Hello everyone,
>>>>>
>>>>> first time on the ML so excuse me if I'm not following well the
>>>>> rules, I'll improve if I get comments.
>>>>>
>>>>> We got one volume "applicatif" on three nodes (2 and 1 arbiter),
>>>>> each following command was made on node ipvr8.xxx:
>>>>>
>>>>> # gluster volume info applicatif
>>>>>
>>>>> Volume Name: applicatif
>>>>> Type: Replicate
>>>>> Volume ID: ac222863-9210-4354-9636-2c822b332504
>>>>> Status: Started
>>>>> Snapshot Count: 0
>>>>> Number of Bricks: 1 x (2 + 1) = 3
>>>>> Transport-type: tcp
>>>>> Bricks:
>>>>> Brick1: ipvr7.xxx:/mnt/gluster-applicatif/brick
>>>>> Brick2: ipvr8.xxx:/mnt/gluster-applicatif/brick
>>>>> Brick3: ipvr9.xxx:/mnt/gluster-applicatif/brick (arbiter)
>>>>> Options Reconfigured:
>>>>> performance.read-ahead: on
>>>>> performance.cache-size: 1024MB
>>>>> performance.quick-read: off
>>>>> performance.stat-prefetch: on
>>>>> performance.io-cache: off
>>>>> transport.address-family: inet
>>>>> performance.readdir-ahead: on
>>>>> nfs.disable: off
>>>>>
>>>>> # gluster volume status applicatif
>>>>> Status of volume: applicatif
>>>>> Gluster process TCP Port RDMA Port
>>>>> Online Pid
>>>>> ------------------------------------------------------------------------------
>>>>> Brick ipvr7.xxx:/mnt/gluster-applicatif/
>>>>> brick 49154 0
>>>>> Y 2814
>>>>> Brick ipvr8.xxx:/mnt/gluster-applicatif/
>>>>> brick 49154 0
>>>>> Y 2672
>>>>> Brick ipvr9.xxx:/mnt/gluster-applicatif/
>>>>> brick 49154 0
>>>>> Y 3424
>>>>> NFS Server on localhost 2049 0
>>>>> Y 26530
>>>>> Self-heal Daemon on localhost N/A N/A Y
>>>>> 26538
>>>>> NFS Server on ipvr9.xxx 2049 0 Y 12238
>>>>> Self-heal Daemon on ipvr9.xxx N/A N/A Y 12246
>>>>> NFS Server on ipvr7.xxx 2049 0 Y 2234
>>>>> Self-heal Daemon on ipvr7.xxx N/A N/A Y 2243
>>>>>
>>>>> Task Status of Volume applicatif
>>>>> ------------------------------------------------------------------------------
>>>>> There are no active volume tasks
>>>>>
>>>>> The volume is mounted with autofs (nfs) in /home/applicatif and
>>>>> one folder is "broken":
>>>>>
>>>>> l /home/applicatif/services/
>>>>> ls: cannot access /home/applicatif/services/snooper: Input/output
>>>>> error
>>>>> total 16
>>>>> lrwxrwxrwx 1 applicatif applicatif 9 Apr 6 15:53 config ->
>>>>> ../config
>>>>> lrwxrwxrwx 1 applicatif applicatif 7 Apr 6 15:54 .pwd -> ../.pwd
>>>>> drwxr-xr-x 3 applicatif applicatif 4096 Apr 12 10:24 querybuilder
>>>>> d????????? ? ? ? ? ? snooper
>>>>> drwxr-xr-x 3 applicatif applicatif 4096 Jul 6 02:57 snooper_new
>>>>> drwxr-xr-x 16 applicatif applicatif 4096 Jul 6 02:58 snooper_old
>>>>> drwxr-xr-x 4 applicatif applicatif 4096 Jul 4 23:45 ssnooper
>>>>>
>>>>> I checked wether there was a heal, and it seems so:
>>>>>
>>>>> # gluster volume heal applicatif statistics heal-count
>>>>> Gathering count of entries to be healed on volume applicatif has
>>>>> been successful
>>>>>
>>>>> Brick ipvr7.xxx:/mnt/gluster-applicatif/brick
>>>>> Number of entries: 8
>>>>>
>>>>> Brick ipvr8.xxx:/mnt/gluster-applicatif/brick
>>>>> Number of entries: 29
>>>>>
>>>>> Brick ipvr9.xxx:/mnt/gluster-applicatif/brick
>>>>> Number of entries: 8
>>>>>
>>>>> But actually in the brick on each server the folder "snooper" is fine.
>>>>>
>>>>> I tried rebooting the servers, restarting gluster after killing
>>>>> every process using it but it's not working.
>>>>>
>>>>> Has anyone already experienced that ? Any help would be nice.
>>>>>
>>>>
>>>> Can you share the output of `gluster volume heal <volname> info`
>>>> and `gluster volume heal <volname> info split-brain`? If the second
>>>> command shows entries, please also share the getfattr output from
>>>> the bricks for these files (getfattr -d -m . -e hex
>>>> /brick/path/to/file).
>>>> -Ravi
>>>>>
>>>>> Thanks a lot !
>>>>>
>>>>> --
>>>>>
>>>>> Cordialement,
>>>>>
>>>>> <http://www.cognix-systems.com/>
>>>>>
>>>>> Florian LELEU
>>>>> Responsable Hosting, Cognix Systems
>>>>>
>>>>> *Rennes* | Brest | Saint-Malo | Paris
>>>>> florian.leleu at cognix-systems.com
>>>>> <mailto:florian.leleu at cognix-systems.com>
>>>>>
>>>>> Tél. : 02 99 27 75 92
>>>>>
>>>>>
>>>>> Facebook Cognix Systems <https://www.facebook.com/cognix.systems/>
>>>>> Twitter Cognix Systems <https://twitter.com/cognixsystems>
>>>>> Logo Cognix Systems <http://www.cognix-systems.com/>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>>
>>>
>>> --
>>>
>>> Cordialement,
>>>
>>> <http://www.cognix-systems.com/>
>>>
>>> Florian LELEU
>>> Responsable Hosting, Cognix Systems
>>>
>>> *Rennes* | Brest | Saint-Malo | Paris
>>> florian.leleu at cognix-systems.com
>>> <mailto:florian.leleu at cognix-systems.com>
>>>
>>> Tél. : 02 99 27 75 92
>>>
>>>
>>> Facebook Cognix Systems <https://www.facebook.com/cognix.systems/>
>>> Twitter Cognix Systems <https://twitter.com/cognixsystems>
>>> Logo Cognix Systems <http://www.cognix-systems.com/>
>>>
>>
>
> --
>
> Cordialement,
>
> <http://www.cognix-systems.com/>
>
> Florian LELEU
> Responsable Hosting, Cognix Systems
>
> *Rennes* | Brest | Saint-Malo | Paris
> florian.leleu at cognix-systems.com <mailto:florian.leleu at cognix-systems.com>
>
> Tél. : 02 99 27 75 92
>
>
> Facebook Cognix Systems <https://www.facebook.com/cognix.systems/>
> Twitter Cognix Systems <https://twitter.com/cognixsystems>
> Logo Cognix Systems <http://www.cognix-systems.com/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170707/c28b3e02/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 4935 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170707/c28b3e02/attachment.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 1444 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170707/c28b3e02/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 1623 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170707/c28b3e02/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 1474 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170707/c28b3e02/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 4935 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170707/c28b3e02/attachment-0001.jpe>
More information about the Gluster-users
mailing list