[Gluster-devel] Input/output error when files in .shard folder are deleted

Wed Oct 26 01:08:30 UTC 2016

Hi,

Pls see the client log below.

[2016-10-24 10:29:51.111603] I [fuse-bridge.c:5171:fuse_graph_setup]
0-fuse: switched to graph 0
[2016-10-24 10:29:51.111662] I [MSGID: 114035]
[client-handshake.c:193:client_set_lk_version_cbk]
0-testHeal-client-2: Server lk version = 1
[2016-10-24 10:29:51.112371] I [fuse-bridge.c:4083:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22
kernel 7.22
[2016-10-24 10:29:51.113563] I [MSGID: 108031]
[afr-common.c:2071:afr_local_discovery_cbk] 0-testHeal-replicate-0:
selecting local read_child testHeal-client-2
[2016-10-24 10:29:51.113604] I [MSGID: 108031]
[afr-common.c:2071:afr_local_discovery_cbk] 0-testHeal-replicate-0:
selecting local read_child testHeal-client-0
[2016-10-24 10:29:51.113630] I [MSGID: 108031]
[afr-common.c:2071:afr_local_discovery_cbk] 0-testHeal-replicate-0:
selecting local read_child testHeal-client-1
[2016-10-24 10:29:54.016802] W [MSGID: 108001]
[afr-transaction.c:789:afr_handle_quorum] 0-testHeal-replicate-0:
/.shard/9061198a-eb7e-45a2-93fb-eb396d1b2727.1: F
ailing MKNOD as quorum is not met
[2016-10-24 10:29:54.019330] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-0:
remote operation failed. Path: (null) (00000000-
0000-0000-0000-000000000000) [Invalid argument]
[2016-10-24 10:29:54.019343] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-2:
remote operation failed. Path: (null) (00000000-
0000-0000-0000-000000000000) [Invalid argument]
[2016-10-24 10:29:54.019373] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-1:
remote operation failed. Path: (null) (00000000-
0000-0000-0000-000000000000) [Invalid argument]
[2016-10-24 10:29:54.019854] E [MSGID: 133010]
[shard.c:1582:shard_common_lookup_shards_cbk] 0-testHeal-shard: Lookup
on shard 1 failed. Base file gfid = 9061198a
-eb7e-45a2-93fb-eb396d1b2727 [Input/output error]
[2016-10-24 10:29:54.020886] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 135: READ => -1
gfid=9061198a-eb7e-45a2-93fb-eb396d1b2727 fd=0x7f70c80d12dc (
Input/output error)
[2016-10-24 10:29:54.118264] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-0:
remote operation failed. Path: (null) (00000000-
0000-0000-0000-000000000000) [Invalid argument]
[2016-10-24 10:29:54.118308] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-2:
remote operation failed. Path: (null)
(00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-10-24 10:29:54.118329] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-1:
remote operation failed. Path: (null)
(00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-10-24 10:29:54.118751] E [MSGID: 133010]
[shard.c:1582:shard_common_lookup_shards_cbk] 0-testHeal-shard: Lookup
on shard 1 failed. Base file gfid =
9061198a-eb7e-45a2-93fb-eb396d1b2727 [Input/output error]
[2016-10-24 10:29:54.118787] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 137: READ => -1
gfid=9061198a-eb7e-45a2-93fb-eb396d1b2727 fd=0x7f70c80d12dc
(Input/output error)
[2016-10-24 10:29:54.119330] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-1:
remote operation failed. Path: (null)
(00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-10-24 10:29:54.119338] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-0:
remote operation failed. Path: (null)
(00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-10-24 10:29:54.119368] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-2:
remote operation failed. Path: (null)
(00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-10-24 10:29:54.119674] E [MSGID: 133010]
[shard.c:1582:shard_common_lookup_shards_cbk] 0-testHeal-shard: Lookup
on shard 1 failed. Base file gfid =
9061198a-eb7e-45a2-93fb-eb396d1b2727 [Input/output error]
[2016-10-24 10:29:54.119715] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 138: READ => -1
gfid=9061198a-eb7e-45a2-93fb-eb396d1b2727 fd=0x7f70c80d12dc
(Input/output error)
[2016-10-24 10:36:13.140414] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-0:
remote operation failed. Path: (null)
(00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-10-24 10:36:13.140451] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-2:
remote operation failed. Path: (null)
(00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-10-24 10:36:13.140461] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-1:
remote operation failed. Path: (null)
(00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-10-24 10:36:13.140956] E [MSGID: 133010]
[shard.c:1582:shard_common_lookup_shards_cbk] 0-testHeal-shard: Lookup
on shard 1 failed. Base file gfid =
9061198a-eb7e-45a2-93fb-eb396d1b2727 [Input/output error]
[2016-10-24 10:36:13.140995] W [fuse-bridge.c:2227:fuse_readv_cbk]
0-glusterfs-fuse: 145: READ => -1
gfid=9061198a-eb7e-45a2-93fb-eb396d1b2727 fd=0x7f70c80d12dc
(Input/output error)
[2016-10-25 03:22:01.220025] I [MSGID: 100011]
[glusterfsd.c:1323:reincarnate] 0-glusterfsd: Fetching the volume file
from server...
[2016-10-25 03:22:01.220938] I
[glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in
volfile, continuing

I also attached the log in this email.

Thanks.

Cwtan

On Wed, Oct 26, 2016 at 12:30 AM, Krutika Dhananjay <kdhananj at redhat.com> wrote:
> Tried it locally on my setup. Worked fine.
>
> Could you please attach the mount logs?
>
> -Krutika
>
> On Tue, Oct 25, 2016 at 6:55 PM, Pranith Kumar Karampuri
> <pkarampu at redhat.com> wrote:
>>
>> +Krutika
>>
>> On Mon, Oct 24, 2016 at 4:10 PM, qingwei wei <tchengwee at gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I am currently running a simple gluster setup using one server node
>>> with multiple disks. I realize that if i delete away all the .shard
>>> files in one replica in the backend, my application (dd) will report
>>> Input/Output error even though i have 3 replicas.
>>>
>>> My gluster version is 3.7.16
>>>
>>> gluster volume file
>>>
>>> Volume Name: testHeal
>>> Type: Replicate
>>> Volume ID: 26d16d7f-bc4f-44a6-a18b-eab780d80851
>>> Status: Started
>>> Number of Bricks: 1 x 3 = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: 192.168.123.4:/mnt/sdb_mssd/testHeal2
>>> Brick2: 192.168.123.4:/mnt/sde_mssd/testHeal2
>>> Brick3: 192.168.123.4:/mnt/sdd_mssd/testHeal2
>>> Options Reconfigured:
>>> cluster.self-heal-daemon: on
>>> features.shard-block-size: 16MB
>>> features.shard: on
>>> performance.readdir-ahead: on
>>>
>>> dd error
>>>
>>> [root at fujitsu05 .shard]# dd of=/home/test if=/mnt/fuseMount/ddTest
>>> bs=16M count=20 oflag=direct
>>> dd: error reading ‘/mnt/fuseMount/ddTest’: Input/output error
>>> 1+0 records in
>>> 1+0 records out
>>> 16777216 bytes (17 MB) copied, 0.111038 s, 151 MB/s
>>>
>>> in the .shard folder where i deleted all the .shard file, i can see
>>> one .shard file is recreated
>>>
>>> getfattr -d -e hex -m.  9061198a-eb7e-45a2-93fb-eb396d1b2727.1
>>> # file: 9061198a-eb7e-45a2-93fb-eb396d1b2727.1
>>> trusted.afr.testHeal-client-0=0x000000010000000100000000
>>> trusted.afr.testHeal-client-2=0x000000010000000100000000
>>> trusted.gfid=0x41b653f7daa14627b1f91f9e8554ddde
>>>
>>> However, the gfid is not the same compare to the other replicas
>>>
>>> getfattr -d -e hex -m.  9061198a-eb7e-45a2-93fb-eb396d1b2727.1
>>> # file: 9061198a-eb7e-45a2-93fb-eb396d1b2727.1
>>> trusted.afr.dirty=0x000000000000000000000000
>>> trusted.afr.testHeal-client-1=0x000000000000000000000000
>>> trusted.bit-rot.version=0x0300000000000000580dde99000e5e5d
>>> trusted.gfid=0x9ee5c5eed7964a6cb9ac1a1419de5a40
>>>
>>> Is this consider a bug?
>>>
>>> Regards,
>>>
>>> Cwtan
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>
>>
>>
>>
>> --
>> Pranith
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: inputOutputError_shardFileDeleted
Type: application/octet-stream
Size: 12240 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20161026/18d178a8/attachment.obj>