[Gluster-devel] Input/output error when files in .shard folder are deleted

Thu Oct 27 05:12:27 UTC 2016

Hi,

I did few more test runs and it seems that it happens during this sequence

1.populate data using dd
2. delete away ALL the shard files in one of the brick .shard folder
3. Trying to access using dd, no error reported
4. umount and mount.
5. Trying to access using dd, no error reported
6. umount and mount.
7. Trying to access using dd and Input/Output error reported

during step 3 and 4, no file is created under the .shard directory
For step 7, a shard file is created with same file name but different
gfid compare to other good replicas.

Below is the client log and brick log with more details in the attached log.

Client log

[2016-10-27 04:34:46.493281] D [MSGID: 0]
[shard.c:3138:shard_common_mknod_cbk] 0-testHeal4-shard: mknod of
shard 1 failed: File exists
[2016-10-27 04:34:46.493351] D [MSGID: 0]
[dht-common.c:2633:dht_lookup] 0-testHeal4-dht: Calling fresh lookup
for /.shard/76bc4b0f-bb18-4736-8327-99098cd0d7ce.1 on
testHeal4-replicate-0
[2016-10-27 04:34:46.494646] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal4-client-0:
remote operation failed. Path: (null)
(00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-10-27 04:34:46.494673] D [MSGID: 0]
[client-rpc-fops.c:2989:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7f9083edc1c8, testHeal4-client-0 returned -1 error:
Invalid argument [Invalid argument]
[2016-10-27 04:34:46.494705] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal4-client-1:
remote operation failed. Path: (null)
(00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-10-27 04:34:46.494710] W [MSGID: 114031]
[client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal4-client-2:
remote operation failed. Path: (null)
(00000000-0000-0000-0000-000000000000) [Invalid argument]
[2016-10-27 04:34:46.494730] D [MSGID: 0]
[client-rpc-fops.c:2989:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7f9083edc1c8, testHeal4-client-1 returned -1 error:
Invalid argument [Invalid argument]
[2016-10-27 04:34:46.494751] D [MSGID: 0]
[client-rpc-fops.c:2989:client3_3_lookup_cbk] 0-stack-trace:
stack-address: 0x7f9083edc1c8, testHeal4-client-2 returned -1 error:
Invalid argument [Invalid argument]
[2016-10-27 04:34:46.495339] D [MSGID: 0]
[afr-common.c:1986:afr_lookup_done] 0-stack-trace: stack-address:
0x7f9083edbb1c, testHeal4-replicate-0 returned -1 error: Input/output
error [Input/output error]
[2016-10-27 04:34:46.495364] D [MSGID: 0]
[dht-common.c:2220:dht_lookup_cbk] 0-testHeal4-dht: fresh_lookup
returned for /.shard/76bc4b0f-bb18-4736-8327-99098cd0d7ce.1 with
op_ret -1 [Input/output error]
[2016-10-27 04:34:46.495374] D [MSGID: 0]
[dht-common.c:2300:dht_lookup_cbk] 0-testHeal4-dht: Lookup of
/.shard/76bc4b0f-bb18-4736-8327-99098cd0d7ce.1 for subvolume
testHeal4-replicate-0 failed [Input/output error]
[2016-10-27 04:34:46.495384] D [MSGID: 0]
[dht-common.c:2363:dht_lookup_cbk] 0-stack-trace: stack-address:
0x7f9083edbb1c, testHeal4-dht returned -1 error: Input/output error
[Input/output error]
[2016-10-27 04:34:46.495395] E [MSGID: 133010]
[shard.c:1582:shard_common_lookup_shards_cbk] 0-testHeal4-shard:
Lookup on shard 1 failed. Base file gfid =
76bc4b0f-bb18-4736-8327-99098cd0d7ce [Input/output error]
[2016-10-27 04:34:46.495406] D [MSGID: 0]
[shard.c:3086:shard_post_lookup_shards_readv_handler] 0-stack-trace:
stack-address: 0x7f9083edbb1c, testHeal4-shard returned -1 error:
Input/output error [Input/output error]
[2016-10-27 04:34:46.495417] D [MSGID: 0]
[defaults.c:1010:default_readv_cbk] 0-stack-trace: stack-address:
0x7f9083edbb1c, testHeal4-write-behind returned -1 error: Input/output
error [Input/output error]
[2016-10-27 04:34:46.495428] D [MSGID: 0]
[read-ahead.c:462:ra_readv_disabled_cbk] 0-stack-trace: stack-address:
0x7f9083edbb1c, testHeal4-read-ahead returned -1 error: Input/output
error [Input/output error]

brick log

[2016-10-27 04:34:46.492055] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: STATFS
scheduled as fast fop
[2016-10-27 04:34:46.492157] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_entrylk+0x93)
[0x7efebb37d633]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 3
[2016-10-27 04:34:46.492180] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: ENTRYLK
scheduled as normal fop
[2016-10-27 04:34:46.492239] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_statfs_cbk+0x112)
[0x7efebb36f8e2]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.492271] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_entrylk_cbk+0xa6)
[0x7efebb3713a6]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.492535] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_mknod+0x80)
[0x7efebb37b690]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.492565] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: MKNOD
scheduled as normal fop
[2016-10-27 04:34:46.492843] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_mknod_cbk+0x5ad)
[0x7efebb383c9d]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.492981] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_xattrop+0x86)
[0x7efebb3789d6]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.493056] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: XATTROP
scheduled as slow fop
[2016-10-27 04:34:46.493128] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_entrylk+0x93)
[0x7efebb37d633]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 3
[2016-10-27 04:34:46.493148] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: ENTRYLK
scheduled as normal fop
[2016-10-27 04:34:46.493214] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_xattrop_cbk+0xd9)
[0x7efebb370579]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.493239] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_entrylk_cbk+0xa6)
[0x7efebb3713a6]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.493490] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.493514] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: LOOKUP
scheduled as fast fop
[2016-10-27 04:34:46.493666] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.493782] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.493986] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.494596] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.494616] D [logging.c:1954:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-10-27 04:34:46.493818] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: LOOKUP
scheduled as fast fop
[2016-10-27 04:34:46.494616] W [MSGID: 115009]
[server-resolve.c:574:server_resolve] 0-testHeal4-server: no
resolution type for (null) (LOOKUP)
[2016-10-27 04:34:46.494650] E [MSGID: 115050]
[server-rpc-fops.c:179:server_lookup_cbk] 0-testHeal4-server: 29:
LOOKUP (null) (00000000-0000-0000-0000-000000000000/76bc4b0f-bb18-4736-8327-99098cd0d7ce.1)
==> (Invalid argument) [Invalid argument]
[2016-10-27 04:34:46.494720] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.494936] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.494967] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: LOOKUP
scheduled as fast fop
[2016-10-27 04:34:46.495108] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.595813] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.595915] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 3
[2016-10-27 04:34:46.596054] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.596162] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.596427] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_entrylk+0x93)
[0x7efebb37d633]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.596453] D [logging.c:1954:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:351:iot_schedule]
0-testHeal4-io-threads: LOOKUP scheduled as fast fop" repeated 2 times
between [2016-10-27 04:34:46.494967] and [2016-10-27 04:34:46.595944]
[2016-10-27 04:34:46.596453] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: ENTRYLK
scheduled as normal fop
[2016-10-27 04:34:46.596551] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_entrylk+0x93)
[0x7efebb37d633]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 3
[2016-10-27 04:34:46.596603] D [logging.c:1954:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-10-27 04:34:46.596611] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_entrylk_cbk+0xa6)
[0x7efebb3713a6]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.596570] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: ENTRYLK
scheduled as normal fop
[2016-10-27 04:34:46.596602] D [MSGID: 0]
[entrylk.c:701:pl_common_entrylk] 0-stack-trace: stack-address:
0x7efecd3db738, testHeal4-locks returned -1 error: Resource
temporarily unavailable [Resource temporarily unavailable]
[2016-10-27 04:34:46.596650] D [MSGID: 0]
[defaults.c:1196:default_entrylk_cbk] 0-stack-trace: stack-address:
0x7efecd3db738, testHeal4-io-threads returned -1 error: Resource
temporarily unavailable [Resource temporarily unavailable]
[2016-10-27 04:34:46.596664] D [MSGID: 0]
[io-stats.c:1811:io_stats_entrylk_cbk] 0-stack-trace: stack-address:
0x7efecd3db738, /mnt/sdc_mssd/testHeal4 returned -1 error: Resource
temporarily unavailable [Resource temporarily unavailable]
[2016-10-27 04:34:46.596676] D [MSGID: 115054]
[server-rpc-fops.c:350:server_entrylk_cbk] 0-testHeal4-server: 34:
ENTRYLK /.shard (be318638-e8a0-4c6d-977d-7a937aa84806) ==> (Resource
temporarily unavailable) [Resource temporarily unavailable]
[2016-10-27 04:34:46.596764] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_entrylk_cbk+0xa6)
[0x7efebb3713a6]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.596791] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_mknod+0x80)
[0x7efebb37b690]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.596813] D [MSGID: 0]
[server-resolve.c:330:resolve_entry_simple] 0-testHeal4-server: inode
(pointer: 0x7efea1aebaa0 gfid:041e3b34-14c2-4bb1-82e2-db352232c3cf
found for path ((null)) while type is RESOLVE_NOT
[2016-10-27 04:34:46.596828] D [MSGID: 115057]
[server-rpc-fops.c:563:server_mknod_cbk] 0-testHeal4-server: 35: MKNOD
(null) (be318638-e8a0-4c6d-977d-7a937aa84806/76bc4b0f-bb18-4736-8327-99098cd0d7ce.1)
==> (File exists) [File exists]
[2016-10-27 04:34:46.596896] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_mknod_cbk+0x5ad)
[0x7efebb383c9d]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.597174] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_entrylk+0x93)
[0x7efebb37d633]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.597199] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: ENTRYLK
scheduled as normal fop
[2016-10-27 04:34:46.597289] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_entrylk_cbk+0xa6)
[0x7efebb3713a6]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.597396] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_entrylk+0x93)
[0x7efebb37d633]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.597571] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_entrylk_cbk+0xa6)
[0x7efebb3713a6]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.597604] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.597632] D [logging.c:1954:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
[2016-10-27 04:34:46.597415] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: ENTRYLK
scheduled as normal fop
[2016-10-27 04:34:46.597632] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: LOOKUP
scheduled as fast fop
[2016-10-27 04:34:46.597864] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.598116] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_mknod+0x80)
[0x7efebb37b690]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.598134] D [MSGID: 0]
[server-resolve.c:330:resolve_entry_simple] 0-testHeal4-server: inode
(pointer: 0x7efea1aebaa0 gfid:041e3b34-14c2-4bb1-82e2-db352232c3cf
found for path ((null)) while type is RESOLVE_NOT
[2016-10-27 04:34:46.598147] D [MSGID: 115057]
[server-rpc-fops.c:563:server_mknod_cbk] 0-testHeal4-server: 39: MKNOD
(null) (be318638-e8a0-4c6d-977d-7a937aa84806/76bc4b0f-bb18-4736-8327-99098cd0d7ce.1)
==> (File exists) [File exists]
[2016-10-27 04:34:46.598205] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_mknod_cbk+0x5ad)
[0x7efebb383c9d]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.598258] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.598301] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: LOOKUP
scheduled as fast fop
[2016-10-27 04:34:46.598580] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.598599] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_entrylk+0x93)
[0x7efebb37d633]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.598619] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: ENTRYLK
scheduled as normal fop
[2016-10-27 04:34:46.598754] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_entrylk_cbk+0xa6)
[0x7efebb3713a6]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.598921] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.598938] W [MSGID: 115009]
[server-resolve.c:574:server_resolve] 0-testHeal4-server: no
resolution type for (null) (LOOKUP)
[2016-10-27 04:34:46.598951] E [MSGID: 115050]
[server-rpc-fops.c:179:server_lookup_cbk] 0-testHeal4-server: 42:
LOOKUP (null) (00000000-0000-0000-0000-000000000000/76bc4b0f-bb18-4736-8327-99098cd0d7ce.1)
==> (Invalid argument) [Invalid argument]
[2016-10-27 04:34:46.599007] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.599059] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.599081] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: LOOKUP
scheduled as fast fop
[2016-10-27 04:34:46.599215] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 3
[2016-10-27 04:34:46.599379] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.599412] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 3
[2016-10-27 04:34:46.599505] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.599584] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.599783] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.599807] D [logging.c:1954:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:351:iot_schedule]
0-testHeal4-io-threads: LOOKUP scheduled as fast fop" repeated 2 times
between [2016-10-27 04:34:46.599081] and [2016-10-27 04:34:46.599439]
[2016-10-27 04:34:46.599806] W [MSGID: 115009]
[server-resolve.c:574:server_resolve] 0-testHeal4-server: no
resolution type for (null) (LOOKUP)
[2016-10-27 04:34:46.599833] E [MSGID: 115050]
[server-rpc-fops.c:179:server_lookup_cbk] 0-testHeal4-server: 46:
LOOKUP (null) (00000000-0000-0000-0000-000000000000/76bc4b0f-bb18-4736-8327-99098cd0d7ce.1)
==> (Invalid argument) [Invalid argument]
[2016-10-27 04:34:46.599893] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:46.600183] D [client_t.c:333:gf_client_ref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server3_3_lookup+0x8b)
[0x7efebb386beb]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(get_frame_from_request+0x257)
[0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(gf_client_ref+0x68)
[0x7efecfadf608] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 2
[2016-10-27 04:34:46.600206] D [MSGID: 0]
[io-threads.c:351:iot_schedule] 0-testHeal4-io-threads: LOOKUP
scheduled as fast fop
[2016-10-27 04:34:46.600336] D [client_t.c:417:gf_client_unref]
(-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_lookup_cbk+0x548)
[0x7efebb3864c8]
-->/usr/lib64/glusterfs/3.7.16/xlator/protocol/server.so(server_submit_reply+0x123)
[0x7efebb368f13] -->/lib64/libglusterfs.so.0(gf_client_unref+0x77)
[0x7efecfadf787] ) 0-client_t:
fujitsu05.dctopenstack.org-6064-2016/10/27-04:34:44:217958-testHeal4-client-1-0-0:
ref-count 1
[2016-10-27 04:34:59.343124] D
[logging.c:1830:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer
timed out. About to flush outstanding messages if present
[2016-10-27 04:34:59.343202] D
[logging.c:1792:__gf_log_inject_timer_event] 0-logging-infra: Starting
timer now. Timeout = 120, current buf size = 5

Thanks.

Regards,

Cwtan

On Wed, Oct 26, 2016 at 8:09 PM, Krutika Dhananjay <kdhananj at redhat.com> wrote:
> Do you also have the brick logs? Looks like the bricks are returning EINVAL
> on lookup
> which AFR is subsequently converting into an EIO. And sharding is merely
> delivering the same error code upwards.
>
> -Krutika
>
> On Wed, Oct 26, 2016 at 6:38 AM, qingwei wei <tchengwee at gmail.com> wrote:
>>
>> Hi,
>>
>> Pls see the client log below.
>>
>> [2016-10-24 10:29:51.111603] I [fuse-bridge.c:5171:fuse_graph_setup]
>> 0-fuse: switched to graph 0
>> [2016-10-24 10:29:51.111662] I [MSGID: 114035]
>> [client-handshake.c:193:client_set_lk_version_cbk]
>> 0-testHeal-client-2: Server lk version = 1
>> [2016-10-24 10:29:51.112371] I [fuse-bridge.c:4083:fuse_init]
>> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22
>> kernel 7.22
>> [2016-10-24 10:29:51.113563] I [MSGID: 108031]
>> [afr-common.c:2071:afr_local_discovery_cbk] 0-testHeal-replicate-0:
>> selecting local read_child testHeal-client-2
>> [2016-10-24 10:29:51.113604] I [MSGID: 108031]
>> [afr-common.c:2071:afr_local_discovery_cbk] 0-testHeal-replicate-0:
>> selecting local read_child testHeal-client-0
>> [2016-10-24 10:29:51.113630] I [MSGID: 108031]
>> [afr-common.c:2071:afr_local_discovery_cbk] 0-testHeal-replicate-0:
>> selecting local read_child testHeal-client-1
>> [2016-10-24 10:29:54.016802] W [MSGID: 108001]
>> [afr-transaction.c:789:afr_handle_quorum] 0-testHeal-replicate-0:
>> /.shard/9061198a-eb7e-45a2-93fb-eb396d1b2727.1: F
>> ailing MKNOD as quorum is not met
>> [2016-10-24 10:29:54.019330] W [MSGID: 114031]
>> [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-0:
>> remote operation failed. Path: (null) (00000000-
>> 0000-0000-0000-000000000000) [Invalid argument]
>> [2016-10-24 10:29:54.019343] W [MSGID: 114031]
>> [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-2:
>> remote operation failed. Path: (null) (00000000-
>> 0000-0000-0000-000000000000) [Invalid argument]
>> [2016-10-24 10:29:54.019373] W [MSGID: 114031]
>> [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-1:
>> remote operation failed. Path: (null) (00000000-
>> 0000-0000-0000-000000000000) [Invalid argument]
>> [2016-10-24 10:29:54.019854] E [MSGID: 133010]
>> [shard.c:1582:shard_common_lookup_shards_cbk] 0-testHeal-shard: Lookup
>> on shard 1 failed. Base file gfid = 9061198a
>> -eb7e-45a2-93fb-eb396d1b2727 [Input/output error]
>> [2016-10-24 10:29:54.020886] W [fuse-bridge.c:2227:fuse_readv_cbk]
>> 0-glusterfs-fuse: 135: READ => -1
>> gfid=9061198a-eb7e-45a2-93fb-eb396d1b2727 fd=0x7f70c80d12dc (
>> Input/output error)
>> [2016-10-24 10:29:54.118264] W [MSGID: 114031]
>> [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-0:
>> remote operation failed. Path: (null) (00000000-
>> 0000-0000-0000-000000000000) [Invalid argument]
>> [2016-10-24 10:29:54.118308] W [MSGID: 114031]
>> [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-2:
>> remote operation failed. Path: (null)
>> (00000000-0000-0000-0000-000000000000) [Invalid argument]
>> [2016-10-24 10:29:54.118329] W [MSGID: 114031]
>> [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-1:
>> remote operation failed. Path: (null)
>> (00000000-0000-0000-0000-000000000000) [Invalid argument]
>> [2016-10-24 10:29:54.118751] E [MSGID: 133010]
>> [shard.c:1582:shard_common_lookup_shards_cbk] 0-testHeal-shard: Lookup
>> on shard 1 failed. Base file gfid =
>> 9061198a-eb7e-45a2-93fb-eb396d1b2727 [Input/output error]
>> [2016-10-24 10:29:54.118787] W [fuse-bridge.c:2227:fuse_readv_cbk]
>> 0-glusterfs-fuse: 137: READ => -1
>> gfid=9061198a-eb7e-45a2-93fb-eb396d1b2727 fd=0x7f70c80d12dc
>> (Input/output error)
>> [2016-10-24 10:29:54.119330] W [MSGID: 114031]
>> [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-1:
>> remote operation failed. Path: (null)
>> (00000000-0000-0000-0000-000000000000) [Invalid argument]
>> [2016-10-24 10:29:54.119338] W [MSGID: 114031]
>> [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-0:
>> remote operation failed. Path: (null)
>> (00000000-0000-0000-0000-000000000000) [Invalid argument]
>> [2016-10-24 10:29:54.119368] W [MSGID: 114031]
>> [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-2:
>> remote operation failed. Path: (null)
>> (00000000-0000-0000-0000-000000000000) [Invalid argument]
>> [2016-10-24 10:29:54.119674] E [MSGID: 133010]
>> [shard.c:1582:shard_common_lookup_shards_cbk] 0-testHeal-shard: Lookup
>> on shard 1 failed. Base file gfid =
>> 9061198a-eb7e-45a2-93fb-eb396d1b2727 [Input/output error]
>> [2016-10-24 10:29:54.119715] W [fuse-bridge.c:2227:fuse_readv_cbk]
>> 0-glusterfs-fuse: 138: READ => -1
>> gfid=9061198a-eb7e-45a2-93fb-eb396d1b2727 fd=0x7f70c80d12dc
>> (Input/output error)
>> [2016-10-24 10:36:13.140414] W [MSGID: 114031]
>> [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-0:
>> remote operation failed. Path: (null)
>> (00000000-0000-0000-0000-000000000000) [Invalid argument]
>> [2016-10-24 10:36:13.140451] W [MSGID: 114031]
>> [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-2:
>> remote operation failed. Path: (null)
>> (00000000-0000-0000-0000-000000000000) [Invalid argument]
>> [2016-10-24 10:36:13.140461] W [MSGID: 114031]
>> [client-rpc-fops.c:2981:client3_3_lookup_cbk] 0-testHeal-client-1:
>> remote operation failed. Path: (null)
>> (00000000-0000-0000-0000-000000000000) [Invalid argument]
>> [2016-10-24 10:36:13.140956] E [MSGID: 133010]
>> [shard.c:1582:shard_common_lookup_shards_cbk] 0-testHeal-shard: Lookup
>> on shard 1 failed. Base file gfid =
>> 9061198a-eb7e-45a2-93fb-eb396d1b2727 [Input/output error]
>> [2016-10-24 10:36:13.140995] W [fuse-bridge.c:2227:fuse_readv_cbk]
>> 0-glusterfs-fuse: 145: READ => -1
>> gfid=9061198a-eb7e-45a2-93fb-eb396d1b2727 fd=0x7f70c80d12dc
>> (Input/output error)
>> [2016-10-25 03:22:01.220025] I [MSGID: 100011]
>> [glusterfsd.c:1323:reincarnate] 0-glusterfsd: Fetching the volume file
>> from server...
>> [2016-10-25 03:22:01.220938] I
>> [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in
>> volfile, continuing
>>
>> I also attached the log in this email.
>>
>> Thanks.
>>
>> Cwtan
>>
>>
>> On Wed, Oct 26, 2016 at 12:30 AM, Krutika Dhananjay <kdhananj at redhat.com>
>> wrote:
>> > Tried it locally on my setup. Worked fine.
>> >
>> > Could you please attach the mount logs?
>> >
>> > -Krutika
>> >
>> > On Tue, Oct 25, 2016 at 6:55 PM, Pranith Kumar Karampuri
>> > <pkarampu at redhat.com> wrote:
>> >>
>> >> +Krutika
>> >>
>> >> On Mon, Oct 24, 2016 at 4:10 PM, qingwei wei <tchengwee at gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I am currently running a simple gluster setup using one server node
>> >>> with multiple disks. I realize that if i delete away all the .shard
>> >>> files in one replica in the backend, my application (dd) will report
>> >>> Input/Output error even though i have 3 replicas.
>> >>>
>> >>> My gluster version is 3.7.16
>> >>>
>> >>> gluster volume file
>> >>>
>> >>> Volume Name: testHeal
>> >>> Type: Replicate
>> >>> Volume ID: 26d16d7f-bc4f-44a6-a18b-eab780d80851
>> >>> Status: Started
>> >>> Number of Bricks: 1 x 3 = 3
>> >>> Transport-type: tcp
>> >>> Bricks:
>> >>> Brick1: 192.168.123.4:/mnt/sdb_mssd/testHeal2
>> >>> Brick2: 192.168.123.4:/mnt/sde_mssd/testHeal2
>> >>> Brick3: 192.168.123.4:/mnt/sdd_mssd/testHeal2
>> >>> Options Reconfigured:
>> >>> cluster.self-heal-daemon: on
>> >>> features.shard-block-size: 16MB
>> >>> features.shard: on
>> >>> performance.readdir-ahead: on
>> >>>
>> >>> dd error
>> >>>
>> >>> [root at fujitsu05 .shard]# dd of=/home/test if=/mnt/fuseMount/ddTest
>> >>> bs=16M count=20 oflag=direct
>> >>> dd: error reading ‘/mnt/fuseMount/ddTest’: Input/output error
>> >>> 1+0 records in
>> >>> 1+0 records out
>> >>> 16777216 bytes (17 MB) copied, 0.111038 s, 151 MB/s
>> >>>
>> >>> in the .shard folder where i deleted all the .shard file, i can see
>> >>> one .shard file is recreated
>> >>>
>> >>> getfattr -d -e hex -m.  9061198a-eb7e-45a2-93fb-eb396d1b2727.1
>> >>> # file: 9061198a-eb7e-45a2-93fb-eb396d1b2727.1
>> >>> trusted.afr.testHeal-client-0=0x000000010000000100000000
>> >>> trusted.afr.testHeal-client-2=0x000000010000000100000000
>> >>> trusted.gfid=0x41b653f7daa14627b1f91f9e8554ddde
>> >>>
>> >>> However, the gfid is not the same compare to the other replicas
>> >>>
>> >>> getfattr -d -e hex -m.  9061198a-eb7e-45a2-93fb-eb396d1b2727.1
>> >>> # file: 9061198a-eb7e-45a2-93fb-eb396d1b2727.1
>> >>> trusted.afr.dirty=0x000000000000000000000000
>> >>> trusted.afr.testHeal-client-1=0x000000000000000000000000
>> >>> trusted.bit-rot.version=0x0300000000000000580dde99000e5e5d
>> >>> trusted.gfid=0x9ee5c5eed7964a6cb9ac1a1419de5a40
>> >>>
>> >>> Is this consider a bug?
>> >>>
>> >>> Regards,
>> >>>
>> >>> Cwtan
>> >>> _______________________________________________
>> >>> Gluster-devel mailing list
>> >>> Gluster-devel at gluster.org
>> >>> http://www.gluster.org/mailman/listinfo/gluster-devel
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Pranith
>> >
>> >
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mnt-sdc_mssd-testHeal4.zip
Type: application/zip
Size: 154418 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20161027/c773ae4a/attachment-0002.zip>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mnt-testHeal4.zip
Type: application/zip
Size: 202431 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20161027/c773ae4a/attachment-0003.zip>