<div dir="ltr">hi Walter,<div><br></div><div>I don't see EIO errors in the log snippet and the messages look fine so far. Can you send across the rest of the log where you see the EIO?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 20 September 2017 at 23:57, Walter Deignan <span dir="ltr"><<a href="mailto:WDeignan@uline.com" target="_blank">WDeignan@uline.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font size="3" face="Arial">I put the share into debug mode and then repeated
the process from a ppc64 client and an x86 client. Weirdly the client logs
were almost identical.</font>
<br>
<br><font size="3" face="Arial">Here's the ppc64 gluster client log of attempting
to create a folder...</font>
<br>
<br><font size="3" face="Arial">-------------</font>
<br>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.344321] D [rpc-clnt-ping.c:93:rpc_clnt_<wbr>remove_ping_timer_locked]
(--> /usr/lib64/libglusterfs.so.0(_<wbr>gf_log_callingfn-0xfdf60)[<wbr>0x3fff9ec56fe0]
(--> /usr/lib64/libgfrpc.so.0(rpc_<wbr>clnt_remove_ping_timer_locked-<wbr>0x26060)[0x3fff9ebd9e20]
(--> /usr/lib64/libgfrpc.so.0(+<wbr>0x1a614)[0x3fff9ebda614] (--> /usr/lib64/libgfrpc.so.0(rpc_<wbr>clnt_submit-0x29300)[<wbr>0x3fff9ebd69b0]
(--> /usr/lib64/glusterfs/3.10.5/<wbr>xlator/protocol/client.so(+<wbr>0x182e0)[0x3fff939182e0]
))))) 0-: <a href="http://10.50.80.102:49152" target="_blank">10.50.80.102:49152</a>: ping timer event already removed</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.345149] D [rpc-clnt-ping.c:93:rpc_clnt_<wbr>remove_ping_timer_locked]
(--> /usr/lib64/libglusterfs.so.0(_<wbr>gf_log_callingfn-0xfdf60)[<wbr>0x3fff9ec56fe0]
(--> /usr/lib64/libgfrpc.so.0(rpc_<wbr>clnt_remove_ping_timer_locked-<wbr>0x26060)[0x3fff9ebd9e20]
(--> /usr/lib64/libgfrpc.so.0(+<wbr>0x1a614)[0x3fff9ebda614] (--> /usr/lib64/libgfrpc.so.0(rpc_<wbr>clnt_submit-0x29300)[<wbr>0x3fff9ebd69b0]
(--> /usr/lib64/glusterfs/3.10.5/<wbr>xlator/protocol/client.so(+<wbr>0x182e0)[0x3fff939182e0]
))))) 0-: <a href="http://10.50.80.103:49152" target="_blank">10.50.80.103:49152</a>: ping timer event already removed</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.345977] D [rpc-clnt-ping.c:93:rpc_clnt_<wbr>remove_ping_timer_locked]
(--> /usr/lib64/libglusterfs.so.0(_<wbr>gf_log_callingfn-0xfdf60)[<wbr>0x3fff9ec56fe0]
(--> /usr/lib64/libgfrpc.so.0(rpc_<wbr>clnt_remove_ping_timer_locked-<wbr>0x26060)[0x3fff9ebd9e20]
(--> /usr/lib64/libgfrpc.so.0(+<wbr>0x1a614)[0x3fff9ebda614] (--> /usr/lib64/libgfrpc.so.0(rpc_<wbr>clnt_submit-0x29300)[<wbr>0x3fff9ebd69b0]
(--> /usr/lib64/glusterfs/3.10.5/<wbr>xlator/protocol/client.so(+<wbr>0x182e0)[0x3fff939182e0]
))))) 0-: <a href="http://10.50.80.104:49152" target="_blank">10.50.80.104:49152</a>: ping timer event already removed</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.346070] D [MSGID:
0] [dht-common.c:1002:dht_<wbr>revalidate_cbk] 0-gv0-dht: revalidate lookup
of / returned with op_ret 0 [Structure needs cleaning]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.347612] D [MSGID:
0] [dht-common.c:2699:dht_lookup] 0-gv0-dht: Calling fresh lookup for /tempdir3
on gv0-replicate-0</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348013] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0-client-1 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348013] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0-client-0 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348083] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0-client-2 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348132] D [MSGID:
0] [afr-common.c:2264:afr_lookup_<wbr>done] 0-stack-trace: stack-address: 0x3fff88001080,
gv0-replicate-0 returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348166] D [MSGID:
0] [dht-common.c:2284:dht_lookup_<wbr>cbk] 0-gv0-dht: fresh_lookup returned
for /tempdir3 with op_ret -1 [No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348195] D [MSGID:
0] [dht-common.c:2297:dht_lookup_<wbr>cbk] 0-gv0-dht: Entry /tempdir3 missing
on subvol gv0-replicate-0</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348220] D [MSGID:
0] [dht-common.c:2068:dht_lookup_<wbr>everywhere] 0-gv0-dht: winding lookup
call to 1 subvols</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348551] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0-client-1 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348551] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0-client-0 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348613] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0-client-2 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348639] D [MSGID:
0] [afr-common.c:2264:afr_lookup_<wbr>done] 0-stack-trace: stack-address: 0x3fff88001080,
gv0-replicate-0 returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348665] D [MSGID:
0] [dht-common.c:1870:dht_lookup_<wbr>everywhere_cbk] 0-gv0-dht: returned with
op_ret -1 and op_errno 2 (/tempdir3) from subvol gv0-replicate-0</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348697] D [MSGID:
0] [dht-common.c:1535:dht_lookup_<wbr>everywhere_done] 0-gv0-dht: STATUS: hashed_subvol
gv0-replicate-0 cached_subvol null</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348740] D [MSGID:
0] [dht-common.c:1596:dht_lookup_<wbr>everywhere_done] 0-gv0-dht: There was
no cached file and unlink on hashed is not skipped /tempdir3</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348783] D [MSGID:
0] [dht-common.c:1599:dht_lookup_<wbr>everywhere_done] 0-stack-trace: stack-address:
0x3fff88001080, gv0-dht returned -1 error: No such file or directory [No
such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348817] D [MSGID:
0] [write-behind.c:2392:wb_<wbr>lookup_cbk] 0-stack-trace: stack-address: 0x3fff88001080,
gv0-write-behind returned -1 error: No such file or directory [No such
file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348845] D [MSGID:
0] [io-cache.c:267:ioc_lookup_<wbr>cbk] 0-stack-trace: stack-address: 0x3fff88001080,
gv0-io-cache returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348875] D [MSGID:
0] [quick-read.c:447:qr_lookup_<wbr>cbk] 0-stack-trace: stack-address: 0x3fff88001080,
gv0-quick-read returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348906] D [MSGID:
0] [md-cache.c:1048:mdc_lookup_<wbr>cbk] 0-stack-trace: stack-address: 0x3fff88001080,
gv0-md-cache returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348955] D [MSGID:
0] [io-stats.c:2186:io_stats_<wbr>lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0 returned -1 error: No such file or directory [No such
file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.348994] D [fuse-resolve.c:61:fuse_<wbr>resolve_entry_cbk]
0-fuse: 00000000-0000-0000-0000-<wbr>000000000001/tempdir3: failed to resolve
(No such file or directory)</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.349589] D [MSGID:
0] [dht-common.c:2699:dht_lookup] 0-gv0-dht: Calling fresh lookup for /tempdir3
on gv0-replicate-0</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.349917] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff8c063900, gv0-client-1 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.349961] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff8c063900, gv0-client-2 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350055] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff8c063900, gv0-client-0 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350076] D [MSGID:
0] [afr-common.c:2264:afr_lookup_<wbr>done] 0-stack-trace: stack-address: 0x3fff8c063900,
gv0-replicate-0 returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350099] D [MSGID:
0] [dht-common.c:2284:dht_lookup_<wbr>cbk] 0-gv0-dht: fresh_lookup returned
for /tempdir3 with op_ret -1 [No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350124] D [MSGID:
0] [dht-common.c:2297:dht_lookup_<wbr>cbk] 0-gv0-dht: Entry /tempdir3 missing
on subvol gv0-replicate-0</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350146] D [MSGID:
0] [dht-common.c:2068:dht_lookup_<wbr>everywhere] 0-gv0-dht: winding lookup
call to 1 subvols</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350473] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff8c063900, gv0-client-1 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350473] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff8c063900, gv0-client-0 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350528] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff8c063900, gv0-client-2 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350553] D [MSGID:
0] [afr-common.c:2264:afr_lookup_<wbr>done] 0-stack-trace: stack-address: 0x3fff8c063900,
gv0-replicate-0 returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350583] D [MSGID:
0] [dht-common.c:1870:dht_lookup_<wbr>everywhere_cbk] 0-gv0-dht: returned with
op_ret -1 and op_errno 2 (/tempdir3) from subvol gv0-replicate-0</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350610] D [MSGID:
0] [dht-common.c:1535:dht_lookup_<wbr>everywhere_done] 0-gv0-dht: STATUS: hashed_subvol
gv0-replicate-0 cached_subvol null</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350635] D [MSGID:
0] [dht-common.c:1596:dht_lookup_<wbr>everywhere_done] 0-gv0-dht: There was
no cached file and unlink on hashed is not skipped /tempdir3</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350660] D [MSGID:
0] [dht-common.c:1599:dht_lookup_<wbr>everywhere_done] 0-stack-trace: stack-address:
0x3fff8c063900, gv0-dht returned -1 error: No such file or directory [No
such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350693] D [MSGID:
0] [write-behind.c:2392:wb_<wbr>lookup_cbk] 0-stack-trace: stack-address: 0x3fff8c063900,
gv0-write-behind returned -1 error: No such file or directory [No such
file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350724] D [MSGID:
0] [io-cache.c:267:ioc_lookup_<wbr>cbk] 0-stack-trace: stack-address: 0x3fff8c063900,
gv0-io-cache returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350756] D [MSGID:
0] [quick-read.c:447:qr_lookup_<wbr>cbk] 0-stack-trace: stack-address: 0x3fff8c063900,
gv0-quick-read returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350790] D [MSGID:
0] [md-cache.c:1048:mdc_lookup_<wbr>cbk] 0-stack-trace: stack-address: 0x3fff8c063900,
gv0-md-cache returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.350818] D [MSGID:
0] [io-stats.c:2186:io_stats_<wbr>lookup_cbk] 0-stack-trace: stack-address:
0x3fff8c063900, gv0 returned -1 error: No such file or directory [No such
file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.351228] D [MSGID:
0] [dht-common.c:2699:dht_lookup] 0-gv0-dht: Calling fresh lookup for /tempdir3
on gv0-replicate-0</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.351573] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0-client-1 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.351573] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0-client-0 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.351637] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0-client-2 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.351664] D [MSGID:
0] [afr-common.c:2264:afr_lookup_<wbr>done] 0-stack-trace: stack-address: 0x3fff88001080,
gv0-replicate-0 returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.351688] D [MSGID:
0] [dht-common.c:2284:dht_lookup_<wbr>cbk] 0-gv0-dht: fresh_lookup returned
for /tempdir3 with op_ret -1 [No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.351715] D [MSGID:
0] [dht-common.c:2297:dht_lookup_<wbr>cbk] 0-gv0-dht: Entry /tempdir3 missing
on subvol gv0-replicate-0</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.351738] D [MSGID:
0] [dht-common.c:2068:dht_lookup_<wbr>everywhere] 0-gv0-dht: winding lookup
call to 1 subvols</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352069] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0-client-0 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352074] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0-client-1 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352120] D [MSGID:
0] [client-rpc-fops.c:2936:<wbr>client3_3_lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0-client-2 returned -1 error: No such file or directory
[No such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352145] D [MSGID:
0] [afr-common.c:2264:afr_lookup_<wbr>done] 0-stack-trace: stack-address: 0x3fff88001080,
gv0-replicate-0 returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352169] D [MSGID:
0] [dht-common.c:1870:dht_lookup_<wbr>everywhere_cbk] 0-gv0-dht: returned with
op_ret -1 and op_errno 2 (/tempdir3) from subvol gv0-replicate-0</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352192] D [MSGID:
0] [dht-common.c:1535:dht_lookup_<wbr>everywhere_done] 0-gv0-dht: STATUS: hashed_subvol
gv0-replicate-0 cached_subvol null</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352212] D [MSGID:
0] [dht-common.c:1596:dht_lookup_<wbr>everywhere_done] 0-gv0-dht: There was
no cached file and unlink on hashed is not skipped /tempdir3</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352231] D [MSGID:
0] [dht-common.c:1599:dht_lookup_<wbr>everywhere_done] 0-stack-trace: stack-address:
0x3fff88001080, gv0-dht returned -1 error: No such file or directory [No
such file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352264] D [MSGID:
0] [write-behind.c:2392:wb_<wbr>lookup_cbk] 0-stack-trace: stack-address: 0x3fff88001080,
gv0-write-behind returned -1 error: No such file or directory [No such
file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352296] D [MSGID:
0] [io-cache.c:267:ioc_lookup_<wbr>cbk] 0-stack-trace: stack-address: 0x3fff88001080,
gv0-io-cache returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352326] D [MSGID:
0] [quick-read.c:447:qr_lookup_<wbr>cbk] 0-stack-trace: stack-address: 0x3fff88001080,
gv0-quick-read returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352361] D [MSGID:
0] [md-cache.c:1048:mdc_lookup_<wbr>cbk] 0-stack-trace: stack-address: 0x3fff88001080,
gv0-md-cache returned -1 error: No such file or directory [No such file
or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352392] D [MSGID:
0] [io-stats.c:2186:io_stats_<wbr>lookup_cbk] 0-stack-trace: stack-address:
0x3fff88001080, gv0 returned -1 error: No such file or directory [No such
file or directory]</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352424] D [fuse-resolve.c:61:fuse_<wbr>resolve_entry_cbk]
0-fuse: 00000000-0000-0000-0000-<wbr>000000000001/tempdir3: failed to resolve
(No such file or directory)</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.352749] D [MSGID:
0] [dht-diskusage.c:96:dht_du_<wbr>info_cbk] 0-gv0-dht: subvolume 'gv0-replicate-0':
avail_percent is: 99.00 and avail_space is: 21425758208 and avail_inodes
is: 99.00</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.353086] D [MSGID:
0] [afr-transaction.c:1934:afr_<wbr>post_nonblocking_entrylk_cbk] 0-gv0-replicate-0:
Non blocking entrylks done. Proceeding to FOP</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.353722] D [MSGID:
0] [dht-selfheal.c:1879:dht_<wbr>selfheal_layout_new_directory] 0-gv0-dht: chunk
size = 0xffffffff / 20466 = 209858.658018</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.353748] D [MSGID:
0] [dht-selfheal.c:1920:dht_<wbr>selfheal_layout_new_directory] 0-gv0-dht: assigning
range size 0xffffffff to gv0-replicate-0</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.353897] D [MSGID:
0] [afr-lk-common.c:448:<wbr>transaction_lk_op] 0-gv0-replicate-0: lk op is
for a transaction</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.354052] D [MSGID:
0] [afr-transaction.c:1883:afr_<wbr>post_nonblocking_inodelk_cbk] 0-gv0-replicate-0:
Non blocking inodelks done. Proceeding to FOP</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.354453] D [MSGID:
0] [afr-lk-common.c:448:<wbr>transaction_lk_op] 0-gv0-replicate-0: lk op is
for a transaction</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.354969] D [MSGID:
109036] [dht-common.c:9527:dht_log_<wbr>new_layout_for_dir_selfheal] 0-gv0-dht:
Setting layout of /tempdir3 with [Subvol_name: gv0-replicate-0, Err: -1
, Start: 0 , Stop: 4294967295 , Hash: 1 ],</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.355226] D [MSGID:
0] [afr-transaction.c:1883:afr_<wbr>post_nonblocking_inodelk_cbk] 0-gv0-replicate-0:
Non blocking inodelks done. Proceeding to FOP</font>
<br><font size="2" face="Consolas">[2017-09-20 13:34:23.355714] D [MSGID:
0] [afr-lk-common.c:448:<wbr>transaction_lk_op] 0-gv0-replicate-0: lk op is
for a transaction</font>
<br><span class="">
<br><font size="3" face="Arial">-Walter Deignan<br>
-Uline IT, Systems Architect</font>
<br>
<br>
<br>
<br></span><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">Amar Tumballi <<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">Walter Deignan <<a href="mailto:WDeignan@uline.com" target="_blank">WDeignan@uline.com</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Cc:
</font><font size="1" face="sans-serif">"<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>
List" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">09/20/2017 01:23 PM</font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">Re: [Gluster-users]
"Input/output error" on mkdir for PPC64 based client</font>
<br>
<hr noshade><div class="HOEnZb"><div class="h5">
<br>
<br>
<br><font size="3">Looks like it is an issue with architecture compatibility
in RPC layer (ie, with XDRs and how it is used). Just glance the logs of
the client process where you saw the errors, which could give some hints.
If you don't understand the logs, share them, so we will try to look into
it.</font>
<br>
<br><font size="3">-Amar</font>
<br>
<br><font size="3">On Wed, Sep 20, 2017 at 2:40 AM, Walter Deignan <</font><a href="mailto:WDeignan@uline.com" target="_blank"><font size="3" color="blue"><u>WDeignan@uline.com</u></font></a><font size="3">>
wrote:</font>
<br><font size="3" face="Arial">I recently compiled the 3.10-5 client from
source on a few PPC64 systems running RHEL 7.3. They are mounting a Gluster
volume which is hosted on more traditional x86 servers.</font><font size="3">
<br>
</font><font size="3" face="Arial"><br>
Everything seems to be working properly except for creating new directories
from the PPC64 clients. The mkdir command gives a "Input/output error"
and for the first few minutes the new directory is inaccessible. I checked
the backend bricks and confirmed the directory was created properly on
all of them. After waiting for 2-5 minutes the directory magically becomes
accessible.</font><font size="3"> <br>
</font><font size="3" face="Arial"><br>
This inaccessible directory issue only appears from the client which created
it. When creating the directory from client #1 I can immediately see it
with no errors from client #2.</font><font size="3"> <br>
</font><font size="3" face="Arial"><br>
Using a pre-compiled 3.10-5 package on an x86 client doesn't show the issue.</font><font size="3">
<br>
</font><font size="3" face="Arial"><br>
I poked around bugzilla but couldn't seem to find anything which matches
this.</font><font size="3"> <br>
</font><font size="2" face="Consolas"><br>
[root@mqdev1 hafsdev1_gv0]# ls -lh</font><font size="3"> </font><font size="2" face="Consolas"><br>
total 8.0K</font><font size="3"> </font><font size="2" face="Consolas"><br>
drwxrwxr-x. 4 mqm mqm 4.0K Sep 19 15:47 data</font><font size="3">
</font><font size="2" face="Consolas"><br>
drwxr-xr-x. 2 root root 4.0K Sep 19 15:47 testdir</font><font size="3"> </font><font size="2" face="Consolas"><br>
[root@mqdev1 hafsdev1_gv0]# mkdir testdir2</font><font size="3"> </font><font size="2" face="Consolas"><br>
mkdir: cannot create directory ‘testdir2’: Input/output error</font><font size="3">
</font><font size="2" face="Consolas"><br>
[root@mqdev1 hafsdev1_gv0]# ls</font><font size="3"> </font><font size="2" face="Consolas"><br>
ls: cannot access testdir2: No such file or directory</font><font size="3">
</font><font size="2" face="Consolas"><br>
data testdir testdir2</font><font size="3"> </font><font size="2" face="Consolas"><br>
[root@mqdev1 hafsdev1_gv0]# ls -lht</font><font size="3"> </font><font size="2" face="Consolas"><br>
ls: cannot access testdir2: No such file or directory</font><font size="3">
</font><font size="2" face="Consolas"><br>
total 8.0K</font><font size="3"> </font><font size="2" face="Consolas"><br>
drwxr-xr-x. 2 root root 4.0K Sep 19 15:47 testdir</font><font size="3"> </font><font size="2" face="Consolas"><br>
drwxrwxr-x. 4 mqm mqm 4.0K Sep 19 15:47 data</font><font size="3">
</font><font size="2" face="Consolas"><br>
d?????????? ? ? ? ?
? testdir2</font><font size="3"> </font><font size="2" face="Consolas"><br>
[root@mqdev1 hafsdev1_gv0]# cd testdir2</font><font size="3"> </font><font size="2" face="Consolas"><br>
-bash: cd: testdir2: No such file or directory</font><font size="3"> <br>
</font><font size="2" face="Consolas"><br>
*Wait a few minutes...*</font><font size="3"> <br>
</font><font size="2" face="Consolas"><br>
[root@mqdev1 hafsdev1_gv0]# ls -lht</font><font size="3"> </font><font size="2" face="Consolas"><br>
total 12K</font><font size="3"> </font><font size="2" face="Consolas"><br>
drwxr-xr-x. 2 root root 4.0K Sep 19 15:50 testdir2</font><font size="3">
</font><font size="2" face="Consolas"><br>
drwxr-xr-x. 2 root root 4.0K Sep 19 15:47 testdir</font><font size="3"> </font><font size="2" face="Consolas"><br>
drwxrwxr-x. 4 mqm mqm 4.0K Sep 19 15:47 data</font><font size="3">
</font><font size="2" face="Consolas"><br>
[root@mqdev1 hafsdev1_gv0]#</font><font size="3"> <br>
</font><font size="3" face="Arial"><br>
My volume config...</font><font size="3"> <br>
</font><font size="2" face="Consolas"><br>
[root@dc-hafsdev1a bricks]# gluster volume info</font><font size="3"> <br>
</font><font size="2" face="Consolas"><br>
Volume Name: gv0</font><font size="3"> </font><font size="2" face="Consolas"><br>
Type: Replicate</font><font size="3"> </font><font size="2" face="Consolas"><br>
Volume ID: a2d37705-05cb-4700-8ed8-<wbr>2cb89376faf0</font><font size="3"> </font><font size="2" face="Consolas"><br>
Status: Started</font><font size="3"> </font><font size="2" face="Consolas"><br>
Snapshot Count: 0</font><font size="3"> </font><font size="2" face="Consolas"><br>
Number of Bricks: 1 x 3 = 3</font><font size="3"> </font><font size="2" face="Consolas"><br>
Transport-type: tcp</font><font size="3"> </font><font size="2" face="Consolas"><br>
Bricks:</font><font size="3"> </font><font size="2" face="Consolas"><br>
Brick1: dc-hafsdev1a.ulinedm.com:/<wbr>gluster/bricks/brick1/data</font><font size="3">
</font><font size="2" face="Consolas"><br>
Brick2: dc-hafsdev1b.ulinedm.com:/<wbr>gluster/bricks/brick1/data</font><font size="3">
</font><font size="2" face="Consolas"><br>
Brick3: dc-hafsdev1c.ulinedm.com:/<wbr>gluster/bricks/brick1/data</font><font size="3">
</font><font size="2" face="Consolas"><br>
Options Reconfigured:</font><font size="3"> </font><font size="2" face="Consolas"><br>
nfs.disable: on</font><font size="3"> </font><font size="2" face="Consolas"><br>
transport.address-family: inet</font><font size="3"> </font><font size="2" face="Consolas"><br>
network.ping-timeout: 2</font><font size="3"> </font><font size="2" face="Consolas"><br>
features.bitrot: on</font><font size="3"> </font><font size="2" face="Consolas"><br>
features.scrub: Active</font><font size="3"> </font><font size="2" face="Consolas"><br>
cluster.server-quorum-ratio: 51%</font><font size="3"> </font><font size="3" color="#8f8f8f"><br>
</font><font size="3" color="#8f8f8f" face="Arial"><br>
-Walter Deignan<br>
-Uline IT, Systems Architect</font><font size="3"><br>
______________________________<wbr>_________________<br>
Gluster-users mailing list</font><font size="3" color="blue"><u><br>
</u></font><a href="mailto:Gluster-users@gluster.org" target="_blank"><font size="3" color="blue"><u>Gluster-users@gluster.org</u></font></a><font size="3" color="blue"><u><br>
</u></font><a href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank"><font size="3" color="blue"><u>http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</u></font></a>
<br><font size="3"><br>
</font>
<br>
<br><font size="3">-- </font>
<br><font size="3">Amar Tumballi (amarts)</font>
<br>
<br></div></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>