<html><head></head><body>I have rdma capability. Will test and report back. I'm still on v 3.12.<br><br><div class="gmail_quote">On January 24, 2019 12:54:26 AM EST, Amar Tumballi Suryanarayan <atumball@redhat.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div dir="ltr">I suspect this is a bug with 'Transport: rdma' part. We have called out for de-scoping that feature as we are lacking experts in that domain right now. Recommend you to use IPoIB option, and use tcp/socket transport type (which is default). That should mostly fix all the issues.<div><br></div><div>-Amar</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 24, 2019 at 5:31 AM Jim Kinney <<a href="mailto:jim.kinney@gmail.com">jim.kinney@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="text-align:left;direction:ltr"><div>That really sounds like a bug with the sharding. I'm not using sharding on my setup and files are writeable (vim) with 2 bytes and no errors occur.</div><div>Perhaps the small size is cached until it's large enough to trigger a write</div><div><br></div><div>On Wed, 2019-01-23 at 21:46 -0200, Lindolfo Meira wrote:</div><blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:2px solid rgb(114,159,207);padding-left:1ex"><pre>Also I noticed that any subsequent write (after the first write with 340 </pre><pre>bytes or more), regardless the size, will work as expected.</pre><pre><br></pre><pre><br></pre><pre>Lindolfo Meira, MSc</pre><pre>Diretor Geral, Centro Nacional de Supercomputação</pre><pre>Universidade Federal do Rio Grande do Sul</pre><pre>+55 (51) 3308-3139</pre><pre><br></pre><pre>On Wed, 23 Jan 2019, Lindolfo Meira wrote:</pre><pre><br></pre><blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:2px solid rgb(114,159,207);padding-left:1ex"><pre>Just checked: when the write is >= 340 bytes, everything works as </pre><pre>supposed. If the write is smaller, the error takes place. And when it </pre><pre>does, nothing is logged on the server. The client, however, logs the </pre><pre>following:</pre><pre><br></pre><pre>[2019-01-23 23:28:54.554664] W [MSGID: 103046] </pre><pre>[rdma.c:3502:gf_rdma_decode_header] 0-rpc-transport/rdma: received a msg </pre><pre>of type RDMA_ERROR</pre><pre><br></pre><pre>[2019-01-23 23:28:54.554728] W [MSGID: 103046] </pre><pre>[rdma.c:3939:gf_rdma_process_recv] 0-rpc-transport/rdma: peer </pre><pre>(<a href="http://172.24.1.6:49152" target="_blank">172.24.1.6:49152</a>), couldn't encode or decode the msg properly or write </pre><pre>chunks were not provided for replies that were bigger than </pre><pre>RDMA_INLINE_THRESHOLD (2048)</pre><pre><br></pre><pre>[2019-01-23 23:28:54.554765] W [MSGID: 114031] </pre><pre>[client-rpc-fops_v2.c:680:client4_0_writev_cbk] 0-gfs-client-5: remote </pre><pre>operation failed [Transport endpoint is not connected]</pre><pre><br></pre><pre>[2019-01-23 23:28:54.554850] W [fuse-bridge.c:1436:fuse_err_cbk] </pre><pre>0-glusterfs-fuse: 1723199: FLUSH() ERR => -1 (Transport endpoint is not </pre><pre>connected)</pre><pre><br></pre><pre><br></pre><pre><br></pre><pre>Lindolfo Meira, MSc</pre><pre>Diretor Geral, Centro Nacional de Supercomputação</pre><pre>Universidade Federal do Rio Grande do Sul</pre><pre>+55 (51) 3308-3139</pre><pre><br></pre><pre>On Wed, 23 Jan 2019, Lindolfo Meira wrote:</pre><pre><br></pre><blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:2px solid rgb(114,159,207);padding-left:1ex"><pre>Hi Jim. Thanks for taking the time.</pre><pre><br></pre><pre>Sorry I didn't express myself properly. It's not a simple matter of </pre><pre>permissions. Users can write to the volume alright. It's when vim and nano </pre><pre>are used, or when small file writes are performed (by cat or echo), that </pre><pre>it doesn't work. The file is updated with the write in the server, but it </pre><pre>shows up as empty in the client.</pre><pre><br></pre><pre>I guess it has something to do with the size of the write, because I ran a </pre><pre>test writing to a file one byte at a time, and it never showed up as </pre><pre>having any content in the client (although in the server it kept growing </pre><pre>accordingly).</pre><pre><br></pre><pre>I should point out that I'm using a sharded volume. But when I was testing </pre><pre>a striped volume, it also happened. Output of "gluster volume info" </pre><pre>follows bellow:</pre><pre><br></pre><pre>Volume Name: gfs</pre><pre>Type: Distribute</pre><pre>Volume ID: b5ef065f-1ba2-481f-8108-e8f6d2d3f036</pre><pre>Status: Started</pre><pre>Snapshot Count: 0</pre><pre>Number of Bricks: 6</pre><pre>Transport-type: rdma</pre><pre>Bricks:</pre><pre>Brick1: pfs01-ib:/mnt/data</pre><pre>Brick2: pfs02-ib:/mnt/data</pre><pre>Brick3: pfs03-ib:/mnt/data</pre><pre>Brick4: pfs04-ib:/mnt/data</pre><pre>Brick5: pfs05-ib:/mnt/data</pre><pre>Brick6: pfs06-ib:/mnt/data</pre><pre>Options Reconfigured:</pre><pre>nfs.disable: on</pre><pre>features.shard: on</pre><pre><br></pre><pre><br></pre><pre><br></pre><pre>Lindolfo Meira, MSc</pre><pre>Diretor Geral, Centro Nacional de Supercomputação</pre><pre>Universidade Federal do Rio Grande do Sul</pre><pre>+55 (51) 3308-3139</pre><pre><br></pre><pre>On Wed, 23 Jan 2019, Jim Kinney wrote:</pre><pre><br></pre><blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:2px solid rgb(114,159,207);padding-left:1ex"><pre>Check permissions on the mount. I have multiple dozens of systems</pre><pre>mounting 18 "exports" using fuse and it works for multiple user</pre><pre>read/write based on user access permissions to the mount point space.</pre><pre>/home is mounted for 150+ users plus another dozen+ lab storage spaces.</pre><pre>I do manage user access with freeIPA across all systems to keep things</pre><pre>consistent.</pre><pre>On Wed, 2019-01-23 at 19:31 -0200, Lindolfo Meira wrote:</pre><blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left:2px solid rgb(114,159,207);padding-left:1ex"><pre>Am I missing something here? A mere write operation, using vim or</pre><pre>nano, cannot be performed on a gluster volume mounted over fuse! What</pre><pre>gives?</pre><pre>Lindolfo Meira, MScDiretor Geral, Centro Nacional de</pre><pre>SupercomputaçãoUniversidade Federal do Rio Grande do Sul+55 (51)</pre><pre>3308-3139_______________________________________________Gluster-users </pre><pre>mailing </pre><a href="mailto:listGluster-users@gluster.org" target="_blank"><pre>listGluster-users@gluster.org</pre></a><pre><br></pre><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank"><pre>https://lists.gluster.org/mailman/listinfo/gluster-users</pre></a><pre><br></pre></blockquote><pre>-- </pre><pre>James P. Kinney III</pre><pre><br></pre><pre>Every time you stop a school, you will have to build a jail. What you</pre><pre>gain at one end you lose at the other. It's like feeding a dog on his</pre><pre>own tail. It won't fatten the dog.</pre><pre>- Speech 11/23/1900 Mark Twain</pre><pre><br></pre><a href="http://heretothereideas.blogspot.com/" target="_blank"><pre>http://heretothereideas.blogspot.com/</pre></a><pre><br></pre><pre><br></pre></blockquote></blockquote></blockquote></blockquote><div><span><pre><pre>-- <br></pre>James P. Kinney III
Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain
<a href="http://heretothereideas.blogspot.com/" target="_blank">http://heretothereideas.blogspot.com/</a>
</pre></span></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</blockquote></div><br>-- <br>Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.</body></html>