<html dir="ltr"><head></head><body style="text-align:left; direction:ltr;"><div>That really sounds like a bug with the sharding. I'm not using sharding on my setup and files are writeable (vim) with 2 bytes and no errors occur.</div><div>Perhaps the small size is cached until it's large enough to trigger a write</div><div><br></div><div>On Wed, 2019-01-23 at 21:46 -0200, Lindolfo Meira wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>Also I noticed that any subsequent write (after the first write with 340 </pre><pre>bytes or more), regardless the size, will work as expected.</pre><pre><br></pre><pre><br></pre><pre>Lindolfo Meira, MSc</pre><pre>Diretor Geral, Centro Nacional de Supercomputação</pre><pre>Universidade Federal do Rio Grande do Sul</pre><pre>+55 (51) 3308-3139</pre><pre><br></pre><pre>On Wed, 23 Jan 2019, Lindolfo Meira wrote:</pre><pre><br></pre><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>Just checked: when the write is >= 340 bytes, everything works as </pre><pre>supposed. If the write is smaller, the error takes place. And when it </pre><pre>does, nothing is logged on the server. The client, however, logs the </pre><pre>following:</pre><pre><br></pre><pre>[2019-01-23 23:28:54.554664] W [MSGID: 103046] </pre><pre>[rdma.c:3502:gf_rdma_decode_header] 0-rpc-transport/rdma: received a msg </pre><pre>of type RDMA_ERROR</pre><pre><br></pre><pre>[2019-01-23 23:28:54.554728] W [MSGID: 103046] </pre><pre>[rdma.c:3939:gf_rdma_process_recv] 0-rpc-transport/rdma: peer </pre><pre>(172.24.1.6:49152), couldn't encode or decode the msg properly or write </pre><pre>chunks were not provided for replies that were bigger than </pre><pre>RDMA_INLINE_THRESHOLD (2048)</pre><pre><br></pre><pre>[2019-01-23 23:28:54.554765] W [MSGID: 114031] </pre><pre>[client-rpc-fops_v2.c:680:client4_0_writev_cbk] 0-gfs-client-5: remote </pre><pre>operation failed [Transport endpoint is not connected]</pre><pre><br></pre><pre>[2019-01-23 23:28:54.554850] W [fuse-bridge.c:1436:fuse_err_cbk] </pre><pre>0-glusterfs-fuse: 1723199: FLUSH() ERR => -1 (Transport endpoint is not </pre><pre>connected)</pre><pre><br></pre><pre><br></pre><pre><br></pre><pre>Lindolfo Meira, MSc</pre><pre>Diretor Geral, Centro Nacional de Supercomputação</pre><pre>Universidade Federal do Rio Grande do Sul</pre><pre>+55 (51) 3308-3139</pre><pre><br></pre><pre>On Wed, 23 Jan 2019, Lindolfo Meira wrote:</pre><pre><br></pre><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>Hi Jim. Thanks for taking the time.</pre><pre><br></pre><pre>Sorry I didn't express myself properly. It's not a simple matter of </pre><pre>permissions. Users can write to the volume alright. It's when vim and nano </pre><pre>are used, or when small file writes are performed (by cat or echo), that </pre><pre>it doesn't work. The file is updated with the write in the server, but it </pre><pre>shows up as empty in the client.</pre><pre><br></pre><pre>I guess it has something to do with the size of the write, because I ran a </pre><pre>test writing to a file one byte at a time, and it never showed up as </pre><pre>having any content in the client (although in the server it kept growing </pre><pre>accordingly).</pre><pre><br></pre><pre>I should point out that I'm using a sharded volume. But when I was testing </pre><pre>a striped volume, it also happened. Output of "gluster volume info" </pre><pre>follows bellow:</pre><pre><br></pre><pre>Volume Name: gfs</pre><pre>Type: Distribute</pre><pre>Volume ID: b5ef065f-1ba2-481f-8108-e8f6d2d3f036</pre><pre>Status: Started</pre><pre>Snapshot Count: 0</pre><pre>Number of Bricks: 6</pre><pre>Transport-type: rdma</pre><pre>Bricks:</pre><pre>Brick1: pfs01-ib:/mnt/data</pre><pre>Brick2: pfs02-ib:/mnt/data</pre><pre>Brick3: pfs03-ib:/mnt/data</pre><pre>Brick4: pfs04-ib:/mnt/data</pre><pre>Brick5: pfs05-ib:/mnt/data</pre><pre>Brick6: pfs06-ib:/mnt/data</pre><pre>Options Reconfigured:</pre><pre>nfs.disable: on</pre><pre>features.shard: on</pre><pre><br></pre><pre><br></pre><pre><br></pre><pre>Lindolfo Meira, MSc</pre><pre>Diretor Geral, Centro Nacional de Supercomputação</pre><pre>Universidade Federal do Rio Grande do Sul</pre><pre>+55 (51) 3308-3139</pre><pre><br></pre><pre>On Wed, 23 Jan 2019, Jim Kinney wrote:</pre><pre><br></pre><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>Check permissions on the mount. I have multiple dozens of systems</pre><pre>mounting 18 "exports" using fuse and it works for multiple user</pre><pre>read/write based on user access permissions to the mount point space.</pre><pre>/home is mounted for 150+ users plus another dozen+ lab storage spaces.</pre><pre>I do manage user access with freeIPA across all systems to keep things</pre><pre>consistent.</pre><pre>On Wed, 2019-01-23 at 19:31 -0200, Lindolfo Meira wrote:</pre><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>Am I missing something here? A mere write operation, using vim or</pre><pre>nano, cannot be performed on a gluster volume mounted over fuse! What</pre><pre>gives?</pre><pre>Lindolfo Meira, MScDiretor Geral, Centro Nacional de</pre><pre>SupercomputaçãoUniversidade Federal do Rio Grande do Sul+55 (51)</pre><pre>3308-3139_______________________________________________Gluster-users </pre><pre>mailing </pre><a href="mailto:listGluster-users@gluster.org"><pre>listGluster-users@gluster.org</pre></a><pre><br></pre><a href="https://lists.gluster.org/mailman/listinfo/gluster-users"><pre>https://lists.gluster.org/mailman/listinfo/gluster-users</pre></a><pre><br></pre></blockquote><pre>-- </pre><pre>James P. Kinney III</pre><pre><br></pre><pre>Every time you stop a school, you will have to build a jail. What you</pre><pre>gain at one end you lose at the other. It's like feeding a dog on his</pre><pre>own tail. It won't fatten the dog.</pre><pre>- Speech 11/23/1900 Mark Twain</pre><pre><br></pre><a href="http://heretothereideas.blogspot.com/"><pre>http://heretothereideas.blogspot.com/</pre></a><pre><br></pre><pre><br></pre></blockquote></blockquote></blockquote></blockquote><div><span><pre><pre>-- <br></pre>James P. Kinney III
Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain
http://heretothereideas.blogspot.com/
</pre></span></div></body></html>