<div dir="ltr"><br><div class="gmail_extra"><div class="gmail_extra">Hi Kotresh,</div><div class="gmail_extra"><br></div><div class="gmail_extra">Please correct me if i am wrong, Once a file write completes and as soon as closes fds, bitrot waits for 120 seconds and starts hashing and update signature for the file in brick. </div><div class="gmail_extra"><br></div><div class="gmail_extra">But, what i am feeling that bitrot takes too much of time to complete hashing.</div><div class="gmail_extra"><br></div><div class="gmail_extra">below is test result i would like to share. </div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">writing data in below path using dd : </div><div class="gmail_extra"><br></div><div class="gmail_extra">/mnt/gluster/data/G (mount point)</div><div class="gmail_extra">-rw-r--r-- 1 root root 10M Sep 20 12:19 test53-bs10M-c1.nul</div><div class="gmail_extra">-rw-r--r-- 1 root root 100M Sep 20 12:19 test54-bs10M-c10.nul</div><div class="gmail_extra"><br></div><div class="gmail_extra">No any other write or read process is going on.</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">Checking file data in one of the brick.</div><div class="gmail_extra"><br></div><div class="gmail_extra">-rw-r--r-- 2 root root 2.5M Sep 20 12:23 test53-bs10M-c1.nul</div><div class="gmail_extra">-rw-r--r-- 2 root root 25M Sep 20 12:23 test54-bs10M-c10.nul</div><div class="gmail_extra"><br></div><div class="gmail_extra">file's stat and getfattr info from brick, after write process completed.</div><div class="gmail_extra"><br></div><div class="gmail_extra">gfstst-node5:/media/disk2/brick2/data/G$ stat test53-bs10M-c1.nul</div><div class="gmail_extra"> File: ‘test53-bs10M-c1.nul’</div><div class="gmail_extra"> Size: 2621440 Blocks: 5120 IO Block: 4096 regular file</div><div class="gmail_extra">Device: 821h/2081d Inode: 536874168 Links: 2</div><div class="gmail_extra">Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)</div><div class="gmail_extra">Access: 2016-09-20 12:23:28.798886647 +0530</div><div class="gmail_extra">Modify: 2016-09-20 12:23:28.994886646 +0530</div><div class="gmail_extra">Change: 2016-09-20 12:23:28.998886646 +0530</div><div class="gmail_extra"> Birth: -</div><div class="gmail_extra"><br></div><div class="gmail_extra">gfstst-node5:/media/disk2/brick2/data/G$ stat test54-bs10M-c10.nul</div><div class="gmail_extra"> File: ‘test54-bs10M-c10.nul’</div><div class="gmail_extra"> Size: 26214400 Blocks: 51200 IO Block: 4096 regular file</div><div class="gmail_extra">Device: 821h/2081d Inode: 536874169 Links: 2</div><div class="gmail_extra">Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)</div><div class="gmail_extra">Access: 2016-09-20 12:23:42.902886624 +0530</div><div class="gmail_extra">Modify: 2016-09-20 12:23:44.378886622 +0530</div><div class="gmail_extra">Change: 2016-09-20 12:23:44.378886622 +0530</div><div class="gmail_extra"> Birth: -</div><div class="gmail_extra"><br></div><div class="gmail_extra">gfstst-node5:/media/disk2/brick2/data/G$ sudo getfattr -m. -e hex -d test53-bs10M-c1.nul</div><div class="gmail_extra"># file: test53-bs10M-c1.nul</div><div class="gmail_extra">trusted.bit-rot.version=0x020000000000000057daa7b50002e5b4</div><div class="gmail_extra">trusted.ec.config=0x0000080501000200</div><div class="gmail_extra">trusted.ec.size=0x0000000000a00000</div><div class="gmail_extra">trusted.ec.version=0x00000000000000500000000000000050</div><div class="gmail_extra">trusted.gfid=0xe2416bd1aae4403c88f44286273bbe99</div><div class="gmail_extra"><br></div><div class="gmail_extra">gfstst-node5:/media/disk2/brick2/data/G$ sudo getfattr -m. -e hex -d test54-bs10M-c10.nul</div><div class="gmail_extra"># file: test54-bs10M-c10.nul</div><div class="gmail_extra">trusted.bit-rot.version=0x020000000000000057daa7b50002e5b4</div><div class="gmail_extra">trusted.ec.config=0x0000080501000200</div><div class="gmail_extra">trusted.ec.size=0x0000000006400000</div><div class="gmail_extra">trusted.ec.version=0x00000000000003200000000000000320</div><div class="gmail_extra">trusted.gfid=0x54e018dd8c5a4bd79e0317729d8a57c5</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">file's stat and getfattr info from brick, after bitrot signature updated.</div><div class="gmail_extra"><br></div><div class="gmail_extra">gfstst-node5:/media/disk2/brick2/data/G$ stat test53-bs10M-c1.nul</div><div class="gmail_extra"> File: ‘test53-bs10M-c1.nul’</div><div class="gmail_extra"> Size: 2621440 Blocks: 5120 IO Block: 4096 regular file</div><div class="gmail_extra">Device: 821h/2081d Inode: 536874168 Links: 2</div><div class="gmail_extra">Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)</div><div class="gmail_extra">Access: 2016-09-20 12:25:31.494886450 +0530</div><div class="gmail_extra">Modify: 2016-09-20 12:23:28.994886646 +0530</div><div class="gmail_extra">Change: 2016-09-20 12:27:00.994886307 +0530</div><div class="gmail_extra"> Birth: -</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">gfstst-node5:/media/disk2/brick2/data/G$ sudo getfattr -m. -e hex -d test53-bs10M-c1.nul</div><div class="gmail_extra"># file: test53-bs10M-c1.nul</div><div class="gmail_extra">trusted.bit-rot.signature=0x0102000000000000006de7493c5c90f643357c268fbaaf461c1567e0334e4948023ce17268403aa37a</div><div class="gmail_extra">trusted.bit-rot.version=0x020000000000000057daa7b50002e5b4</div><div class="gmail_extra">trusted.ec.config=0x0000080501000200</div><div class="gmail_extra">trusted.ec.size=0x0000000000a00000</div><div class="gmail_extra">trusted.ec.version=0x00000000000000500000000000000050</div><div class="gmail_extra">trusted.gfid=0xe2416bd1aae4403c88f44286273bbe99</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">gfstst-node5:/media/disk2/brick2/data/G$ stat test54-bs10M-c10.nul</div><div class="gmail_extra"> File: ‘test54-bs10M-c10.nul’</div><div class="gmail_extra"> Size: 26214400 Blocks: 51200 IO Block: 4096 regular file</div><div class="gmail_extra">Device: 821h/2081d Inode: 536874169 Links: 2</div><div class="gmail_extra">Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)</div><div class="gmail_extra">Access: 2016-09-20 12:25:47.510886425 +0530</div><div class="gmail_extra">Modify: 2016-09-20 12:23:44.378886622 +0530</div><div class="gmail_extra">Change: 2016-09-20 12:38:05.954885243 +0530</div><div class="gmail_extra"> Birth: -</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">gfstst-node5:/media/disk2/brick2/data/G$ sudo getfattr -m. -e hex -d test54-bs10M-c10.nul</div><div class="gmail_extra"># file: test54-bs10M-c10.nul</div><div class="gmail_extra">trusted.bit-rot.signature=0x010200000000000000394c345f0b0c63ee652627a62eed069244d35c4d5134e4f07d4eabb51afda47e</div><div class="gmail_extra">trusted.bit-rot.version=0x020000000000000057daa7b50002e5b4</div><div class="gmail_extra">trusted.ec.config=0x0000080501000200</div><div class="gmail_extra">trusted.ec.size=0x0000000006400000</div><div class="gmail_extra">trusted.ec.version=0x00000000000003200000000000000320</div><div class="gmail_extra">trusted.gfid=0x54e018dd8c5a4bd79e0317729d8a57c5</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">(Actual time taken for reading file from brick for md5sum)</div><div class="gmail_extra"><br></div><div class="gmail_extra">gfstst-node5:/media/disk2/brick2/data/G$ time md5sum test53-bs10M-c1.nul</div><div class="gmail_extra">8354dcaa18a1ecb52d0895bf00888c44 test53-bs10M-c1.nul</div><div class="gmail_extra"><br></div><div class="gmail_extra">real 0m0.045s</div><div class="gmail_extra">user 0m0.007s</div><div class="gmail_extra">sys 0m0.003s</div><div class="gmail_extra"><br></div><div class="gmail_extra">gfstst-node5:/media/disk2/brick2/data/G$ time md5sum test54-bs10M-c10.nul</div><div class="gmail_extra">bed3c0a4a1407f584989b4009e9ce33f test54-bs10M-c10.nul</div><div class="gmail_extra"><br></div><div class="gmail_extra">real 0m0.166s</div><div class="gmail_extra">user 0m0.062s</div><div class="gmail_extra">sys 0m0.011s</div><div class="gmail_extra"><br></div><div class="gmail_extra">As you can see that 'test54-bs10M-c10.nul' file took around 12 minutes to update bitort signature (pls refer stat output for the file).</div><div class="gmail_extra"><br></div><div class="gmail_extra">what would be the cause for such a slow read?. Any limitation in read data from brick?</div><div class="gmail_extra"><br></div><div class="gmail_extra">Also, i am seeing this line bitd.log, what does this mean?</div><div class="gmail_extra">[bit-rot.c:1784:br_rate_limit_signer] 0-glsvol1-bit-rot-0: [Rate Limit Info] "tokens/sec (rate): 131072, maxlimit: 524288<br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks </div><div class="gmail_extra">Amudhan P</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_quote">On Mon, Sep 19, 2016 at 1:00 PM, Kotresh Hiremath Ravishankar <span dir="ltr"><<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Amudhan,<br>
<br>
Thanks for testing out the bitrot feature and sorry for the delayed response.<br>
Please find the answers inline.<br>
<br>
Thanks and Regards,<br>
Kotresh H R<br>
<span class="gmail-"><br>
----- Original Message -----<br>
> From: "Amudhan P" <<a href="mailto:amudhan83@gmail.com">amudhan83@gmail.com</a>><br>
> To: "Gluster Users" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
> Sent: Friday, September 16, 2016 4:14:10 PM<br>
> Subject: Re: [Gluster-users] 3.8.3 Bitrot signature process<br>
><br>
> Hi,<br>
><br>
> Can anyone reply to this mail.<br>
><br>
> On Tue, Sep 13, 2016 at 12:49 PM, Amudhan P < <a href="mailto:amudhan83@gmail.com">amudhan83@gmail.com</a> > wrote:<br>
><br>
><br>
><br>
> Hi,<br>
><br>
> I am testing bitrot feature in Gluster 3.8.3 with disperse EC volume 4+1.<br>
><br>
> When i write single small file (< 10MB) after 2 seconds i can see bitrot<br>
> signature in bricks for the file, but when i write multiple files with<br>
> different size ( > 10MB) it takes long time (> 24hrs) to see bitrot<br>
> signature in all the files.<br>
<br>
</span> The default timeout for signing to happen is 120 seconds. So the signing will happen<br>
120 secs after the last fd gets closed on that file. So if the file is being written<br>
continuously, it will not be signed until 120 secs after it's last fd is closed.<br>
<span class="gmail-">><br>
> My questions are.<br>
> 1. I have enabled scrub schedule as hourly and throttle as normal, does this<br>
> make any impact in delaying bitrot signature?<br>
</span> No.<br>
<span class="gmail-">> 2. other than "bitd.log" where else i can watch current status of bitrot,<br>
> like number of files added for signature and file status?<br>
</span> Signature will happen after 120 sec of last fd closure, as said above.<br>
There is not status command which tracks the signature of the files.<br>
But there is bitrot status command which tracks the number of files<br>
scrubbed.<br>
<br>
#gluster vol bitrot <volname> scrub status<br>
<span class="gmail-"><br>
<br>
> 3. where i can confirm that all the files in the brick are bitrot signed?<br>
<br>
</span> As said, signing information of all the files is not tracked.<br>
<span class="gmail-"><br>
> 4. is there any file read size limit in bitrot?<br>
<br>
</span> I didn't get. Could you please elaborate this ?<br>
<span class="gmail-"><br>
> 5. options for tuning bitrot for faster signing of files?<br>
<br>
</span> Bitrot feature is mainly to detect silent corruption (bitflips) of files due to long<br>
term storage. Hence the default is 120 sec of last fd closure, the signing happens.<br>
But there is a tune able which can change the default 120 sec but that's only for<br>
testing purposes and we don't recommend it.<br>
<br>
gluster vol get master features.expiry-time<br>
<br>
For testing purposes, you can change this default and test.<br>
><br>
> Thanks<br>
> Amudhan<br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</blockquote></div><br></div></div>