<div dir="ltr">Hi Davide,<div><br></div><div>The options information is already provided in prior e-mail, see the termbin.con link for the options of the volume after the 4.1.6 upgrade. </div><div><br></div><div><div style="color:rgb(80,0,80)">The gluster options set on the volume are:<br></div><div style="color:rgb(80,0,80)"><a href="https://termbin.com/yxtd" target="_blank">https://termbin.com/yxtd</a></div></div><div><br></div><div>This is the other piece:</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)"># gluster v info export
</span><br> <br>Volume Name: export
<br>Type: Replicate
<br>Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff
<br>Status: Started
<br>Snapshot Count: 0
<br>Number of Bricks: 1 x 2 = 2
<br>Transport-type: tcp
<br>Bricks:
<br>Brick1: 10.0.1.7:/bricks/hdds/brick
<br>Brick2: 10.0.1.6:/bricks/hdds/brick
<br>Options Reconfigured:
<br>performance.stat-prefetch: on
<br>performance.cache-min-file-size: 0
<br>network.inode-lru-limit: 65536
<br>performance.cache-invalidation: on
<br>features.cache-invalidation: on
<br>performance.md-cache-timeout: 600
<br>features.cache-invalidation-timeout: 600
<br>performance.cache-samba-metadata: on
<br>transport.address-family: inet
<br>server.allow-insecure: on
<br>performance.cache-size: 10GB
<br>cluster.server-quorum-type: server
<br>nfs.disable: on
<br>performance.io-thread-count: 64
<br>performance.io-cache: on
<br>cluster.lookup-optimize: on
<br>cluster.readdir-optimize: on
<br>server.event-threads: 5
<br>client.event-threads: 5
<br>performance.cache-max-file-size: 256MB
<br>diagnostics.client-log-level: INFO
<br>diagnostics.brick-log-level: INFO
<br>cluster.server-quorum-ratio: 51%<br>
<br></span></div><div><font face="monospace">Now I did create a backup of /var/lib/glusterd so if you tell me how to pull information from there to compare I can do it.</font></div><div><font face="monospace"><br></font></div><div><font face="monospace">I compared the file /var/lib/glusterd/vols/export/info and it is the same in both, though entries are in different order.</font></div><div><font face="monospace"><br></font></div><div><font face="monospace">Diego</font></div><div><font face="monospace"><br></font></div><div><br></div><div><span style="font-family:monospace">
<br></span></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jan 15, 2019 at 5:03 AM Davide Obbi <<a href="mailto:davide.obbi@booking.com">davide.obbi@booking.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jan 15, 2019 at 2:18 AM Diego Remolina <<a href="mailto:dijuremo@gmail.com" target="_blank">dijuremo@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Dear all,<div><br></div><div>I was running gluster 3.10.12 on a pair of servers and recently upgraded to 4.1.6. There is a cron job that runs nightly in one machine, which rsyncs the data on the servers over to another machine for backup purposes. The rsync operation runs on one of the gluster servers, which mounts the gluster volume via fuse on /export.</div><div><br></div><div>When using 3.10.12, this process would start at 8:00PM nightly, and usually end up at around 4:30AM when the servers had been freshly rebooted. From this point, things would start taking a bit longer and stabilize ending at around 7-9AM depending on actual file changes and at some point the servers would start eating up so much ram (up to 30GB) and I would have to reboot them to bring things back to normal as the file system would become extremely slow (perhaps the memory leak I have read was present on 3.10.x).</div><div><br></div><div>After upgrading to 4.1.6 over the weekend, I was shocked to see the rsync process finish in about 1 hour and 26 minutes. This is compared to 8 hours 30 mins with the older version. This is a nice speed up, however, I can only ask myself what has changed so drastically that this process is now so fast. Have there really been improvements in 4.1.6 that could speed this up so dramatically? In both of my test cases, there would had not really been a lot to copy via rsync given the fresh reboots are done on Saturday after the sync has finished from the day before. </div><div><br></div><div>In general, the servers (which are accessed via samba for windows clients) are much faster and responsive since the update to 4.1.6. Tonight I will have the first rsync run which will actually have to copy the day's changes and will have another point of comparison.</div><div><br></div><div>I am still using fuse mounts for samba, due to prior problems with vsf =gluster, which are currently present in Samba 4.8.3-4, and already documented in bugs, for which patches exist, but no official updated samba packages have been released yet. Since I was going from 3.10.12 to 4.1.6 I also did not want to change other things to make sure I could track any issues just related to the change in gluster versions and eliminate other complexity.</div><div><br></div><div>The file system currently has about 16TB of data in</div><div>5142816 files and 696544 directories<br></div><div><br></div><div>I've just ran the following code to count files and dirs and it took 67mins 38.957 secs to complete in this gluster volume:</div><div><a href="https://github.com/ChristopherSchultz/fast-file-count" target="_blank">https://github.com/ChristopherSchultz/fast-file-count</a><br></div><div><br></div><div><div># time ( /root/sbin/dircnt /export )</div><div>/export contains 5142816 files and 696544 directories</div><div><br></div><div>real 67m38.957s</div><div>user 0m6.225s</div><div>sys 0m48.939s</div></div><div><br></div><div>The gluster options set on the volume are:<br></div><div><a href="https://termbin.com/yxtd" target="_blank">https://termbin.com/yxtd</a><br></div><div><br></div><div><div># gluster v status export</div><div>Status of volume: export</div><div>Gluster process TCP Port RDMA Port Online Pid</div><div>------------------------------------------------------------------------------</div><div>Brick 10.0.1.7:/bricks/hdds/brick 49157 0 Y 13986</div><div>Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y 9953</div><div>Self-heal Daemon on localhost N/A N/A Y 21934</div><div>Self-heal Daemon on 10.0.1.5 N/A N/A Y 4598</div><div>Self-heal Daemon on 10.0.1.6 N/A N/A Y 14485</div><div><br></div><div>Task Status of Volume export</div><div>------------------------------------------------------------------------------</div><div>There are no active volume tasks</div></div><div><br></div><div>Truth, there is a 3rd server here, but no bricks on it.</div><div><br></div><div>Thoughts?</div><div><br></div><div>Diego</div></div></div><div id="gmail-m_-4021393732076721680m_8084651329793795211gmail-m_7462352325940458688gmail-m_-6479459361629161759DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br>
<table style="border-top:1px solid rgb(211,212,222)">
<tbody><tr>
<td style="width:55px;padding-top:13px"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon" target="_blank"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" style="width: 46px; height: 29px;" width="46" height="29"></a></td>
<td style="width:470px;padding-top:12px;color:rgb(65,66,78);font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px">Virus-free. <a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link" style="color:rgb(68,83,234)" target="_blank">www.avast.com</a>
</td>
</tr>
</tbody></table><a href="#m_-4021393732076721680_m_8084651329793795211_m_7462352325940458688_m_-6479459361629161759_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"></a></div></div></div></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br clear="all"></div><div>Hi Diego,</div><div><br></div><div>Besides the actual improvements made in the code i think new releases might implement volume options by default that before might have had different setting. I would have been interesting to diff "gluster volume get <volname> all" befor and after the upgrade. Just for curiosity and i am trying to figure out volume options for rsync kind of workloads can you share the command output anyway along with gluster volume info <volname>?</div><div><br></div><div>thanks</div><div><br></div></div>
</blockquote></div>