<div dir="ltr"><div>I have checked more and mount the volume in another region (in region c), the ping time from region c is as follows:<br><br><div>ping 10.0.0.1 & 10.0.0.3 are bellow <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span> </span>time=12 ms</span></div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ping 10.0.0.5 & 10.0.0.6 are more than</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> </span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);float:none;display:inline"><span> </span>time=32 ms<br><br></span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);float:none;display:inline">I expect the bricks with lower ping time to be selected at write time, but still the brick selection is not as desired and those bricks with more ping time are selected. I change the cluster.halo-max-latency to 20, but this not affect anything</span></span>.<br><br></div><div>on more thing is, the previous email I wrote was not with the right result, I though that by changing the range to [0-999999] everything will be ok, but my today experience shows that I was wrong.<br><br><br></div><div>any help will be appreciated ;) <br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 5, 2018 at 4:04 PM, atris adam <span dir="ltr"><<a href="mailto:atris.adam@gmail.com" target="_blank">atris.adam@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div></div>I have mounted the halo glusterfs volume in debug mode, and the output is as follows:<br>.<br>.<br>.<br>[2018-02-05 11:42:48.282473] D [rpc-clnt-ping.c:211:rpc_clnt_<wbr>ping_cbk] 0-test-halo-client-1: Ping latency is 0ms<br>[2018-02-05 11:42:48.282502] D [MSGID: 0] [afr-common.c:5025:afr_get_<wbr>halo_latency] 0-test-halo-replicate-0: Using halo latency 10<br>[2018-02-05 11:42:48.282525] D [MSGID: 0] [afr-common.c:4820:__afr_<wbr>handle_ping_event] 0-test-halo-client-1: Client ping @ 140032933708544 ms<br>.<br>.<br>.<br>[2018-02-05 11:42:48.393776] D [MSGID: 0] [afr-common.c:4803:find_worst_<wbr>up_child] 0-test-halo-replicate-0: Found worst up child (1) @ 140032933708544 ms latency<br>[2018-02-05 11:42:48.393803] D [MSGID: 0] [afr-common.c:4903:__afr_<wbr>handle_child_up_event] 0-test-halo-replicate-0: Marking child 1 down, doesn't meet halo threshold (10), and > halo_min_replicas (2)<br>.<br>.<br>.<br><br></div><div>I think these debug output means:<br></div>As the ping time for test-halo-client-1 (brick2) is (0.5ms) and it is not under halo threshold (10 ms), this false decision for selecting bricks happen to halo.<br></div>I can not set the halo threshold to 0 because:<br><br>#gluster vol set test-halo cluster.halo-max-latency 0<br>volume set: failed: '0' in 'option halo-max-latency 0' is out of range [1 - 99999]<br><br></div>so I think the range [1 - 99999] should change to [0 - 99999], so I can get the desired brick selection for halo feature, am I right? If not, why the halo decide to mark down the best brick which has ping time bellow 0.5ms?<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Feb 4, 2018 at 2:27 PM, atris adam <span dir="ltr"><<a href="mailto:atris.adam@gmail.com" target="_blank">atris.adam@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I have 2 data centers in two different region, each DC have 3 severs, I have created glusterfs volume with 4 replica, this is glusterfs volume info output:<div><br></div><div><div><br></div><div>Volume Name: test-halo</div><div>Type: Replicate</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 1 x 4 = 4</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 10.0.0.1:/mnt/test1</div><div>Brick2: 10.0.0.3:/mnt/test2</div><div>Brick3: 10.0.0.5:/mnt/test3</div><div>Brick4: 10.0.0.6:/mnt/test4</div><div>Options Reconfigured:</div><div>cluster.halo-shd-max-latency: 5</div><div>cluster.halo-max-latency: 10</div><div>cluster.quorum-count: 2</div><div>cluster.quorum-type: fixed</div><div>cluster.halo-enabled: yes</div><div>transport.address-family: inet</div><div>nfs.disable: on</div></div><div><br></div><div>bricks with ip 10.0.0.1 & 10.0.0.3 are in region A and bricks with ip 10.0.0.5 & 10.0.0.6 are in region B</div><div><br></div><div><br></div><div>when I mount the volume in region A, I except the data first store in brick1 & brick2, then asynchronously the data copies in region B, on brick3 & brick4.</div><div><br></div><div>Am I write? this is what halo claims?</div><div><br></div><div>If yes, unfortunately, this not happen to me, no differ I mount the volume in region A or mount the volume in region B, all the data are copied in brick3 & brick4 and no data copies in brick1 & brick2.</div><div><br></div><div>ping bricks ip from region A is as follows:</div><div>ping 10.0.0.1 & 10.0.0.3 are bellow <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span> </span>time=0.500 ms</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">ping 10.0.0.5 & 10.0.0.6 are more than</span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> </span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);float:none;display:inline"><span> </span>time=20 ms</span>
<br></span></div><div><br></div><div>What is the logic that the halo select the bricks to write to?if it is the access time, so when I mount the volume in region A, the ping time to brick1 & brick2 is bellow 0.5 ms, but the halo select the brick3 & brick4!!!!</div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">glusterfs version is:</span></div><div><span style="text-align:start;text-indent:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><div>glusterfs 3.12.4</div><div><br></div><div>I really need to work with halo feature, But I am not successful to run this case, Can anyone help me soon??</div><div><br></div><div><br></div><div>Thx alot</div></span></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>