<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi Nithya,</div><div class=""><br class=""></div><div class="">thank you, my answers are in lines.</div><br class=""><div><blockquote type="cite" class=""><div class="">Il giorno 08 ott 2018, alle ore 10:43, Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" class="">nbalacha@redhat.com</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Mauro,<div class=""><br class=""></div><div class="">Yes, a rebalance consists of 2 operations for every directory:</div><div class=""><ol class=""><li class="">Fix the layout for the new volume config (newly added or removed bricks)<br class=""></li><li class="">Migrate files to their new hashed subvols based on the new layout<br class=""></li></ol><div class=""><br class=""></div></div><div class="">Are you running a rebalance because you added new bricks to the volume ? As per an earlier email you have already run a fix-layout.</div></div></div></blockquote><div><br class=""></div><div>Yes, we added new bricks to the volume and we already executed fix-layout before.</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="">On s04, please check the rebalance log file to see why the rebalance failed.</div></div></div></blockquote><div><br class=""></div><div>On s04, rebalance failed after the following errors (before these lines no errors were found):</div><div><br class=""></div><div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.359634] I [MSGID: 109063] [dht-layout.c:716:dht_layout_normalize] 0-tier2-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=2 overlaps=0</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.362424] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-7: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.362504] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-7: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.362525] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-7: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.363105] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-8: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.363163] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-8: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.363180] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-8: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.364920] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-11: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.364969] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-11: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.364985] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-11: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.366864] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-6: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.366912] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-6: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.366926] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-6: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.374818] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-9: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.374866] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-9: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.374879] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-9: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.406076] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-10: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.406145] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-10: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:37.406183] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-10: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.039835] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-11: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.039911] E [MSGID: 109006] [dht-selfheal.c:673:dht_selfheal_dir_xattr_cbk] 0-tier2-dht: layout setxattr failed on tier2-disperse-11, path:/ gfid:00000000-0000-0000-0000-000000000001 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.039944] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-11: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.039958] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-11: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.040441] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-7: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.040480] E [MSGID: 109006] [dht-selfheal.c:673:dht_selfheal_dir_xattr_cbk] 0-tier2-dht: layout setxattr failed on tier2-disperse-7, path:/ gfid:00000000-0000-0000-0000-000000000001 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.040518] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-7: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.040534] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-7: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.061789] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-9: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.061830] E [MSGID: 109006] [dht-selfheal.c:673:dht_selfheal_dir_xattr_cbk] 0-tier2-dht: layout setxattr failed on tier2-disperse-9, path:/ gfid:00000000-0000-0000-0000-000000000001 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.061859] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-9: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.061873] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-9: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.062283] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-8: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.062323] E [MSGID: 109006] [dht-selfheal.c:673:dht_selfheal_dir_xattr_cbk] 0-tier2-dht: layout setxattr failed on tier2-disperse-8, path:/ gfid:00000000-0000-0000-0000-000000000001 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.062353] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-8: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.062367] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-8: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.064613] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-6: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.064655] E [MSGID: 109006] [dht-selfheal.c:673:dht_selfheal_dir_xattr_cbk] 0-tier2-dht: layout setxattr failed on tier2-disperse-6, path:/ gfid:00000000-0000-0000-0000-000000000001 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.064685] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-6: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.064700] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-6: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.064727] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-10: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.064766] E [MSGID: 109006] [dht-selfheal.c:673:dht_selfheal_dir_xattr_cbk] 0-tier2-dht: layout setxattr failed on tier2-disperse-10, path:/ gfid:00000000-0000-0000-0000-000000000001 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.064794] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-10: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:51.064815] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-10: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.695948] I [dht-rebalance.c:4512:gf_defrag_start_crawl] 0-tier2-dht: gf_defrag_start_crawl using commit hash 3720343841</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.696837] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-11: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.696906] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-11: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.696924] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-11: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.697549] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-7: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.697599] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-7: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.697620] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-7: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.704120] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-8: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.704262] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-8: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.704342] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-8: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.707260] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-10: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.707312] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-10: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.707329] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-10: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.718301] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-6: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.718350] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-6: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:53.718367] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-6: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:55.626130] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-9: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:55.626207] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-9: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:55.626228] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-9: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:55.626231] I [MSGID: 109081] [dht-common.c:4379:dht_setxattr] 0-tier2-dht: fixing the layout of /</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:55.862374] I [dht-rebalance.c:5063:gf_defrag_status_get] 0-glusterfs: TIME: Estimated total time to complete (size)= 0 seconds, seconds left = 0</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:55.862440] I [MSGID: 109028] [dht-rebalance.c:5143:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 20.00 secs</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:13:55.862460] I [MSGID: 109028] [dht-rebalance.c:5147:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.476927] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-11: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.477020] E [MSGID: 109006] [dht-selfheal.c:673:dht_selfheal_dir_xattr_cbk] 0-tier2-dht: layout setxattr failed on tier2-disperse-11, path:/ gfid:00000000-0000-0000-0000-000000000001 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.477077] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-11: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.477094] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-11: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.477644] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-7: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.477695] E [MSGID: 109006] [dht-selfheal.c:673:dht_selfheal_dir_xattr_cbk] 0-tier2-dht: layout setxattr failed on tier2-disperse-7, path:/ gfid:00000000-0000-0000-0000-000000000001 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.477726] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-7: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.477740] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-7: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.477853] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-8: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.477894] E [MSGID: 109006] [dht-selfheal.c:673:dht_selfheal_dir_xattr_cbk] 0-tier2-dht: layout setxattr failed on tier2-disperse-8, path:/ gfid:00000000-0000-0000-0000-000000000001 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.477923] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-8: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.477937] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-8: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.486862] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-6: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.486902] E [MSGID: 109006] [dht-selfheal.c:673:dht_selfheal_dir_xattr_cbk] 0-tier2-dht: layout setxattr failed on tier2-disperse-6, path:/ gfid:00000000-0000-0000-0000-000000000001 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.486929] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-6: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.486944] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-6: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.493872] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-10: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.493912] E [MSGID: 109006] [dht-selfheal.c:673:dht_selfheal_dir_xattr_cbk] 0-tier2-dht: layout setxattr failed on tier2-disperse-10, path:/ gfid:00000000-0000-0000-0000-000000000001 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.493939] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-10: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.493954] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-10: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.494560] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-9: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.494598] E [MSGID: 109006] [dht-selfheal.c:673:dht_selfheal_dir_xattr_cbk] 0-tier2-dht: layout setxattr failed on tier2-disperse-9, path:/ gfid:00000000-0000-0000-0000-000000000001 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.494624] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-9: Insufficient available children for this request (have 0, need 4)</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.494640] E [MSGID: 122037] [ec-common.c:2040:ec_update_size_version_done] 0-tier2-disperse-9: Failed to update version and size [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.795320] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-7: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.795366] E [MSGID: 109039] [dht-common.c:3113:dht_find_local_subvol_cbk] 0-tier2-dht: getxattr err for dir [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.795796] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-11: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.795834] E [MSGID: 109039] [dht-common.c:3113:dht_find_local_subvol_cbk] 0-tier2-dht: getxattr err for dir [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.804770] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-6: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.804803] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-10: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.804811] E [MSGID: 109039] [dht-common.c:3113:dht_find_local_subvol_cbk] 0-tier2-dht: getxattr err for dir [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.804850] E [MSGID: 109039] [dht-common.c:3113:dht_find_local_subvol_cbk] 0-tier2-dht: getxattr err for dir [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.808500] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-8: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.808563] E [MSGID: 109039] [dht-common.c:3113:dht_find_local_subvol_cbk] 0-tier2-dht: getxattr err for dir [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.812431] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_update_cbk] 0-tier2-disperse-9: Failed to get size and version [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.812468] E [MSGID: 109039] [dht-common.c:3113:dht_find_local_subvol_cbk] 0-tier2-dht: getxattr err for dir [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.812497] E [MSGID: 0] [dht-rebalance.c:4336:dht_get_local_subvols_and_nodeuuids] 0-tier2-dht: local subvolume determination failed with error: 5 [Input/output error]</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.812700] I [MSGID: 109028] [dht-rebalance.c:5143:gf_defrag_status_get] 0-tier2-dht: Rebalance is failed. Time taken is 37.00 secs</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.812720] I [MSGID: 109028] [dht-rebalance.c:5147:gf_defrag_status_get] 0-tier2-dht: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0</span></div><div><span style="font-size: 14px;" class="">[2018-10-06 00:14:12.812870] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7efe75d18e25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x5623973d64b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x5623973d632b] ) 0-: received signum (15), shutting down</span></div><div class=""><span style="font-size: 14px;" class=""><br class=""></span></div></div><div>Regards,</div><div>Mauro</div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""></div><div class="">Regards,</div><div class="">Nithya</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On 8 October 2018 at 13:22, Mauro Tridici <span dir="ltr" class=""><<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">Hi All,</div><div class=""><br class=""></div><div class="">for your information, this is the current rebalance status:</div><div class=""><br class=""></div><div class=""><span class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[root@s01 ~]# gluster volume rebalance tier2 status</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</span></div></span><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> localhost 551922 20.3TB 2349397 0 61849 in progress 55:25:38</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s02-stg 287631 13.2TB 959954 0 30262 in progress 55:25:39</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s03-stg 288523 12.7TB 973111 0 30220 in progress 55:25:39</span></div><span class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s04-stg 0 0Bytes 0 0 0 failed 0:00:37</span></div></span><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s05-stg 0 0Bytes 0 0 0 completed 48:33:03</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s06-stg 0 0Bytes 0 0 0 completed 48:33:02</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Estimated time left for rebalance to complete : 1023:49:56</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">volume rebalance: tier2: success</span></div></div><div class=""><br class=""></div><div class="">Rebalance is migrating files on s05, s06 servers and on s04 too (although it is marked as failed).</div><div class="">s05 and s06 tasks are completed.</div><div class=""><br class=""></div><div class="">Questions:</div><div class=""><br class=""></div><div class="">1) it seems that rebalance is moving files, but it is fixing the layout also, is it normal?</div><div class="">2) when rebalance will be completed, what we need to do before return the gluster storage to the users? We have to launch rebalance again in order to involve s04 server too or a fix-layout to eventually fix some error on s04?</div><span class=""><div class=""><br class=""></div><div class="">Thank you very much,</div><div class="">Mauro</div><div class=""><br class=""></div><br class=""></span><div class=""><blockquote type="cite" class=""><div class=""><div class="h5"><div class="">Il giorno 07 ott 2018, alle ore 10:29, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> ha scritto:</div><br class="m_-5191715533486204971Apple-interchange-newline"></div></div><div class=""><div class=""><div class="h5"><div style="word-wrap:break-word" class=""><div class=""><br class=""></div><div class="">Hi All,</div><div class=""><br class=""></div><div class="">some important updates about the issue mentioned below.</div><div class="">After rebalance failed on all the servers, I decided to:</div><div class=""><br class=""></div><div class="">- stop gluster volume</div><div class="">- reboot the servers</div><div class="">- start gluster volume</div><div class="">- change some gluster volume options</div><div class="">- start the rebalance again</div><div class=""><br class=""></div><div class="">The options that I changed are listed below after reading some threads on gluster users mailing list:</div><div class=""><br class=""></div><div class="">BEFORE CHANGE:</div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">gluster volume set tier2 network.ping-timeout 02</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">gluster volume set all cluster.brick-multiplex on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">gluster volume set tier2 cluster.server-quorum-ratio 51%</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">gluster volume set tier2 cluster.server-quorum-type server</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class="">gluster volume set tier2 cluster.quorum-type auto</div></div><div class=""><br class=""></div><div class="">AFTER CHANGE:</div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">gluster volume set tier2 network.ping-timeout 42</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">gluster volume set all cluster.brick-multiplex off</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">gluster volume set tier2 cluster.server-quorum-ratio none</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">gluster volume set tier2 cluster.server-quorum-type none</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">gluster volume set tier2 cluster.quorum-type none</span></div></div><div class=""><br class=""></div><div class="">The result was that rebalance starts moving data from s01, s02 ed s03 servers to s05 and s06 servers (the new added ones), but it failed on s04 server after 37 seconds.</div><div class="">The rebalance is still running and moving data as you can see from the output:</div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[root@s01 ~]# gluster volume rebalance tier2 status</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> localhost 286680 12.6TB 1217960 0 43343 in progress 32:10:24</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s02-stg 126291 12.4TB 413077 0 21932 in progress 32:10:25</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s03-stg 126516 11.9TB 433014 0 21870 in progress 32:10:25</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s04-stg 0 0Bytes 0 0 0 failed 0:00:37</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s05-stg 0 0Bytes 0 0 0 in progress 32:10:25</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s06-stg 0 0Bytes 0 0 0 in progress 32:10:25</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Estimated time left for rebalance to complete : 624:47:48</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">volume rebalance: tier2: success</span></div></div><div class=""><br class=""></div><div class="">When rebalance will be completed, we are planning to re-launch it to try to involve s04 server also.</div><div class="">Do you have some idea about what happened in my previous message and why, now, rebalance it’s running although it’s not involve s04 server?</div><div class=""><br class=""></div><div class="">In attachment the complete tier2-rebalance.log file related to s04 server.</div><div class=""><br class=""></div><div class=""><div class="">Thank you very much for your help,</div><div class="">Mauro</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""></div></div></div></div><span id="m_-5191715533486204971cid:1BCD329C-5AAD-4D01-9A37-E87A1CE408BF@openvpn" class=""><tier2-rebalance.log.gz></span><div style="word-wrap:break-word" class=""><div class=""></div><br class=""><div class=""><blockquote type="cite" class=""><span class=""><div class="">Il giorno 06 ott 2018, alle ore 02:01, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> ha scritto:</div><br class="m_-5191715533486204971Apple-interchange-newline"></span><div class=""><div style="word-wrap:break-word" class=""><span class=""><div class=""><br class=""></div><div class="">Hi All,</div><div class=""><br class=""></div><div class="">since we need to restore gluster storage as soon as possible, we decided to ignore the few files that could be lost and to go ahead.</div><div class="">So we cleaned all bricks content of servers s04, s05 and s06.</div><div class=""><br class=""></div><div class="">As planned some days ago, we executed the following commands:</div><div class=""><br class=""></div><div class=""><div class=""><i class="">gluster peer detach s04</i></div><div class=""><i class="">gluster peer detach s05</i></div><div class=""><i class="">gluster peer detach s06</i></div><div class=""><i class=""><br class=""></i></div><div class=""><i class="">gluster peer probe s04</i></div><div class=""><i class="">gluster peer probe s05</i></div><div class=""><i class="">gluster peer probe s06</i></div><div class=""><i class=""><br class=""></i></div><div class=""><i class="">gluster volume add-brick tier2 s04-stg:/gluster/mnt1/brick s05-stg:/gluster/mnt1/brick s06-stg:/gluster/mnt1/brick s04-stg:/gluster/mnt2/brick s05-stg:/gluster/mnt2/brick s06-stg:/gluster/mnt2/brick s04-stg:/gluster/mnt3/brick s05-stg:/gluster/mnt3/brick s06-stg:/gluster/mnt3/brick s04-stg:/gluster/mnt4/brick s05-stg:/gluster/mnt4/brick s06-stg:/gluster/mnt4/brick s04-stg:/gluster/mnt5/brick s05-stg:/gluster/mnt5/brick s06-stg:/gluster/mnt5/brick s04-stg:/gluster/mnt6/brick s05-stg:/gluster/mnt6/brick s06-stg:/gluster/mnt6/brick s04-stg:/gluster/mnt7/brick s05-stg:/gluster/mnt7/brick s06-stg:/gluster/mnt7/brick s04-stg:/gluster/mnt8/brick s05-stg:/gluster/mnt8/brick s06-stg:/gluster/mnt8/brick s04-stg:/gluster/mnt9/brick s05-stg:/gluster/mnt9/brick s06-stg:/gluster/mnt9/brick s04-stg:/gluster/mnt10/brick s05-stg:/gluster/mnt10/brick s06-stg:/gluster/mnt10/brick s04-stg:/gluster/mnt11/brick s05-stg:/gluster/mnt11/brick s06-stg:/gluster/mnt11/brick s04-stg:/gluster/mnt12/brick s05-stg:/gluster/mnt12/brick s06-stg:/gluster/mnt12/brick force</i></div><div class=""><i class=""><br class=""></i></div><div class=""><i class="">gluster volume rebalance tier2 fix-layout start</i></div></div><div class=""><i class=""><br class=""></i></div><div class=""><span class="">Everything seem to be fine and fix-layout ended.</span></div><div class=""><span class=""><br class=""></span></div></span><div class=""><span class=""><span class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(41,249,20);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[root@s01 ~]# gluster volume rebalance tier2 status </span></div></span><div class=""><div class="h5"><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(41,249,20);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> Node status run time in h:m:s</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(41,249,20);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> --------- ----------- ------------</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(41,249,20);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> localhost fix-layout completed 12:11:6</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(41,249,20);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s02-stg fix-layout completed 12:11:18</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(41,249,20);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s03-stg fix-layout completed 12:11:12</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(41,249,20);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s04-stg fix-layout completed 12:11:20</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(41,249,20);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s05-stg fix-layout completed 12:11:14</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(41,249,20);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> s06-stg fix-layout completed 12:10:47</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(41,249,20);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">volume rebalance: tier2: success</span></div></div></div></span></div><div class=""><div class="h5"><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[root@s01 ~]# gluster volume info</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0);min-height:18px" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><br class="m_-5191715533486204971webkit-block-placeholder"></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Volume Name: tier2</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Type: Distributed-Disperse</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Volume ID: a28d88c5-3295-4e35-98d4-<wbr class="">210b3af9358c</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Status: Started</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Snapshot Count: 0</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Number of Bricks: 12 x (4 + 2) = 72</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Transport-type: tcp</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Bricks:</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick1: s01-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick2: s02-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick3: s03-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick4: s01-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick5: s02-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick6: s03-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick7: s01-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick8: s02-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick9: s03-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick10: s01-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick11: s02-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick12: s03-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick13: s01-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick14: s02-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick15: s03-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick16: s01-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick17: s02-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick18: s03-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick19: s01-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick20: s02-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick21: s03-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick22: s01-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick23: s02-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick24: s03-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick25: s01-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick26: s02-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick27: s03-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick28: s01-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick29: s02-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick30: s03-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick31: s01-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick32: s02-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick33: s03-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick34: s01-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick35: s02-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick36: s03-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick37: s04-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick38: s05-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick39: s06-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick40: s04-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick41: s05-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick42: s06-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick43: s04-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick44: s05-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick45: s06-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick46: s04-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick47: s05-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick48: s06-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick49: s04-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick50: s05-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick51: s06-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick52: s04-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick53: s05-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick54: s06-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick55: s04-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick56: s05-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick57: s06-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick58: s04-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick59: s05-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick60: s06-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick61: s04-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick62: s05-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick63: s06-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick64: s04-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick65: s05-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick66: s06-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick67: s04-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick68: s05-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick69: s06-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick70: s04-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick71: s05-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick72: s06-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Options Reconfigured:</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">network.ping-timeout: 42</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.scrub: Active</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.bitrot: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.inode-quota: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.quota: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.client-io-threads: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.min-free-disk: 10</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.quorum-type: none</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">transport.address-family: inet</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">nfs.disable: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">server.event-threads: 4</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">client.event-threads: 4</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.lookup-optimize: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.readdir-ahead: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.parallel-readdir: off</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.readdir-optimize: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.cache-invalidation: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.cache-invalidation-<wbr class="">timeout: 600</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.stat-prefetch: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.cache-<wbr class="">invalidation: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.md-cache-timeout: 600</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">network.inode-lru-limit: 50000</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><a href="http://performance.io/" target="_blank" class="">performance.io</a>-cache: off</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">disperse.cpu-extensions: auto</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><a href="http://performance.io/" target="_blank" class="">performance.io</a>-thread-count: 16</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.quota-deem-statfs: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.default-soft-limit: 90</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.server-quorum-type: none</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">diagnostics.latency-<wbr class="">measurement: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">diagnostics.count-fop-hits: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.brick-multiplex: off</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.server-quorum-ratio: 51%</span></div></div><div class=""><br class=""></div><div class="">The last step should be the data rebalance between the servers, but rebalance failed soon with a lot of errors like the following ones:</div><div class=""><br class=""></div><div class=""><div class="">[2018-10-05 23:48:38.644978] I [MSGID: 114035] [client-handshake.c:202:<wbr class="">client_set_lk_version_cbk] 0-tier2-client-70: Server lk version = 1</div><div class="">[2018-10-05 23:48:44.735323] I [dht-rebalance.c:4512:gf_<wbr class="">defrag_start_crawl] 0-tier2-dht: gf_defrag_start_crawl using commit hash 3720331860</div><div class="">[2018-10-05 23:48:44.736205] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_<wbr class="">update_cbk] 0-tier2-disperse-7: Failed to get size and version [Input/output error]</div><div class="">[2018-10-05 23:48:44.736266] E [MSGID: 122034] [ec-common.c:613:ec_child_<wbr class="">select] 0-tier2-disperse-7: Insufficient available children for this request (have 0, need 4)</div><div class="">[2018-10-05 23:48:44.736282] E [MSGID: 122037] [ec-common.c:2040:ec_update_<wbr class="">size_version_done] 0-tier2-disperse-7: Failed to update version and size [Input/output error]</div><div class="">[2018-10-05 23:48:44.736377] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_<wbr class="">update_cbk] 0-tier2-disperse-8: Failed to get size and version [Input/output error]</div><div class="">[2018-10-05 23:48:44.736436] E [MSGID: 122034] [ec-common.c:613:ec_child_<wbr class="">select] 0-tier2-disperse-8: Insufficient available children for this request (have 0, need 4)</div><div class="">[2018-10-05 23:48:44.736459] E [MSGID: 122037] [ec-common.c:2040:ec_update_<wbr class="">size_version_done] 0-tier2-disperse-8: Failed to update version and size [Input/output error]</div><div class="">[2018-10-05 23:48:44.736460] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_<wbr class="">update_cbk] 0-tier2-disperse-10: Failed to get size and version [Input/output error]</div><div class="">[2018-10-05 23:48:44.736537] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_<wbr class="">update_cbk] 0-tier2-disperse-9: Failed to get size and version [Input/output error]</div><div class="">[2018-10-05 23:48:44.736571] E [MSGID: 122034] [ec-common.c:613:ec_child_<wbr class="">select] 0-tier2-disperse-10: Insufficient available children for this request (have 0, need 4)</div><div class="">[2018-10-05 23:48:44.736574] E [MSGID: 122034] [ec-common.c:613:ec_child_<wbr class="">select] 0-tier2-disperse-9: Insufficient available children for this request (have 0, need 4)</div><div class="">[2018-10-05 23:48:44.736604] E [MSGID: 122037] [ec-common.c:2040:ec_update_<wbr class="">size_version_done] 0-tier2-disperse-9: Failed to update version and size [Input/output error]</div><div class="">[2018-10-05 23:48:44.736604] E [MSGID: 122037] [ec-common.c:2040:ec_update_<wbr class="">size_version_done] 0-tier2-disperse-10: Failed to update version and size [Input/output error]</div><div class="">[2018-10-05 23:48:44.736827] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_<wbr class="">update_cbk] 0-tier2-disperse-11: Failed to get size and version [Input/output error]</div><div class="">[2018-10-05 23:48:44.736887] E [MSGID: 122034] [ec-common.c:613:ec_child_<wbr class="">select] 0-tier2-disperse-11: Insufficient available children for this request (have 0, need 4)</div><div class="">[2018-10-05 23:48:44.736904] E [MSGID: 122037] [ec-common.c:2040:ec_update_<wbr class="">size_version_done] 0-tier2-disperse-11: Failed to update version and size [Input/output error]</div><div class="">[2018-10-05 23:48:44.740337] W [MSGID: 122040] [ec-common.c:1097:ec_prepare_<wbr class="">update_cbk] 0-tier2-disperse-6: Failed to get size and version [Input/output error]</div><div class="">[2018-10-05 23:48:44.740381] E [MSGID: 122034] [ec-common.c:613:ec_child_<wbr class="">select] 0-tier2-disperse-6: Insufficient available children for this request (have 0, need 4)</div><div class="">[2018-10-05 23:48:44.740394] E [MSGID: 122037] [ec-common.c:2040:ec_update_<wbr class="">size_version_done] 0-tier2-disperse-6: Failed to update version and size [Input/output error]</div><div class="">[2018-10-05 23:48:50.066103] I [MSGID: 109081] [dht-common.c:4379:dht_<wbr class="">setxattr] 0-tier2-dht: fixing the layout of /</div></div><div class=""><br class=""></div><div class="">In attachment you can find the first logs captured during the rebalance execution.</div><div class="">In your opinion, is there a way to restore the gluster storage or all the data have been lost?</div><div class=""><br class=""></div><div class="">Thank you in advance,</div><div class="">Mauro</div><div class=""><br class=""></div><div class=""></div></div></div></div><span id="m_-5191715533486204971cid:1C722A07-5FB4-4759-BB6C-AE4D17E25230@openvpn" class=""><rebalance_log.txt></span><div style="word-wrap:break-word" class=""><div class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div class=""><blockquote type="cite" class=""><div class="">Il giorno 04 ott 2018, alle ore 15:31, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> ha scritto:</div><br class="m_-5191715533486204971Apple-interchange-newline"><div class=""><div style="word-wrap:break-word" class=""><div class=""><br class=""></div><div class="">Hi Nithya,</div><div class=""><br class=""></div><div class="">thank you very much.</div><div class="">This is the current “gluster volume info” output after removing bricks (and after peer detach command).</div><div class=""><br class=""></div><div class=""><span class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[root@s01 ~]# gluster volume info</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0);min-height:18px" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><br class="m_-5191715533486204971webkit-block-placeholder"></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Volume Name: tier2</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Type: Distributed-Disperse</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Volume ID: a28d88c5-3295-4e35-98d4-<wbr class="">210b3af9358c</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Status: Started</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Snapshot Count: 0</span></div></span><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Number of Bricks: 6 x (4 + 2) = 36</span></div><div class=""><div class="h5"><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Transport-type: tcp</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Bricks:</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick1: s01-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick2: s02-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick3: s03-stg:/gluster/mnt1/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick4: s01-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick5: s02-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick6: s03-stg:/gluster/mnt2/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick7: s01-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick8: s02-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick9: s03-stg:/gluster/mnt3/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick10: s01-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick11: s02-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick12: s03-stg:/gluster/mnt4/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick13: s01-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick14: s02-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick15: s03-stg:/gluster/mnt5/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick16: s01-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick17: s02-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick18: s03-stg:/gluster/mnt6/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick19: s01-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick20: s02-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick21: s03-stg:/gluster/mnt7/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick22: s01-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick23: s02-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick24: s03-stg:/gluster/mnt8/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick25: s01-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick26: s02-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick27: s03-stg:/gluster/mnt9/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick28: s01-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick29: s02-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick30: s03-stg:/gluster/mnt10/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick31: s01-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick32: s02-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick33: s03-stg:/gluster/mnt11/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick34: s01-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick35: s02-stg:/gluster/mnt12/brick</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick36: s03-stg:/gluster/mnt12/brick</span></div></div></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Options Reconfigured:</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">network.ping-timeout: 0</span></div><span class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.scrub: Active</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.bitrot: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.inode-quota: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.quota: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.client-io-threads: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.min-free-disk: 10</span></div></span><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.quorum-type: auto</span></div><span class=""><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">transport.address-family: inet</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">nfs.disable: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">server.event-threads: 4</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">client.event-threads: 4</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.lookup-optimize: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.readdir-ahead: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.parallel-readdir: off</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.readdir-optimize: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.cache-invalidation: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.cache-invalidation-<wbr class="">timeout: 600</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.stat-prefetch: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.cache-<wbr class="">invalidation: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.md-cache-timeout: 600</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">network.inode-lru-limit: 50000</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><a href="http://performance.io/" target="_blank" class="">performance.io</a>-cache: off</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">disperse.cpu-extensions: auto</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><a href="http://performance.io/" target="_blank" class="">performance.io</a>-thread-count: 16</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.quota-deem-statfs: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">features.default-soft-limit: 90</span></div></span><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.server-quorum-type: server</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">diagnostics.latency-<wbr class="">measurement: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">diagnostics.count-fop-hits: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.brick-multiplex: on</span></div><div style="margin:0px;line-height:normal;font-family:'Courier New';color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">cluster.server-quorum-ratio: 51%</span></div></div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Mauro</div><br class=""><div class=""><blockquote type="cite" class=""><div class="">Il giorno 04 ott 2018, alle ore 15:22, Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank" class="">nbalacha@redhat.com</a>> ha scritto:</div><br class="m_-5191715533486204971Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class="">Hi Mauro,<div class=""><br class=""></div><div class=""><br class=""></div><div class="">The files on s04 and s05 can be deleted safely as long as those bricks have been removed from the volume and their brick processes are not running.</div><div class=""><br class=""></div><div class=""><br class=""><pre class="m_-5191715533486204971gmail-aLF-aPX-K0-aPE" style="display:block;margin:0px;word-wrap:break-word;background-color:rgb(255,255,255);text-align:left;text-decoration-style:initial;text-decoration-color:initial"><font face="arial, helvetica, sans-serif" class=""><span style="font-size:14px;white-space:pre-wrap" class="">.glusterfs/indices/xattrop/<wbr class="">xattrop-* are links to files that need to be healed.
.glusterfs/quarantine/stub-<wbr class="">00000000-0000-0000-0000-<wbr class="">000000000008 links to files that bitrot (if enabled)says are corrupted. (none in this case)</span></font><font class=""><span style="font-size:14px;white-space:pre-wrap" class=""><br class=""></span></font></pre><pre class="m_-5191715533486204971gmail-aLF-aPX-K0-aPE" style="display:block;margin:0px;word-wrap:break-word;background-color:rgb(255,255,255);text-align:left;text-decoration-style:initial;text-decoration-color:initial"><font face="arial, helvetica, sans-serif" class=""><span style="font-size:14px;white-space:pre-wrap" class=""><br class=""></span></font></pre><pre class="m_-5191715533486204971gmail-aLF-aPX-K0-aPE" style="display:block;margin:0px;word-wrap:break-word;background-color:rgb(255,255,255);text-align:left;text-decoration-style:initial;text-decoration-color:initial"><font face="arial, helvetica, sans-serif" class=""><span style="font-size:14px;white-space:pre-wrap" class=""><br class=""></span></font></pre><pre class="m_-5191715533486204971gmail-aLF-aPX-K0-aPE" style="display:block;margin:0px;word-wrap:break-word;background-color:rgb(255,255,255);text-align:left;text-decoration-style:initial;text-decoration-color:initial"><font face="arial, helvetica, sans-serif" class=""><span style="font-size:14px;white-space:pre-wrap" class="">I will get back to you on s06. Can you please provide the output of gluster volume info again?</span></font></pre></div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Nithya<br class=""><div class=""><br class=""></div><div class=""><br class=""></div></div></div></div><div class=""><div class="h5"><div class="gmail_extra"><br class=""><div class="gmail_quote">On 4 October 2018 at 13:47, Mauro Tridici <span dir="ltr" class=""><<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><br class=""><div class="">
<span class="m_-5191715533486204971m_-7415967095465071885Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="word-wrap:break-word" class=""><span class="m_-5191715533486204971m_-7415967095465071885Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="font-family:Helvetica;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-transform:none;white-space:normal;word-spacing:0px" class=""></div>Dear Ashish, Dear Nithya,</span></div><div style="word-wrap:break-word" class=""><span class="m_-5191715533486204971m_-7415967095465071885Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><br class=""></span></div><div style="word-wrap:break-word" class=""><span class="m_-5191715533486204971m_-7415967095465071885Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px">I’m writing this message only to summarize and simplify the information about the "not migrated” files left on removed bricks on server s04, s05 and s06.</span></div><div style="word-wrap:break-word" class=""><span style="text-align:-webkit-auto" class="">In attachment, you can find 3 files (1 file for each server) containing the “not migrated” files lists and related brick number.</span></div><div style="word-wrap:break-word" class=""><span style="text-align:-webkit-auto" class=""><br class=""></span></div><div style="word-wrap:break-word" class=""><span style="text-align:-webkit-auto" class="">In particular:</span></div><div style="word-wrap:break-word" class=""><ul class=""><li class=""><span style="text-align:-webkit-auto" class="">s04 and s05 bricks contain only not migrated files in hidden directories “/gluster/mnt#/brick/.glusterf<wbr class="">s" (I could delete them, doesn’t it?)</span></li><li class=""><span style="text-align:-webkit-auto" class="">s06 bricks contain </span></li><ul class=""><li class=""><span style="text-align:-webkit-auto" class="">not migrated files in hidden directories </span>“/gluster/mnt#/bri<wbr class="">ck/.glusterfs”;</li><li class="">not migrated files with size equal to 0;</li><li class="">not migrated files with size greater than 0.</li></ul></ul><div class=""><br class=""></div><div class="">I think it was necessary to collect and summarize information to simplify your analysis.</div><div class="">Thank you very much,</div><div class="">Mauro</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""></div></div></span></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></blockquote></div><br class=""></div></div></blockquote></div><br class=""><div class="">
<span class="m_-5191715533486204971Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;border-spacing:0px"><div style="word-wrap:break-word" class=""><span class="m_-5191715533486204971Apple-style-span" style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;border-spacing:0px"><div style="font-family: Helvetica; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><br class=""></div></span></div></span></div></div></blockquote></div></div></div></blockquote></div></body></html>